Confirmation bias in neural nets?

[Bill Spight]

I just stumbled across this paper about confirmation bias in humans based upon previous choices. https://www.sciencedirect.com/science/a ... 2218309825

Here are the highlights of the paper:

Highlights
•
People’s interpretation of new evidence is often biased by their previous choices

•
Talluri, Urai et al. developed a new task for probing the underlying mechanisms

•
Evidence consistent with an observer’s initial choice is processed more efficiently

•
This “choice-induced gain change” affects both perceptual and numerical decisions

Emphasis mine.

While, OC, brains are not the same as neural nets, the highlighted point seems to me to similar to how neural net bots work, both in play and in training. In play, initial choices based upon prior learning (with some randomness) are used to build the game tree, and thus, are processed more efficiently. In self-play training, the points identified as important by each player get more processing. In actual brains, neural connections that get more processing (activation) tend to be reinforced. I don't know if the same kind of thing applies to AI neural nets, but if so, "Confirmation Bias through Selective Overweighting of Choice-Consistent Evidence" may apply to them, as well.

[yoyoma]

Yes I think some problems with NNs (especially when Reinforcement Learning gets involved) are similar to confirmation bias in humans. Just recently someone did a study of a positive feedback problem that could cause a NN to become convinced certain moves are best and ignore the alternatives:

https://github.com/leela-zero/leela-zero/issues/2230

[MikeKyle]

I've heard Catalin Taranu 5p mention a few times something his teacher would say. I may get the quote wrong but I think it's something along the lines of

If you find that you want to play a move then just keep reading until you discover that it's the right move

I kind of feel like confirmation bias might lead to blindspots and isolated mistakes, but might be pretty good for go strength.

[Bill Spight]

I've heard Catalin Taranu 5p mention a few times something his teacher would say. I may get the quote wrong but I think it's something along the lines of

If you find that you want to play a move then just keep reading until you discover that it's the right move

I kind of feel like confirmation bias might lead to blindspots and isolated mistakes, but might be pretty good for go strength.

I would word that slightly differently.

If you find that you want to play a move then just keep trying the prove it wrong until you discover that it's the right move.

[John Fairbairn]

Bill: Good and timely topic. I think it can apply also to efforts to understand why AI bots are stronger.

We look at their games and notice first, or most intently, those moves which we think we can understand on the basis of existing knowledge. It seems easy enough to find things that represent an improvement that we can then, a posteriori, justify in a way that we find very convincing because it confirms what we already know! We could know it just a little better, it seems, but we do know it!

I suspect the early (and continuing) focus on josekis derives from precisely this approach. Josekis are an area we feel we must know pretty well.

I mention this because of something I read just yesterday. I had felt early on that the focus on josekis was a bit misguided. My own hunch was that the real explanation of the strength of bots was primarily in the fact that they didn't make careless mistakes like us (ladders excepted) but they probably also saw more in the centre of the board than we will ever be able to.

I had no proof, of course, but felt it rather strongly. I was accordingly rather shaken this week when I read something by a Japanese pro in which he said they have concluded something different: the reason the bots don't like the small knight's move shimari is because it is overconcentrated. Well, overconcentration is something I know a lot about. I make it more than most people. So I started looking at some games in that light. These were pro games, not AI games, but nevertheless games in which the pros were clearly trying to play like bots. Blow me, just about every move could be explained as an attempt to force overconcentration or to resist it - all these shoulder hits, attachments, playing close to thickness... Everything fits. It must be true because I know what overconcentration is, after all

But a little more seriously, the thing I have picked up most from recently studying the Genjo-Chitoku games (or, more precisely, their commentaries) is how fixated pros are with the efficiency of plays. For example, the sheer number of comments on forcing moves and timing is staggering.

And overconcentration is just an aspect of efficiency. Unlike us, a bot can measure efficiency quite easily. It occurred to me therefore that if they do ever learn to talk about go, they won't ever use any of our terms (except to pander to us). Every explanation will basically be just "this is the most efficient move in this sort of position," all backed up by what is essentially just looking up results stored in a massive database. This is essentially what the recent Elf exercise amounts to.

But if that, or what the Japanese pros say, is close to correct, it does at least tell us very specifically what we need to study more: efficiency. And the beauty of doing that is that even if it wide of the mark, it can't do any harm. But we are in the same position as the bots on that. We need to find a way to talk about it

[mhlepore]

...In self-play training, the points identified as important by each player get more processing. In actual brains, neural connections that get more processing (activation) tend to be reinforced. I don't know if the same kind of thing applies to AI neural nets, but if so, "Confirmation Bias through Selective Overweighting of Choice-Consistent Evidence" may apply to them, as well.

Presuming that there is a legitimate reason the points were identified for more processing in the first place (i.e., because they increase the chance of winning), then I don't see how this leads to confirmation bias. Especially if we are talking about a self-play system where millions of rounds of learning can take place. Biases that are based on something suboptimal would be exploited and weeded out.

Or am I giving these algorithms too much credit? (I probably am, but I'll keep the question as a straw man)

[Tryss]

You need to understand how the training of a neural network like LZ (or Alpha Zero, or ELF) works :

Step 0 : You start with a random network that predict "best move" for a board and the winner.

Step 1 : You play games with this network (with randomness)

Step 2 : You then use these games to train a new network that predict the moves of the winner of these games and the winner

Step 3 : you test if this new network is better than the old one, if it is, it become the new network. Return to step 1


So obviously, this approach may get the network stuck in a local optimum worse than the global optimum. Fortunatly, it seems that a little randomness allow for enough exploration to get the level to a superhuman strength, but it doesn't means that it can't happens.

[Bill Spight]

I suspect the early (and continuing) focus on josekis derives from precisely this approach. Josekis are an area we feel we must know pretty well.

I mention this because of something I read just yesterday. I had felt early on that the focus on josekis was a bit misguided.

Well, it seems to me that all of a sudden we have glimpsed the joseki of 200 years into the future. That's a pretty big deal. But I also think that that is not the main difference between us and the bots.

My own hunch was that the real explanation of the strength of bots was primarily in the fact that they didn't make careless mistakes like us (ladders excepted) but they probably also saw more in the centre of the board than we will ever be able to.

Yes, I think that the bots understand the centre even better than the sainted Go Seigen, and that is a major advance. They understand the sides better, too.

I had no proof, of course, but felt it rather strongly. I was accordingly rather shaken this week when I read something by a Japanese pro in which he said they have concluded something different: the reason the bots don't like the small knight's move shimari is because it is overconcentrated.

Sounds right to me. My crude influence heuristic doesn't like it, either, for that reason. FWIW.

Well, overconcentration is something I know a lot about. I make it more than most people. So I started looking at some games in that light. These were pro games, not AI games, but nevertheless games in which the pros were clearly trying to play like bots. Blow me, just about every move could be explained as an attempt to force overconcentration or to resist it - all these shoulder hits, attachments, playing close to thickness... Everything fits. It must be true because I know what overconcentration is, after all

But a little more seriously, the thing I have picked up most from recently studying the Genjo-Chitoku games (or, more precisely, their commentaries) is how fixated pros are with the efficiency of plays. For example, the sheer number of comments on forcing moves and timing is staggering.

Interesting observation.

And overconcentration is just an aspect of efficiency. Unlike us, a bot can measure efficiency quite easily.

Well, they certainly do not try to measure it. It seems plausible that a good theory of their play would have the concept of efficiency, however. As for overconcentration, one crude measure might be the difference between Black and White plays in a given region.

It occurred to me therefore that if they do ever learn to talk about go, they won't ever use any of our terms (except to pander to us).

I think that we will need new concepts. ::)

Every explanation will basically be just "this is the most efficient move in this sort of position," all backed up by what is essentially just looking up results stored in a massive database. This is essentially what the recent Elf exercise amounts to.

Humans are rather good at coming up with explanations, and good ones, too. A massive database is not necessarily a problem. I have mentioned how researchers were able to reproduce, within a close margin of error, the predictions of a big data algorithm, using only three parameters. In the next decade we are going to come up with some new insights into go.

[Bill Spight]

So obviously, this approach may get the network stuck in a local optimum worse than the global optimum. Fortunatly, it seems that a little randomness allow for enough exploration to get the level to a superhuman strength, but it doesn't means that it can't happens.

Oh, I am not aware of getting stuck in a local optimum, or even that the law of diminishing returns has kicked in. But there does seem to be some path dependency. Greater exploration might help reduce blind spots, but it might also slow down improvement.

For instance, suppose that we trained two zero bots to play against each other, rather than against themselves. If one of them found a blind spot in the other, it could learn to exploit it, and the other bot could learn to correct it. But training two bots at the same time would increase development time.

[moha]

Greater exploration might help reduce blind spots, but it might also slow down improvement.

Not necessarily significantly. Depending on the bot, some of these blind spots develop because a key move is never looked at. So the bias comes from always selecting the better move from two previous candidates (reinforcing it), never a third non-candidate. This can be corrected relatively cheaply by allocating a few visits to non-candidate moves as well.

A common assumption is that the strongest playing configuration should be used to generate training games. This leads to looking at narrow range of moves as that happens to maximize playing strength (per visits). But the strongest player is not necessarily the best teacher, and the benefits of exploration is different from a training view than during playing.

[Bill Spight]

Greater exploration might help reduce blind spots, but it might also slow down improvement.

Not necessarily significantly. Depending on the bot, some of these blind spots develop because a key move is never looked at. So the bias comes from always selecting the better move from two previous candidates (reinforcing it), never a third non-candidate. This can be corrected relatively cheaply by allocating a few visits to non-candidate moves as well.

Right.

A common assumption is that the strongest playing configuration should be used to generate training games. This leads to looking at narrow range of moves as that happens to maximize playing strength (per visits). But the strongest player is not necessarily the best teacher, and the benefits of exploration is different from a training view than during playing.

Emphasis mine.

Back when I was considering such things, it was plain that a good player needed to be able to refute bad moves, but there are some bad moves that a good opponent will never play, and so the ability to refute them may not be learned or may be lost. (Efficient learning involves forgetting.) So when I was thinking about a training regime, I thought that it might be a good idea, before testing Alpha(n+1) against Alpha(n), that Alpha(n) had to beat all the previous Alphas, to prove that it could refute their bad plays.

[Aidoneus]

So obviously, this approach may get the network stuck in a local optimum worse than the global optimum. Fortunatly, it seems that a little randomness allow for enough exploration to get the level to a superhuman strength, but it doesn't means that it can't happens.

Sounds like premature convergence, a subject much studied in designing genetic algorithms, as well as experience with evolution in isolated populations.

[Kirby]

There is a classic problem in machine learning - the tradeoff between "exploration" and "exploitation". It comes up in reinforcement learning quite a bit.

Basically, there's a balance between exploiting what you already "know" and exploring what's yet to be known. Here's some lecture slides by our good friend, David Silver: http://www0.cs.ucl.ac.uk/staff/d.silver ... les/XX.pdf

The idea of confirmation bias in neural nets reminds me a little bit of this. I suppose leaning toward the "exploitation" side too much in RL would have a similar effect to confirmation bias.

Anyway, a takeaway from this is that there is a balance between reinforcing what you already know and exploring new options. Either strategy, in isolation, is suboptimal in many domains.

[Gomoto]

Confirmation bias theory is an example for a confirmation bias.

[jlt]

Back when I was considering such things, it was plain that a good player needed to be able to refute bad moves, but there are some bad moves that a good opponent will never play, and so the ability to refute them may not be learned or may be lost. (Efficient learning involves forgetting.) So when I was thinking about a training regime, I thought that it might be a good idea, before testing Alpha(n+1) against Alpha(n), that Alpha(n) had to beat all the previous Alphas, to prove that it could refute their bad plays.

I don't think there is any need to worry. I would be surprised if, for some n, LeelaZero(n) didn't beat LeelaZero(n-10) more than 50% of the time.