What about the number of visits? Often when the human move is not the same as the bot's top choice, the human move gets relatively few visits by comparison with the top choice. Few visits means unreliable winrate estimates. In the GoGoD commentaries by Elf, human plays with fewer than 500 visits are not directly evaluated. Instead, they inherit their winrate estimates from Elf's next top choice. Even better, IMO, is to make the human play directly and see how the bot evaluates it. Then you have a comparable comparison.

If the winrate estimates of a bot's top choices change suddenly on another top choice, that indicates a horizon effect. The bot sees something that makes it change its winrate estimates. If that happens regularly, then the bot needs more rollouts to make reliable evaluations.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

My vague recollection is this doesn't seem related to complex fighting or horizont effect. Few visits are more of a reason (I have 1060 and use 15b nets, and only casual walkthrough my games). But the point is, this doesn't seem to happen with LZ 15b but frequent with Katago 15b.

It would be interesting to see test matches where the value/policy nets are swapped independently between LZ and Kata. (I suspect this effect might be caused by LZ policy being better and more focused on key lines, thus allowing better evaluations with the first few dozen visits.)

I upgraded my notebook computer's graphics driver from ASUS, which is per advice from Intel Driver Support & Assistant (a tool), it seems OpenCL 1.2 and 2.0 are in place, Leela Zero GPU version fails (see below), but Leela Zero CPU-only version still works.


C:\WINDOWS\system32>C:/leela-zero-0.17-win64/leelaz.exe -g -b 0 -w C:/leela-zero-0.17-win64/266_40b_15926.gz
Using OpenCL batch size of 5
Using 10 thread(s).
RNG seed: 10561116271007652040
Using per-move time margin of 0.00s.
BLAS Core: Haswell
Detecting residual layers...v1...256 channels...40 blocks.
Initializing OpenCL (autodetecting precision).
Detected 2 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.112
Platform profile: FULL_PROFILE
Platform name: NVIDIA CUDA
Platform vendor: NVIDIA Corporation
Device ID: 0
Device name: GeForce 920M
Device type: GPU
Device vendor: NVIDIA Corporation
Device driver: 388.57
Device speed: 954 MHz
Device cores: 2 CU
Device score: 1112
Platform version: OpenCL 2.0
Platform profile: FULL_PROFILE
Platform name: Intel(R) OpenCL
Platform vendor: Intel(R) Corporation
Device ID: 1
Device name: Intel(R) HD Graphics 5500
Device type: GPU
Device vendor: Intel(R) Corporation
Device driver: 20.19.15.4404
Device speed: 900 MHz
Device cores: 24 CU
Device score: 620
Device ID: 2
Device name: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
Device type: CPU
Device vendor: Intel(R) Corporation
Device driver: 5.2.0.10094
Device speed: 2200 MHz
Device cores: 4 CU
Device score: 520
Selected platform: NVIDIA CUDA
Selected device: GeForce 920M
with OpenCL 1.2 capability.
Error creating OpenCL context: clCreateContext: 999

"Few visits" is how many? 10? 100? 1000?

As mentioned 1060 on 15b for casual reviews, so 10-15 sec per move, which gives around 10k visits in total. So inferior alternatives may not get more than 100 visits. That's low for sure, but again the point is the difference between LZ and Kata behavior.

Anyway I wouldn't trust any bot's judgment on a move that gets less than 100 visits, and in a complex fight more visits are needed. LZ may be better than KataGo in some cases, but on low visits, 15-block LZ easily misreads ladders...

Well, the alternative is 0 visits - I either do a quick review or not.

On second thought there may be more to this. During the next few moves the top move does have a significant part of the 10k, still the drop only happens later... Will take a closer look next time.

Yes, there is more to it. It's to do with the way confidence bounds are used in the search. You can have a second-choice move where the initial evaluation is 55%, after a couple of thousand playouts it's gone down to 50% while the first-choice move is still up at 52%. Perhaps another thousand playouts would reveal that the second choice move is really a 40% move, but why do those playouts once you're confident that the move is second-best? The goal of the algorithm is to identify the best move, not to accurately rank all the moves. If you do want an accurate evaluation of each move, you have to click on each move and evaluate them one by one (or else used a modified version of the software, as we've discussed elsewhere).

In my list of ladder positions that I'll post once I have an uninterrupted couple of hours for writing about go software, I've got a concrete example of this. Sorry about the delay...

I have a quote with an ASUS GPU with 1066 Supper chipset now, how many visits per second with LZ266 should I expect?

On the other hands, Based on a China benchmark post for LZ up to 24x, 2060 has the highest performace to cost ratio, moreover, the 20 series has tensor core fearure, which boosts AI performance.Haven't seen similar comments in English.

But this is not during search. As I wrote above the effect/drop occurred after next few moves actually played in the game (all the bot's top choices).

Ah, I was thinking of the case where the second choice move turns out to be much worse than advertised. But you're describing something different: if you play out the top choice move, the evaluation still changes. Sorry for the misundestanding.

Are you able to show us a specific position where this happens?

No, I just wrote this from memory (but will take a better look next time). I used to use LZ for these quick reviews, only recently tried switching to Kata, then a few times I run into this.

Basically, I search for significant mistakes only, where the reason and the better alternative is easy to explain when seen. So if I see a searched but not chosen move, with reasonably close eval, I go on. If the move was actually a serious mistake it should drop on next move. But instead I find nothing major, and keep moving on a few more moves (mostly top choices), then the eval is much worse. After some time consuming back and forth, the best explanation I find is this earlier move.

Besides the above policy differences, another part of the reason may be LZ eval (trained partially on Elf data) is more saturated than Kata, so drops are faster and more recognizable.