Life In 19x19

And I have sgf files for them: 4m_symm_without33 in https://github.com/isty2e/baduk-opening-book

Not exactly Sabaki, but I think KaTrain might be helpful for you, since you can play against a weaker, rank-tunable version of KataGo.

IMO, a review program should devote about the same number of rollouts to each play that it compares with the other. I am unaware of any program that does that, but a human can make the bot do that. :)

The list of sensible moves will vary person by person, and by concentrating on the best moves ...

The last time parity benchmark I did was between KataGo v1.3.2+g170-20x256-1.9G (g170-b20c256x2-s1913382912-d435450331) and Leela Zero v0.17+#263. I tested with 2xRTX2080Ti, with 60 threads for KataGo and 20 threads for LZ. The results were:

KataGo (1600 po) vs Leela Zero (1000 po): KataGo won 279 ...

I consider fixed playout tests more suitable for such purposes, since every engine will reuse the search tree and number of playouts is supposed to be more correlated to time limit than the number of visits is.

But such results are inherently harder to interpret and less portable, and IF/WHEN you are unable to test on the exact conditions of later use, narrower tests that are affected by less factors and thus more consistent can be more informative with less danger of being misleading. Advantages and ...

Do you imply that 1000 vs 1000 visits will be consistent with 10000 vs 10000 visits, but 1500 vs 1000 won't be so with 15000 vs 10000, for example?
Yes, this is also the exact meaning of the scalability graph you linked.

You are right about the latter, but the former is less justified. As both ...

With a playout based test, the side that tend to support more tree reuse has, say, a constant 1.5x effective search advantage. This is similar to a smaller, weaker but faster net, which (with proportional visits, or time based test) can win low search matches but lose high search matches.

Such a ...

Being stronger somewhere at identical visits normally determines the rest of the curve

Not necessarily, I suppose. For example, there are U-shaped curve, where a weaker network benefits from more visits at sweet spots, but falls of at higher visits due to scaling. However, another group consists ...

The difference in scalability means that net A needs 1.5x more visits around 1000 visit (to compensate for being weaker) but 2.5x more around 10000 visits. In both cases it is weaker than B, and would (obviously) lose at 1.0x visits.

Network strengths will not (or rarely) swap, what usually ...

The margin of victory will be different, but not the winner (assuming identical visits - unlike with identical playouts).

Those linked tests used non-identical visits, thus widened the test up to a new factor (scalability). But this was on purpose there, since the test was not about raw strength ...

A weaker net with sharper policy that allows more tree reuse trades search quality for (effective) quantity. At 1000 playouts it may win your test, and you may think it is "stronger in real world", but at 10000 playouts (where search quality starts to matter more) it may lose.

Thus your results ...

For example, if you clear the tree each move, fixed playout tests are heavily affected (the same amount of playouts / work will do less effective search) while fixed visit tests are less so (single threaded at least).

Yes, clearing the search tree will certainly affect the results, but I don't ...

For example, tree reuse may work quite differently for high-visit and low-visit scenarios.
How?

Then test results that included tree reuse extent may become less relevant than narrower ones.
I'm not sure what you are meaning by "wide" and "narrow" here. And the search tree will be reused in ...

The more things you test at the same time (ie. network strength plus policy sharpness / tree reuse intensity) the harder to measure those things independently (same as with hw and other external factors).

The policy sharpness and therefore tree reuse as well are strongly bound to the nature of ...