Life In 19x19

This post is the result of time spent looking in detail at the SGF files output
by Lizzie and KataGo.

1. Sometimes the percentage win rate goes up when the 'correct' move is played.
The two images show how the analysis changed in a no-komi game with 5000 playouts.
KataGo increases White's win rate from 29.3% to 31.2%.
There is presumably some benefit from playing the correct move, in that the
game is now one move nearer its end, which slightly reduces the fog of uncertainty.
However, a change of 1.9% feels excessive if that is the only factor at work.

(With more playouts, KataGo slightly prefers White Q5.)

2. As can be seen from the screen captures, Lizzie shows different win rates
in the main board display and the win-rate graph. Data in the SGF file
reflects the win-rate graph. Why are the two not updated in parallel?

3. Lizzie stores the data from KataGo using lengthy 'LZ' tags in the SGF file
This tag was not mentioned in the thread on extending SGF
(viewtopic.php?f=10&t=16425). Has 'LZ' become a de facto
standard?

My preferred SGF editor for Linux is Quarry, which ignores LZ tags
when reading an SGF file and preserves them on writing. This is
apparently correct behaviour, though other editors behave differently.

4. Is Lizzie now abandoned? What are the cool kids using now with Linux?

For reference, I have Lizzie 0.7.4 (patched version) and KataGo 1.7.4.

thirdfogie wrote: 1. Sometimes the percentage win rate goes up when the 'correct' move is played.
The two images show how the analysis changed in a no-komi game with 5000 playouts.
KataGo increases White's win rate from 29.3% to 31.2%.
There is presumably some benefit from playing the correct move, in that the
game is now one move nearer its end, which slightly reduces the fog of uncertainty.
However, a change of 1.9% feels excessive if that is the only factor at work.

I don't know what happened in this specific example, but AIs will change their opinions over time when evaluating a position. It could be as simple as looking at the position with a different number of playouts.

4. Is Lizzie now abandoned? What are the cool kids using now with Linux?

Don't know about cool kids, but I made this.

FWIW - the difference between the percentage/points shown on the blue move and the percentage/points shown on the next blue move, after playing the blue move, is what I take as an AI's precision. I've never seen it as high as 2 points though, rather in the area of 0.7-0.8, given sufficient playouts on each move.

Hi bernds, thanks for the reply. I will try more KataGo playouts to check whether doing so reduces
the gain from playing the correct move.

I tried installing q5Go, but failed. The ZIP file was unpacked into a suitable directory.
It turns out that the qmake package was missing, so... Next... However, qmake then complains as follows and gives up... Since I don't know what a QT module is or how to get one, I gave up too.
The system runs Debian bullseye (V11.3) with XFCE.
Maybe my barebones approach is part of the problem.

thirdfogie wrote:

Code: Select all

Project ERROR: Unknown module(s) in QT: core gui widgets xml network multimedia svg sql

Since I don't know what a QT module is or how to get one, I gave up too.
The system runs Debian bullseye (V11.3) with XFCE.
Maybe my barebones approach is part of the problem.

Yeah, you need all the Qt5 packages as well. Unfortunately these are a bit confusing on Debian based systems. I think you probably need at least There was some discussion here.

2. As can be seen from the screen captures, Lizzie shows different win rates
in the main board display and the win-rate graph. Data in the SGF file
reflects the win-rate graph. Why are the two not updated in parallel?

The winrate graph presumably is looking at the root value, which is an average across all the different moves' values weighted by their visit count (or something similar to that). This will generally make it slightly worse for the side to move than the winrate of the top move for that player, but it is more robust. The winrate of the top move for a player may itself be slightly too optimistic for that player on average (due to bias from taking the maximum of a large number of noisy values).

Also, you may be looking for an option like "winrate_always_black" or something like that in the config. I *think* Lizzie by default shows the winrate from the perspective of the side to move rather than only from black's perspective, which if that's the case and I'm not just misremembering, is a really poor default. It is entirely inconsistent with the winrate you'd want to graph, and it makes it far more difficult to understand how the value changes as you play a move because rather than simply observing whether it went up or down, you instead have to mentally subtract from 100% every time the side to move changes. It would be great if the Go world standardized on showing black's winrate and score only, the same way that Chess has mostly standardized on reporting bot evaluations from the perspective of the first player to move in Chess.

I have now managed to get q5Go running. The hidden section gives the installation
details, mainly for my own use if the work needs to be repeated on a new PC or
with a new version of Debian. The rest of this post is a mixture of questions about q5Go
and an attempt to compare it to Lizzie, pending a decision on whether to switch.
At first glance, q5Go has a lot of functions and is therefore more complex than
Lizzie and harder to understand. I am also mourning Lizzie's keyboard
shortcuts. For example, Lizzie's space bar switches analysis on and off,
Ctrl-0/Ctrl-1 changes the analysis engine and 'a' starts an analysis by
prompting the user to specify the number of playouts. This is much more
convenient than working through the menus in q5Go, especially since I don't
understand them yet.

After running an analysis, I tried saving. (It was not clear what saving meant
in q5Go, which is why I wanted to try it.) q5Go said "Unable to save" without
saying why. The next day, following a reboot, saving always worked. I have no
idea what changed. The "Export" menu item also failed, but I have not retried it.

q5Go saves extra data using "QKGV" tags, which answers the question
whether "LV" is a de facto standard. The structure of the data in "QKGV"
tags seems very different but I haven't studied it.

I tried to find out whether one program analyses faster than the other. Both
programs were set to analyse the same game from the same point using KataGo
with the same model and settings, AFAICT. The results were confusing and
inconclusive.

In 2 minutes, Lizzie managed to get 38.8k playouts from KataGo. g5Go managed
11.4k visits. I don't know whether visits and playouts are the same things,
and it is possible that the report taken from q5Go is only for visits to the best
move. The GPU temperature was similar in both cases. The only definite
finding is that Lizzie uses more CPU time, distributed evenly over the
available four cores, whereas g5Go mainly uses one core.

Measurement is harder with g5Go because 1) I don't really know the right way to
stop the analysis - the GPU often continues to run after I think it should have
stopped - and 2) the analysis detail disappears from the screen when I do
(try to) stop it. It would be better to analyse a whole game using both programs,
but that would be time-consuming and boring. Also, the two programs use different
methods to control the analysis: Lizzie limits the number of visits but q5Go limits
the time per move.

Since the only stupid question is the one you did not ask, here goes.

Q1. The attached image shows a screenshot from an analysis run. The assessment
graph at the bottom left is too small to be useful even with g5Go using all the
1920x1080 screen. (The attachment was shrunk for posting.) The
screenshots from Lizzie at the head of this thread are of a useful size. Can
the graph in q5Go be made bigger? There are lots of examples of big graphs
in the help, but it is not clear how to access them.

Q2. Lizzie's assessment graph shows the percentage win-rate, the points lead
and the blunders on the same axes. This is often cluttered but it is useful.
Can g5Go show more than one thing at a time?

Q3. When Lizzie saves an SGF file, she saves all the analysis inside LZ tags.
One can then reopen the SGF file later and review the analysis without
repeating it. Is such an off-line review possible with q5Go?

Q4. When analysing in batch mode, where does the output show up? I couldn't
find anything. Maybe this problem was related to the "unable to save" problem.

Thanks for your patience.

Edited for clarity by thirdfogie, 31 March, 2022.

thirdfogie wrote:In 2 minutes, Lizzie managed to get 38.8k playouts from KataGo. g5Go managed 11.4k visits.

If the KataGo executable and options are the same, performance should be identical. It's probably a reporting difference.

Can the graph in q5Go be made bigger? There are lots of examples of big graphs
in the help, but it is not clear how to access them.

You can drag the separators, or even drag the entire graph to a different position in the window. You can then also save the layout. When opening a window it'll try to restore the default layout; if you then add more panes (for example by switching on the win-rate graph if it wasn't open already) you can press "L" to reload the layout for the new set of panes.

Q2. Lizzie's assessment graph shows the percentage win-rate, the points lead
and the blunders on the same axes. This is often cluttered but it is useful.
Can g5Go show more than one thing at a time?

I don't quite know what Lizzie does, but it doesn't sound like something q5Go can do at this point. You can switch the graph between score and winrate mode.

Q3. When Lizzie saves an SGF file, she saves all the analysis inside LZ tags.
One can then reopen the SGF file later and review the analysis without
repeating it. Is such an off-line review possible with q5Go?

Scores and winrates are saved. If you're doing live analysis you have to add variations yourself if you want to save them (shift-click or middle-click). You can also use batch analysis to make a file that has the all the top lines at every move.

Q4. When analysing in batch mode, where does the output show up?

There's a button with a "new board window" icon below the job queue.

Hi bernds, thanks for all the explanations and help.

I have learnt more about the differences in CPU usage between Lizzie and q5Go
when running the same analysis. The attachment shows the greater CPU usage by
Lizzie (on the left) compared to q5Go (on the right). Running the Linux "top"
utility explained it all: both programs run KataGo at about 104% CPU load, but
Lizzie updates the screen much more: Xorg runs at 70% CPU with Lizzie but
only at 10% to 30% with q5Go.

Another difference is that Lizzie re-uses saved analyses more than q5Go. For
example, if I quit Lizzie then restart it, open the same file and start
analysis at the same move, Lizzie somehow remembers the previous analysis and
restarts from that point. Lizzie even does this if the SGF file was deleted
and replaced by a fresh copy from an archive somewhere. q5Go always restarts
its analysis from the beginning, at least in my hands. This difference
probably explains my initial impression that Lizzie was faster.

The aggressive cacheing by Lizzie bedevils any attempt at detailed speed comparison.
The main conclusion is that GPU performance is not the only factor in analysis
speed. Are you reading this, Robert Jasiek?

I will now go back to using Lizzie, and work on the original project,
which was to extract useful textual comments from SGF files created
by KataGo. I had hoped to find a ready-made tool, but it looks like
I will have to write something myself.