Announcing GoReviewPartner - v0.15 with Pachi and PhoenixGo

[pnprog]

Hi everybody!

Some of you may have noticed from this thread in the computer go sub forum that I was working on a tool to help using Leela (the bot from the excellent work of Gian-Carlo that everybody knows). I had opened a GitHub project and kept working on it for some time.

In fact, I had arrived to a point where I though I should announce it to the community. And that's when L19 went down for two weeks... At first I waited, then moved to some other fun Go/Leela related project.

And today I was showed a similar project on Reddit, so it prompted me to finally make this announcement.

*edit: screen shoot*

v0.4.png (91.4 KiB) Viewed 48983 times

So here it is, you can find a v0.2 release on the Github project. This tools requires to have both GnuGo and Leela working on the computer. It offers two features:

• The first feature "Run *.sgf analysis" will use Leela to evaluate all positions in the provided *.sgf file. It will create a *.rsgf file (for "reviewed sgf file") made of your game an all the variations proposed by Leela for each moves. For each variation, it adds the win probability of Leela. Also, for all position, it will add Gnugo score estimation. The result is a very big, but valid, *.sgf file, but I think it's hardly usable from a standard SGF editor. One parameter that can be adjusted is the time per move that Leela should spend evaluating the position. I typically use 30s or 60s to review my own games, so the analysis time is quite long... But I am free to do something else during that time.
• The second feature "Open *.rsgf for review" is there to make reviewing the result of the analysis as simple as possible. It shows two gobans: on the left the game as it went ; on the right, the different propositions from Leela. Move your mouse pointer over the different positions and the sequence is displayed. The score estimation is displayed as well. In case you want to explore further, you can open that position in a third board and play out variations alone or against Leela/Gnugo.

Some more informations:

• I could not try it a lot under Windows, but it should be ok.
• Version 6 to 8 of Leela should work, although I mainly used V8 so far
• I have not GPU card, so I only tried the CPU version. But I guess the GPU one should work the same way.
• Normally, different sizes of Goban work as well (<- i did not try that in a while)
• The leela_analysis.py part can be ran from a terminal with a *.sgf filename as argument. And so can be used in an script. I only play online during luch time at work, and use my software combined with this script to have my game analysed during the afternoon so that it's ready to review when I am back home.

How to install:

• For Linux users, the best is to download the python source code from Github then run the main.py file using python. In the setting part, you will have to indicate the command line for Leela and Gnugo. For example, I use /home/pnprog/Leela080GTP/leela_062_linux_x64 --gtp --noponder and gnugo --mode gtp
• For windows users, I have packaged a *.exe file with Leela and Gnugo, you can download it from there. Once it is downloaded, unzip and run the GoReviewPartner.exe

Just a few things I want to add:

• I am just an amateur go player and an amateur programmer, so do not take that software too seriously. I am just interested in experimenting with what computer could bring to us and how learning Go could change in the future.
• All the kudos should go to Gian-Carlo, Leela is the real thing, not my software. Also big thanks to the author(s) of the gomill python library.
• I won't have any problem if somebody else borrow some of the ideas I put in this software and make something better (<- that should not be that difficult in fact), even for some commercial software. I am just fond of Go and programming, so this is my way to contribute to Go. But the source code is licensed under GPL3, keep that in mind if you want to reuse some part of my (ugly) code.

For now, I am interested in Rayon, the open source Go bot that seems quite strong. I haven't been able to make it work on my computer, but if I succeed, I may add support for this bot into my software. If you notice some bugs, or have some idea for improvement, let me know in this thread.

Have a good day!
pnprog

[pleiade67]

Thank you very much for this awesome review tool. Tested and running smoothly under windows (works also for 9x9 games).

Wish list:

- an option to select a given subset of moves to analyse. Focusing on the critical part of a game with a deeper analysis might be useful:
see for instance (!) viewtopic.php?p=217103#p217103

- support for other bots (maybe in parallel) would be great : Ray, Fuego, pachi. Useful for overcoming the limitations or blunders of a single bot. Again: viewtopic.php?p=217103#p217103

Best Regards,

[mbrezu]

This is a great tool! Many thanks for making this available.

I second the request for more focused analysis.

I also have a minor enhancement request: I tried your Windows archive and it worked great. I tried to replace Leela 0.8 with Leela 0.9 and something breaks (I get a Python exception about releasing an unlocked lock).

+ [['R14', 'R14 O17 P14', 55.06], ['C13', 'C13 C17 C18 C16 G17 E18 D18 E17 E16 D
16 F16 B18', 51.27], ['D12', 'D12 G16 E15 E14 F15 D16 F17 C17 D18', 50.65], ['C9
', 'C9 C17 C18 C16', 50.06], ['C15', 'C15 C14 C16 D13 F17 C8 B3 D6 D5', 49.63],
['E15', 'E15 E14 E16 D13', 49.55]]
55.06%/44.94% <type 'exceptions.ValueError'> leela_analysis.pyc 242
'R' is not in list
releasing lock
leaving thread
Exception in Tkinter callback
Traceback (most recent call last):
File "Tkinter.pyc", line 1537, in __call__
File "Tkinter.pyc", line 587, in callit
File "leela_analysis.pyc", line 276, in follow_analysis
error: release unlocked lock

[pleiade67]

Tested on Mac OSX.

Also have a python error after some moves:

====move 30 ['g3', 'g3 c2 h2 h4 h3 j3 j2 e1 f1 c6 b6 d1 f2 c3 j4 j5 j3', 82.08] ~ g3
17.92%/82.08% move 31/53 W+11.0 leela play black
====move 31 ['g2', 'g2 g3 h3 b2 f2 e2 a2 h4 h2 g4 c8 j3 c7 c6 b8 a8 b3 b1 d1 e1 d2 j4 f1 c2', 37.4] ~ g2
37.4%/62.6% <type 'exceptions.ValueError'> leela_analysis.py 242
invalid literal for int() with base 10: 'ass'
releasing lock
leaving thread
Exception in Tkinter callback
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 1537, in __call__
return self.func(*args)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 587, in callit
func(*args)
File "leela_analysis.py", line 276, in follow_analysis
self.lock2.release()
error: release unlocked lock

Nota Bene: relaunched the review and was not able to reproduce this error (assume there is some random seed in the Monte Carlo simulation)

Best,

[pleiade67]

Additional questions and suggestions:

- what is the meaning of the blue / red color for the letters representing the different variations ?
Would be nice to have two different color when going from win (winrate > 50% --> blue) to loss (winrate > 50% --> red) for the current color (black or white).
- would it be possible to display the number of playouts (nodes ?) leading to the displayed variation and to the estimated winrate ?

Thanks again

[pnprog]

Dear both,

Thanks for trying the software, it's a relief that is (sort of) works under windows.

Locking at the logs you posted, my guess is that the software does not handle some cases when Leela chose to pass of resign. In fact; it normally handles both cases, but when Leela only proposes one move during the follow up for the main line of play (a book move, or an answer to a forcing move) the program goes one step deeper by asking Leela to play the following move as well, and repeat that several times if needed.
And I guess that if "Pass" or "Resign" appear during at that time, the program try to interpret it as game coordinates [one letter][one number] (ie D16) and fail. That's would be the meaning of 'R' is not in list ("R" from "Resign") and invalid literal for int() with base 10: 'ass' ("ass" as in "Pass").
The good news it that's should be an easy fix. I guess you were trying with 9x9 games to get this sort of answer from the early moves.

I also have a minor enhancement request: I tried your Windows archive and it worked great. I tried to replace Leela 0.8 with Leela 0.9 and something breaks (I get a Python exception about releasing an unlocked lock).

If my guess above is correct, then it should not be an issue with Leela 9 I think, although for some reason, Leela 9 won't run on my computer so I cannot be sure. I will borrow a computer from my friend to try that.

- an option to select a given subset of moves to analyse. Focusing on the critical part of a game with a deeper analysis might be useful:
see for instance (!) http://www.lifein19x19.com/viewtopic.php?p=217103#p217103

I guess I will propose something similar to what is used when printing a document from the computer, where one can print all pages, or only a subset like "1,2,10-12" something like this.

- support for other bots (maybe in parallel) would be great : Ray, Fuego, pachi. Useful for overcoming the limitations or blunders of a single bot. Again: http://www.lifein19x19.com/viewtopic.php?p=217103#p217103

This is the direction I want to go in fact. The specificity of Leela is that it exchanges gtp commands from one channel (standard input/output) and drops a lot of useful information in a separate channel (standard error). GnuGo won't say anything on the standard error channel, but has an enhanced set of gtp commands to ask for several things, and I plan to use that in the future.
I need to check what Ray/Fuego/Pachi have to offer (maybe I should get in touch with their authors). One way is to have them play successive moves for each alternative line of play, but it would be quite time consuming. In fact, I did something like this with The Many Faces Of Go in the past, it was a big mess! (because that bot has not gtp mode)

Maybe in a few years, there will be new standards for the Go Text Protocol or for the SGF format that will be more "game review oriented". But for now, we have to do with what is available.

Now, trying to avoid one bot blunder using another boot is going to be quite difficult. One may would be to ask for the second bot it's best play, and if not already in Leela proposals, have Leela play that move and evaluate it. But to use your Shikshin and Leela game example, what Leela move number would you consider as a blunder? From the moment that Shikshin managed to create this game deciding ko, Leela has no choice but to try to play it and look for the biggest threat on the board. It's a game deciding ko, so Shikshin would not reply to it anyway. Was the real blunder the move 172 (move 171 was sente and that Leela ignored it) ? Would Fuego foresee the sequence?

- what is the meaning of the blue / red color for the letters representing the different variations ?

In fact, on the right goban are all variations from Leela, from the best winning rate (A) to lowest winning rate (B, C, D). If there is one in blue, it's the one played during the game. It's an immediate way to see if one of the Leela moves has been played in actual game, without having to look at the left goban.

Would be nice to have two different color when going from win (winrate > 50% --> blue) to loss (winrate > 50% --> red) for the current color (black or white).

Something similar should be possible and easy (and useful), I will think about that.

- would it be possible to display the number of playouts (nodes ?) leading to the displayed variation and to the estimated winrate ?

This should not be a problem.

Tested on Mac OSX.

I curious about that, I did not notice there was a Mac OSX version.

Just now, I found a new bug with OGS games I introduced just before releasing the software (OGS marks the first stone as a sort of handicap stone in the SGF file and GnuGo dislikes it...) so I will fixe that and implement some of the above proposals quickly. I plan to release an update this weekend.

best regards,

[pleiade67]

Thank you for your detailed reply.

For Mac OSX: there is an OSX version of the GTP engine which is packaged in:
https://www.sjeng.org/dl/Leela080GTP.zip
https://www.sjeng.org/dl/Leela090GTP.zip
(along with the win and linux version).

The only issue I had with the OSX version was the location of the boost library, which should be copied in the following directories:

libboost_program_options-mt.dylib --> /usr/local/opt/boost/lib/.
libboost_thread-mt.dylib --> /usr/local/opt/boost/lib/.
libboost_system-mt.dylib --> /usr/local/opt/boost/lib/.

Best

[pnprog]

Hi!

The only issue I had with the OSX version was the location of the boost library, which should be copied in the following directories:

libboost_program_options-mt.dylib --> /usr/local/opt/boost/lib/.
libboost_thread-mt.dylib --> /usr/local/opt/boost/lib/.
libboost_system-mt.dylib --> /usr/local/opt/boost/lib/.

Is this is a general issue for Leela running on MAX OSX? or is this an issue with Python? (what is that boost library?) In the second case, do you mind if I put this information somewhere in the (yet to be done) documentation? I myself does not possess a Mac OSX, but it could be useful to others.

By the way, I just fixed the issue that you mentioned (...I hope I have). GitHub source code has been updated and should be working fine now. I will make a new bundle version this weekend.

Best regards,
pnprog

[pleiade67]

Hi!

The only issue I had with the OSX version was the location of the boost library, which should be copied in the following directories:

libboost_program_options-mt.dylib --> /usr/local/opt/boost/lib/.
libboost_thread-mt.dylib --> /usr/local/opt/boost/lib/.
libboost_system-mt.dylib --> /usr/local/opt/boost/lib/.

Is this is a general issue for Leela running on MAX OSX? or is this an issue with Python? (what is that boost library?) In the second case, do you mind if I put this information somewhere in the (yet to be done) documentation? I myself does not possess a Mac OSX, but it could be useful to others.

By the way, I just fixed the issue that you mentioned (...I hope I have). GitHub source code has been updated and should be working fine now. I will make a new bundle version this weekend.

Best regards,
pnprog

Thanks !

For the Mac OSX version of leelaGTP: it requires the boost library (this is explained in the README file of leelaGTP and is not related to python). Boost is a standard multi-purpose C++ library: https://en.wikipedia.org/wiki/Boost_(C%2B%2B_libraries)

LeelaGTP readme for macos:
You need the boost libraries installed. The recommended way is to use Homebrew and do a "brew install boost".

Boost was already installed on my machine but not in the directory required by the leelaGTP Mac executable. Thus I had to create the directory: /usr/local/opt/boost/lib/ and copy the necessary boost files listed above in it. This is only necessary if boost is not at the ‘right' place (depends on the method used to install it on Mac). This could indeed be useful to others...

[Javaness2]

I tried the software under windows as well, and wow it worked. I would like to echo the feature request of being able to set the program to remove just a specific range of moves. In reality I think that it is very unlikely that somebody will want to review ALL the moves in a game. Nice work.

[mb76]

Thanks for this tool! I want to report a bug when analyzing a .SGF with a handicap setting:

****
b handicap stone at 3 3 => d4
b handicap stone at 3 9 => k4
b handicap stone at 3 15 => q4
b handicap stone at 9 3 => d10
b handicap stone at 9 9 => k10
b handicap stone at 9 15 => q10
b handicap stone at 15 3 => d16
b handicap stone at 15 9 => k16
b handicap stone at 15 15 => q16
Exception in Tkinter callback
Traceback (most recent call last):
File "Tkinter.pyc", line 1537, in __call__
File "main.py", line 41, in launch_analysis
File "leela_analysis.pyc", line 83, in __init__
File "leela_analysis.pyc", line 351, in initialize
File "gtp.pyc", line 170, in set_free_handicap
IndexError: list index out of range
****

The .SGF's are included in this post. I use the Windows version.

[pnprog]

Hi everybody!

I have been quite busy implementing pretty much everything that was discussed in the thread so far.
Main list of changes is:

• Leela 0.9 is now working fine
• Possibility to perform the analysis only on subset of move, ex: "1-20,15,30-60" would analyses moves 1 to 20, plus move 15, plus move 30 to 60
• The number of playouts (nodes) is indicated
• Blue color for win rate>50%, red otherwise (black letter indicate the move played in the game)
• Plenty of bug fixes (including the handicap stones bug), and small interface improvement

One significant change: in the previous version,it used to sort the variations (A, B, C, D...) based on winrate probabilities from Leela. It now follows the same order as Leela. So sometime, some low ranked variation (D, E, F...) can have a higher winrate that the first variation A. It typically happens when Leela did not run enough simulation to check this possibility and ensure it is really good. So the blue/red color may be a little confusing, compare the number of variations to be sure. Also, no winrate value is indicated for "bookmoves", fuseki moves that Leela plays without thinking.

Windows users will find the bundled download (version 0.3) there, it includes Leela 0.9
Linux/Mac users should download the sources directly from the Github project.
Let me know if it works as expected.

Next:

• I guess I should make a bit of documentation (user manual)
• One feature I want to add is the possibility to have the variation sequence played move by move by turning the mouse wheel. Currently, many variations from Leela go too far for me to follow without having to open the position to play it manually This is especially true in 9x9
• I should add the coordinate on the goban
• Maybe confronting Leela with Gnugo's move to see if it can help Leela avoid blunder

Best regards,
pnprog

[pleiade67]

Hello,

tested the new GoReviewPartner (0.3) version for Mac. Thanks for the new features and the bug corrections. Definitely a great tool .

I have tested it on the Leela-Shikshin game (194 moves game, with the leela blunder at move 191) and observed a strange behaviour on the last (189-194) moves.
The settings were:
Analysis time per move: 120
Review Fuzzy Stone: 0.2 (what is the meaning of this parameter ?)
Moves Analysed: 160-194
The sgf and resulting rsgf may be dowloaded here:
http://dl.free.fr/b0jpDpDVh

- From move 189, the number of playouts were dramatically reduced (going from +200000 down to ~ 1000 or less)
- The last moves (191-194) are not displayed properly in GoReviewPartner (?)

Maybe this is a 'bug' in leela and the reason for the blunder ?

Additional wish list :
- Buttons for +10 , -10 moves (a slider would also be great, but it is less straightforward...)
- Goto a given move.

Best regards,

[roy7]

- From move 189, the number of playouts were dramatically reduced (going from +200000 down to ~ 1000 or less)

Leela will make snap moves if the Net Prob % of the move with highest Net Prob % is 8x greater than the Net Prob % of the 2nd highest Net Prob % move. The idea is if it's super likely a human pro makes the move, Leela should just do it as well. But in unusual board states the policy network can be confused and rate high Net Prob % for moves a human would never do in a life/death situation. This is what leads to the "blunder" moves, I think.

[pleiade67]

Thank you for the explanation!
How do you know these details ? Are you the author of Leela ?
Or is there a technical paper available ?