Announcing GoReviewPartner - v0.15 with Pachi and PhoenixGo

[pnprog]

For whatever reason, Pandanet assigns a negative komi to black rather than a positive komi to white which throws Leela for a loop toward the end of the game. In this example, Leela considers the game to be over around move 31 and would resign as black even though I actually won by 4.5 points during the actual match. This makes reviewing the entire match rather difficult as the last 1/3 of my moves aren't evaluated. I don't think I have the option of editing the komi when creating the game on Pandanet, but I've tried changing the komi when selecting the file to review in GRP (removing the negative, doubling the given value, selecting a positive value like 3.5 used on OGS) but none of my mods have worked; Leela still thinks black loses and resigns.

I just had a look at it, and it is definitively a GRP bug, not a issue with Leela handling negative komi. I opened the game with GoGui and asked Leela to play some end game moves, and Leela sees black winning by 4.5pt, with 100% confidence.

So it is a GRP bug, I don't know yet what causes this bug (seems like the black and white probabilities are switched). Anyway, thanks for reporting, this is helping me a lot!

[sleepyEDB]

Awesome! I'm glad this was well-received and that I could be of help.


sleepy

[pnprog]

AQ 2.1.1 (GPU with Cuda) does not handle handicap games properly.
For example suggestion moves 8/233(G), 15/233(C), 28/233(C) and 64/233(E) overlaps the already placed handicap stone (RGSF).

For information, I fixed this bug in the development version. The bug was also affecting Ray.

[pnprog]

I will see what's doable, ideally, the best would be to recognize those three possibilities:

• Even game that start with empty board or black first move
• Handicap games that start with empty board or white first move
• Games with already placed stones of both colors (like in the very ancient Chinese way to play, with cross fuseki already in place). Currently, GRP does not manage that.

I implemented that in the development version, I am quite satisfied with the solution:

• GRP first checks how handicap stones are declared by the SGF file
• GRP then checks how many stones are already played on the SGF first move
• GRP then tries to play a same number of free handicap stones as declared by the SGF file, using existing black stones on the board, then play the remaining stones (including possible white stones) as conventional moves

This way, even if the SGF starts with the board position at move 45 of a handicap game, the analysis should be ok.

[pnprog]

Awesome! I'm glad this was well-received and that I could be of help.

In fact, those seemingly unimportant bugs quite often hide deeper and more vicious bugs that are hard to detect and recreate. So chasing bugs that appear in exotic situations is in fact useful.

And this may be the case for that bug. I have been scratching my head all the morning with it:

• GRP gives white loosing during the game
• GoGui gives black winning at the end of the game (and globally disagree with GRP when whatever position I would jump at to check)
• When I ran Gogui and GRP in parallel, checking Leela's output on both software, moves by moves, then GoGui agrees with GRP that black is loosing

That was really confusing, because I could see exact same board postion (Leela even provide some sort of hash for a board position) and very different win rate. I mean, I really started to doubt my own sanity at some point

Could it be that previous win rate calculation affected the following win rate calculation? To check that, I modified Leela's dedicated code in GRP so that, instead of undoing moves the "normal way", she undoes moves by complete board reset and replay to previous position. That's the way AQ and Ray work already because they don't accept the standard undo command.

And then, finally, GRP agrees with Gogui that black indeed is winning this game:

Win rate before modification:

before.png (8.96 KiB) Viewed 17097 times

Win rate after modification:

after.png (10.65 KiB) Viewed 17097 times

So at this point, I believe this is an issue with Leela, I will check a bit further then probably contact Gian-Carlo Pascutto for this one.

[ez4u]

Thanks very much for working on this so hard!

[gdgcb]

Hi
Very appreciate your sharing and it works quite well for analysing games.
Do you have any plan to release an updated version that can support ELF OpenGo hash?

[sleepyEDB]

So at this point, I believe this is an issue with Leela, I will check a bit further then probably contact Gian-Carlo Pascutto for this one.

Wow, awesome stuff! Thank you for your hard work on a great program...where's the donate button?!


sleepy

[pnprog]

So at this point, I believe this is an issue with Leela, I will check a bit further then probably contact Gian-Carlo Pascutto for this one.

Wow, awesome stuff! Thank you for your hard work on a great program...where's the donate button?!

Haha, no donation needed, working on GRP is already quite rewarding

I was able to reproduce the issue on another 9x9 game, as well as an 19x19 game. From what I understand, this bug appears under the following conditions:

• A negative komi is used
• Leela must be asked to think over the game first move

So this bug went unnoticed because reversed komi is rarely used (even more on 9x9) and on 19x19, users of GRP usually let Leela use her joseki book move, so Leela does not really analysed the first move. When those two conditions are met, the MonteCarlo simulation seems to go wrong for the following rolls.

I sent my finding to Leela's author:

Thanks, this should be enough to investigate the problem.

I am not actively working on Leela 0.x right now, as Leela Zero is still
taking up all of my time, but if I ever get back to it, I will try to
fix this bug.

I will patch GRP in the next version so that Leela use the "clear board then replay up to previous move" procedure instead of conventional undo. I will do the same for all bots, as it will make the code cleaner.

In the meantime sleepyEDB, for further analysis, tell GRP to start the analysis at move 2, it avoids that bug.

[pnprog]

Hi!

Hi
Very appreciate your sharing and it works quite well for analysing games.
Do you have any plan to release an updated version that can support ELF OpenGo hash?

Hum... let me see my check list:

• Is ELF strong? =>
• Is ELF Open Source? =>
• Is ELF working on Linux? =>
• Are the ELF developers friendly toward the Go community =>

I can look as much a much as I can, I just see not reason not to implement support for it, so yes, I will add support for it.
The only aspect that could delay it is potential difficulties to run it on my (pretty deprecated) computer. So it will just be a matter or time.

[dfan]

The latest version of Leela Zero (as of yesterday) can run with the ELF-format weights, so I suspect that will be the easiest way for anyone to review games using the brains of ELF Open Go.

[gdgcb]

Hi!

Hi
Very appreciate your sharing and it works quite well for analysing games.
Do you have any plan to release an updated version that can support ELF OpenGo hash?

Hum... let me see my check list:

• Is ELF strong? =>
• Is ELF Open Source? =>
• Is ELF working on Linux? =>
• Are the ELF developers friendly toward the Go community =>

I can look as much a much as I can, I just see not reason not to implement support for it, so yes, I will add support for it.
The only aspect that could delay it is potential difficulties to run it on my (pretty deprecated) computer. So it will just be a matter or time.

As the GO program is getting far more stronger than human, I find no meaning to play with computer
But can help me improve go skills by reviewing games.
Your program is quite helpful. Thanks very much

[Dontbtme]

Hi
You said my explanations weren't clear last time, plus I have a better idea of what would be (in my opinion) the best/easiest way to propose the settings, so let me try again:
_ First, when you analyze a game, it records EVERYTHING: every variation, every data into the rsgf. Next, if you wanna review the rsgf, a window pops up asking you what you wanna see: how many variations per move, from one or two color's point of view, every mistake or only those with more than a given win rate drop, etc. (the window would have the settings used last time, so you would either confirm or change them)
What that means is that you can review your game one way the first time, then another the next time, USING THE SAME RSGF: the settings didn't change what was recorded, only what was showed during a given review session.
_ Second, when you convert the rsgf, since everything was recorded on it, a window would also pop up asking you what you want in the sgf (it would have the settings used last time, so you would either confirm or change them). Like for the review, you can ask to create an sgf from one or two color's point of view, what variations to keep and so on... plus what kind of data you want to show up as comments. In my case, I only keep the win rate shift between my move and the variation. For example:
"White Value Network win probability:
• before c6: 54.48%
• after c6: 52.43% (-2.05pp)"
Like before, you could convert your rsgf one way at first, and then another (for whatever reason) the next time, since the rsgf got EVERYTHING recorded anyway. For example, you could make a separate sgf for each color, or you could have an sgf with only one variation and another one with many, or one with big mistakes only and another one with every mistakes, etc. using THE SAME RSGF.

Here's the kind of sgf I edit myself nowadays based on the rsgf (I couldn't put it here directly, apparently, so here's the Eidogo link):
http://eidogo.com/#tNq4zS60
That's the way I like my sgf (clear and simple) but obviously that's only one example: someone else could chose different settings.

What do you think?

[sleepyEDB]

Haha, no donation needed, working on GRP is already quite rewarding

Fair enough, but the offer stands should you change your mind!

In the meantime sleepyEDB, for further analysis, tell GRP to start the analysis at move 2, it avoids that bug.

The next time I run into negative komi I will definitely do that. Thanks!


sleepy

[MikeKyle]

Thank you so much for all your work on this brilliant program.

I'd like to second dontbtme's last post: it would be really brilliant to do all the analysis, but then be able to choose the depth of review when you actually start the reviewing. Sometimes I want to see all the interesting sequences the AI would expect in my games but at other times I just want to concentrate on understanding my big mistakes. To have everything stored but then be able to filter it to a set number of variations or a certain mistake threshold would be really ideal.


To volunteer a more tame idea - would it be possible to allow batch analysis of a few sgf files? I guess the file picker could just allow you to pick more than one file and then it would queue up the files for processing?