Announcing GoReviewPartner - v0.15 with Pachi and PhoenixGo

[pnprog]

Hi everybody!

Here come v0.7 !

Admittedly, I am late compared to my initial planning, but here it is, this version supports the zakki version of Ray (aka "Rayon") the most promising Opensource bot.

And this is more or less the only new feature of this version. Other notable change is that it now proposes to directly open the rsgf file for review at the end of the analysis. It also improves (decreases) the size of the rsgf file by removing duplicated comments.

I prepared the usual "Windows bundle", but because it now includes Ray, its size is much bigger. For other OS, it is best to download the code on GitHub and run it as an usual python script.

Unfortunately, I was not able to compile Ray on Linus so far. If somebody succeeds to do that for version 4.2.5, then I would be happy to offer a place to store the binaries online for everybody to download.

I did not test the Ray analysis thoroughly, at least not the gpu version (because I don't have one), so i will need your help to try that. The default settings in the Windows bundle run both Leela and Ray with no gpu, so make sure to change that if you want to try with gpu.

EDIT: For Ray to run, it needs CNTK to be installed. CNTK can be downloaded there: https://github.com/Microsoft/CNTK/releases

[Tiktaalik]

Thanks for the lovely tool. I enjoyed using it with Leela.
However, I could not run it with Rayon for some reason. I got the following error message:

-----

bot selection: Ray
white only
========= move selection
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204]
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1540, in __call__
return self.func(*args)
File "/home/nikoryu/Go/programs/goreviewpartner/toolbox.py", line 337, in start
self.popup=RunAnalysis(newtop,self.filename,move_selection,intervals)
File "/home/nikoryu/Go/programs/goreviewpartner/ray_analysis.py", line 39, in __init__
self.initialize()
File "/home/nikoryu/Go/programs/goreviewpartner/ray_analysis.py", line 291, in initialize
alert("Could not run Ray using the command from config.ini file (\""+Ray_command_line+"\")")
NameError: global name 'Ray_command_line' is not defined

-----

Do you have any idea how to fix this?

[pnprog]

Hi!

Thanks for the lovely tool. I enjoyed using it with Leela.
However, I could not run it with Rayon for some reason. I got the following error message:

This is apparently an issue running Ray. I forgot to mention that Ray needs CNTK to be used. CNTK can be downloaded there: https://github.com/Microsoft/CNTK/releases

Please have a try after installing CNTK and let me know.

[Raspingbard]

I keep reading where GoReviewPartner runs on the Mac, but all I see is "For Linux/Mac, it is best to get the source code from Github." What do I do once I download the source code? Could someone give me step-by-step instructions once that's done?

(I already have the Leela GTP engine running under another GUI on the Mac.)

Thanks

[Tiktaalik]

This is apparently an issue running Ray. I forgot to mention that Ray needs CNTK to be used. CNTK can be downloaded there: https://github.com/Microsoft/CNTK/releases

Please have a try after installing CNTK and let me know.

Actually, I had already installed CNTK and was running Rayon using GPU.
Anyway, I found that the main problem was that I used a wrong path to ray binary..
I compiled a couple of different versions of Rayon and then moved some of them to my archive directory, so I somehow mixed those up and used a non-existing path..

A minor thing I would like to inform you is that there was a typo and a syntax error in one of your scripts.
Specifically, in line 291 of your "ray_analysis.py", "Ray_command_line" should be "ray_command_line".
Also in this line, it seems you cannot simply use "+" sign to join error message and the ray command line, which is a list. I just borrowed a code you used in the "leela_analysis.py" and modified it as:
alert("Could not run Ray using the command from config.ini file:\n"+" ".join(ray_command_line))
This enables the GUI pop-up as you intended.
Anyway, the fact that this part of the script is triggered means that your Ray program is not running properly, so this is really a minor thing, but just so you know..

[pnprog]

This is apparently an issue running Ray. I forgot to mention that Ray needs CNTK to be used. CNTK can be downloaded there: https://github.com/Microsoft/CNTK/releases

Please have a try after installing CNTK and let me know.

Actually, I had already installed CNTK and was running Rayon using GPU.
Anyway, I found that the main problem was that I used a wrong path to ray binary..
I compiled a couple of different versions of Rayon and then moved some of them to my archive directory, so I somehow mixed those up and used a non-existing path..

A minor thing I would like to inform you is that there was a typo and a syntax error in one of your scripts.
Specifically, in line 291 of your "ray_analysis.py", "Ray_command_line" should be "ray_command_line".
Also in this line, it seems you cannot simply use "+" sign to join error message and the ray command line, which is a list. I just borrowed a code you used in the "leela_analysis.py" and modified it as:
alert("Could not run Ray using the command from config.ini file:\n"+" ".join(ray_command_line))
This enables the GUI pop-up as you intended.
Anyway, the fact that this part of the script is triggered means that your Ray program is not running properly, so this is really a minor thing, but just so you know..

Good job in solving your issue and finding that bug in my python code.

I am currently travelling until end of July, "away from keyboard", so I cannot work on GoReviewPartner for now. I will fix that when I am back.

[Tapani]

My other idea is to have a graph in the app, below the left board for example, that will show you graphically where the game went wrong.

If anyone else is interested in something like this - the following Linux shell-script will read the rsgf file and plot a graph over black's win probability.

Code: Select all

#!/bin/sh

if [ X$1 = X -o ! -f $1 ]
then
	echo Need rsgf file as argument
	exit 1
fi

grep -e '^C\[Move' -e '^Leela' $1 | tr '\n' ' ' | tr ']' '\n' | cut -d ' ' -f 3,11 | cut -d '%' -f1 | grep -v Move > /tmp/g.dat
gnuplot -p -e "set yrange [0:100]; plot '/tmp/g.dat' with lines"
rm -f /tmp/g.dat

It is a 30 minute three-liner, so don't expect too much

Oh, and it only works with Leela's analysis output.

[djhbrown]

win% is as much a function of how bad opp's last move was as it is of how good your move was. a more meaningful indicator of where your play was deficient (according to your chosen oracle) is the difference between win% of your move and win% of your oracle's best move for that same position.

graphs are useful tools for seeing the relationship between two variables, so if you wanted to see how your performance behaves over time, a graph of win% over time would be a suitable instrument. on the other hand, colour coding of your moves would show you which of them were below par:

your move win% - oracle best move win%
< - 5% red
+- 5% green
> 5% yellow (you significantly bettered the machine)

5 is an arbitrary number; a more statistically meaningful one might be sd of the distribution of historical differences - then, over several games, you would discover whether or not you were improving. another way of judging that would be to glance at your rank graph, which will probably tell you that you are not because you fondly imagine that you only lost because you went wrong on one particular move, whereas it is your overall strategy and tactics and reading ability that are wanting because you are trying to reinvent the wheel, learning by experience of playing other weak players instead of reading a book and standing on the shoulders of wise (wo)men down the ages. alphago needed millions of trials to improve by trial and error and self-flagellation, what makes you think you are any different?

[Tapani]

win% is as much a function of how bad opp's last move was as it is of how good your move was. a more meaningful indicator of where your play was deficient (according to your chosen oracle) is the difference between win% of your move and win% of your oracle's best move for that same position.

graphs are useful tools for seeing the relationship between two variables, so if you wanted to see how your performance behaves over time, a graph of win% over time would be a suitable instrument. on the other hand, colour coding of your moves would show you which of them were below par:

your move win% - oracle best move win%
< - 5% red
+- 5% green
> 5% yellow (you significantly bettered the machine)

5 is an arbitrary number; a more statistically meaningful one might be sd of the distribution of historical differences - then, over several games, you would discover whether or not you were improving. another way of judging that would be to glance at your rank graph, which will probably tell you that you are not because you fondly imagine that you only lost because you went wrong on one particular move, whereas it is your overall strategy and tactics and reading ability that are wanting because you are trying to reinvent the wheel, learning by experience of playing other weak players instead of reading a book and standing on the shoulders of wise (wo)men down the ages. alphago needed millions of trials to improve by trial and error and self-flagellation, what makes you think you are any different?

Sorry if my quick-hack script offended you somehow. How about posting your code for those move-by-move oracle comparisons - it sounds cool indeed.

[Uberdude]

Crazy-sensei, the online analysis version of CrazyStone from Remi Coulom, displays mistake bars like this, showing how much your move differs from the bot's suggestion. I think it works nicely: see the red bars on https://www.crazy-sensei.com/?lang=en&l ... =47&move=0 for example.

[pnprog]

If anyone else is interested in something like this - the following Linux shell-script will read the rsgf file and plot a graph over black's win probability.

I just ran it and it works well
But the grep/cut command looks like black magic to me

Seems like the win rate graph is becoming more and more popular those days. Maybe I should consider adding one into GoReviewPartner.

I just had a look at the link to Crazy Sensei provided by Uberdude, it's quite similar to GoReviewPartner, the interface is nice.
Regarding the graph, it is described as:

Red bars indicate errors. Errors of Black are below the middle line, and errors of White are above.

I think it should read more like:

Red bars indicate where Crazy Sensie believes he could have find a better move

To build upon djhbrown comment, and if my understanding is correct, Crazy Sensei compares the win rate % of the players moves with that of its own move for each board position. Then, there are three cases:

• A better win rate is found, then a red bar is displayed
• The same move was played, or the win rate is exactly the same, no display
• A lower win rate is found, meaning that the player played a move that Crazy Sensei evaluates as better than it's own play...

The third case is apparently ignored by Crazy Sensei (yes, it makes sense from Crazy Sensei's author point of view ). But still it would be interesting (it would be fair) to display that as well.

I will consider that if I implement a graph into GoReviewPartner.

[djhbrown]

re graphs: my comment was a response to John Smith's suggestion. in it, i tried to explain that win rate graphs are misleading, in that they don't necessarily show you where you went wrong, since your opponent can be going wrong too.

my suggestion was, and is, that instead of a separate display, you colour in your stones on the board according to their "temperature" ie how hot (good) each move is as adjudged by your oracle; i chose red/green/yellow since it will work whether you are black or white (an alternative colour scheme could be red = significantly worse than oracle, green = better; otherwise normal stone colour). i imagine it might produce some interesting and possibly informative colour patterns across the board by the end of the game. grp could have a toggle button to switch colours on/off.

graphs are useful visual tools for comparing two variables, showing the distribution of one variable (such as prowess at Go, or blood pressure, or weight, or money in the bank) over another (such as space or time; eg you vs everyone else, or you vs yourself over time). graphs are handy for providing an oversight of things like trends, or variability, or distribution. they can correct basic misunderstandings, such as the notion of "middle class", a term which implies a normal distribution of income or capital across a population. it's a term which derives from intuitive notions that there are three kinds of people: rich, middle and poor. in US culture, it is tacitly assumed by economics commentators that most people are middle class - but nothing could be further from the truth! here is a graph of Australian household wealth in 2008; if you were to make one for US or indeed any country in the modern era, you would find a similar picture: a tiny fraction with most of the wealth, and all other class intervals (eg deciles) having virtually nothing by comparison, as shown by the flatline shape of the graph on the left, apart from its top end. so there are just two classes: them and us. there is no middle.

[pnprog]

Hi there!

So I have been playing around a bit with graphs in GoReviewPartner this week. Here is what it looks like so far:

As an example, I took one game from the Game Analysis section between negapesuo and wqz2017, ran the analysis at 20s/move on my old computer, and get that graph for player negapesuo (black):

chart.png (4.16 KiB) Viewed 21683 times

This is black player win rate, starting from a position higher than 50% (that was a 2 stones handicap game) with black managing to keep his advantage during game the until the final blunder at the end that cost him the game.

So you will notice red and green rectangles, here is what they mean:

• Red means that for that position, Leela believes she has find a better move than negapesuo. If that move had been played instead of actual game move, then the win rate would have been increased by the red quantity.
• Green means that Leela believes that negapesuo came up with a better move that Leela itself. If Leela's move had been played instead of the actual game move, then the win rate would have decreased by the red quantity

So red means margin for improvement, and green means better than Leela.

The very last move win rate, when added the red quantity, exceed 50%, meaning that this blunder was indeed a game reversal move.

A bar diagram may not be conveying this idea very well, I am not so sure, because it is not obvious that the red bar is higher than the grey bar, and that the green bar is lower than grey bar. If I replace the grey bars with a blue line, it would look like this:

chart1.png (4.41 KiB) Viewed 21683 times

But somehow, I don't like it that much.

You will notice this graph is only for one player, not both players at the time (it's becoming too confusing then). In fact, only half the moves are displayed (even moves for black player here, because handicap stones so white played first). And so GoReviewPartner would propose 3 graphs:

• The usual win rate graph
• The comparison between Leela/Ray and the black player (like the one I showed)
• he comparison between Leela/Ray and the white player

Then, I plan to add a few things to make it easier to use, like displaying detailed stats for each moves when the mouse is over the bar, and clicking on the bar would go directly to that move in the analysis windows.

[Dontbtme]

Wow, that's huge! Bravo!
I think one way you could change how you feel about the grey bars would be to just lighten their shade a bit so that the red would shine more by contrast...
Just an idea.
But it's looking promising already, so good job!

[pnprog]

Wow, that's huge! Bravo!
I think one way you could change how you feel about the grey bars would be to just lighten their shade a bit so that the red would shine more by contrast...
Just an idea.
But it's looking promising already, so good job!

Thank four your idea!

I think I will go for this intermediate solution (this is also a way to make the grey bars lighter in fact):

chart2.png (4.72 KiB) Viewed 21669 times

Some area have no grey bar at all, indicating no data from Leela, I don't know why yet, I will need to look into it.