It is currently Wed Mar 22, 2017 11:10 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 15 posts ] 
Author Message
Offline
 Post subject: Understanding Leela console message
Post #1 Posted: Thu Oct 20, 2016 8:11 am 
Beginner

Posts: 17
Liked others: 0
Was liked: 8
Rank: OGS 7 kyu
Dear all,

I am working on a program to use Leela to help review my games. And for that, I am parsing the standard output of Leela in console when running in gtp mode. I am not so clear with the meaning of some Leela's messages. In fact, I am not so clear with most of the output, so I am coming with a lot of questions.
Let me show an example (using Leela 6.2 under Linux).

Here I run Leela in gtp:
Code:
./leela_062_linux_x64 --gtp

Then in the terminal, I input some game moves using gtp commands, as follow:
Code:
play black q4
play white d16
play black q17
play white d3
play black d5
play white f4
play black c3
play white c2
play black c4
play white e2
play black c9
play white q15
play black r13
play white r17
play black r18
play white q16
play black p17
play white r14
play black q13
play white s17


Note I did not mention anything specific about komi; so this would give black a advantage in Leela's eye. Also; this game if from Henning W. as published on that thread: http://lifein19x19.com/forum/viewtopic.php?f=4&t=13669.
At that point, the go board position should be like this:
Click Here To Show Diagram Code
[go]$$c Current game position, black to play
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . X . . |
$$ | . . . . . . . . . . . . . . X X O O . |
$$ | . . . O . . . . . . . . . . . O . . . |
$$ | . . . . . . . . . . . . . . . O . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . . X X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . . . . . . . . . . . |
$$ | . . X . . O . . . . . . . . . X . . . |
$$ | . . X O . . . . . . . . . . . . . . . |
$$ | . . O . O . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

Then, I ask Leela to play the next move with that gtp command:
Code:
genmove black


Here is Leela output:
Code:
MC winrate=0.375000 score=W+7.0
Nodes: 8333, Win: 51.44%, PV: n16 o15 n15 o14 q10 l17
Nodes: 19888, Win: 51.62%, PV: n16 o15 n13 o16 o17
Nodes: 31234, Win: 51.26%, PV: n16 o15 o13 o16 o17 s18 k17

n16 ->   28871 (U: 51.35%) (R: 51.71%:   29322) (N: 64.1%) PV: n16 o15 o13 o16 o17 s18 n15
m17 ->    4769 (U: 50.93%) (R: 51.75%:    7161) (N: 15.3%) PV: m17 o16 o17 r9
s18 ->       0 (U:  0.00%) (R:  0.00%:    1104) (N:  7.9%) PV: s18
o15 ->       0 (U:  0.00%) (R:  0.00%:    1183) (N:  6.4%) PV: o15
o16 ->       0 (U:  0.00%) (R:  0.00%:    1297) (N:  2.4%) PV: o16
n17 ->       0 (U:  0.00%) (R:  0.00%:    2387) (N:  1.1%) PV: n17
====================================
28871 visits, score 51.35% (from 51.29%) PV: n16 o15 o13 o16 o17 s18 n15

33640 visits, 3080 nodes, 4028 vps

= n16

So now (for what I understand), Leela proposes to play n16 as the next move. And the expected follow up is as follow: o15 o13 o16 o17 s18 n15
And other good move that Leela considered is M17, with that follow up: o16 o17 r9

For those two possibilities, what are the meanings of the complete lines? for example, for N16, what the meaning of "U: 51.35%", "R: 51.71%: 29322" and "N: 64.1%" in
Code:
n16 ->   28871 (U: 51.35%) (R: 51.71%:   29322) (N: 64.1%) PV: n16 o15 o13 o16 o17 s18 n15


The other potential moves (S18, O15, O16, N17) have U and R value at 0%, what the meaning for that? Are those moves that were immediately refuted by Leela (bad moves)?

Regarding the fist line output:
Code:
MC winrate=0.375000 score=W+7.0
How to interpret it? 37.5% of chance to win for black according to Monte Carlo search ?

Those 3 lines:
Code:
Nodes: 8333, Win: 51.44%, PV: n16 o15 n15 o14 q10 l17
Nodes: 19888, Win: 51.62%, PV: n16 o15 n13 o16 o17
Nodes: 31234, Win: 51.26%, PV: n16 o15 o13 o16 o17 s18 k17
Are they potential variations on the main line of play, N16?

And finally, for that line:
Code:
28871 visits, score 51.35% (from 51.29%) PV: n16 o15 o13 o16 o17 s18 n15
The score of 51.35% is from the U value above (my guess), then what about that "from 51.29%"? Is that the winning rate estimated by Leela prior to play N16?

Ok, so now the situation on the board is as follow:
Click Here To Show Diagram Code
[go]$$c Leela just played N16
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . X . . |
$$ | . . . . . . . . . . . . . . X X O O . |
$$ | . . . O . . . . . . . . # . . O . . . |
$$ | . . . . . . . . . . . . . . . O . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . . X X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . . . . . . . . . . . |
$$ | . . X . . O . . . . . . . . . X . . . |
$$ | . . X O . . . . . . . . . . . . . . . |
$$ | . . O . O . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]


If I now input the following gtp command:
Code:
mc_score
I got:
Code:
= W+6.5


W+6.5 is and estimation of Leela result at this point of the game?

Thanks in advance for your replies. I am coming with a lot of questions, but I feel that there a lot of potential from learning from Leela's output, for players several stones weaker than Leela (like me). I am trying to make the extration of such information automatic and displayed in an adequate way to review games.

Best regards,


This post by pnprog was liked by: Drew
Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #2 Posted: Sat Oct 22, 2016 2:50 pm 
Beginner

Posts: 15
Liked others: 10
Was liked: 17
Please note that Leela GUI works well on GNU/Linux under Wine. I'm attaching below an illustrative screenshot of the running analysis for the game you mention. (This is with Leela 0.7.0, Wine 1.9.21.) So, using native Leela GUI under Wine might be an easy out-of-the-box option to analyse your games on GNU/Linux... Otherwise you can get the meaning of some of the console output variables by comparing the two, for example see "N %" and "Net prob %".

Attachment:
leela-analysis-demo-linux.png
leela-analysis-demo-linux.png [ 174.1 KiB | Viewed 1380 times ]


This post by tsuboniwa was liked by: Drew
Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #3 Posted: Mon Oct 24, 2016 10:10 pm 
Lives with ko
User avatar

Posts: 222
Location: 中国
Liked others: 176
Was liked: 66
Rank: infant
I can get the vanilla Leela to work with PlayOnLinux, but Leela070_OpenCL crashes. Have you had success running that in Wine? Do you know the dependencies?

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #4 Posted: Wed Oct 26, 2016 7:35 am 
Beginner

Posts: 17
Liked others: 0
Was liked: 8
Rank: OGS 7 kyu
tsuboniwa wrote:
Please note that Leela GUI works well on GNU/Linux under Wine. I'm attaching below an illustrative screenshot of the running analysis for the game you mention. (This is with Leela 0.7.0, Wine 1.9.21.) So, using native Leela GUI under Wine might be an easy out-of-the-box option to analyse your games on GNU/Linux... Otherwise you can get the meaning of some of the console output variables by comparing the two, for example see "N %" and "Net prob %".

Attachment:
The attachment leela-analysis-demo-linux.png is no longer available
Many thanks for the reply and pictures!

From what I can see on your picture, the "other potential moves" (d6, m4, o4, r13, r14, o17 and n17 in your running analysis) are in fact moves that where given too low probabilities to be played by the neural network and thus were not analysed with MC search, but not necessarily bad moves.

So the "Win%" column probably corresponds to the "U:" variable in console and the "Net Prob%" column is "N:" variable.

I still do not understand the "R:" variable meaning, and I now guess it is probably linked with "from" in the last console message "28871 visits, score 51.35% (from 51.29%) PV: n16 o15 o13 o16 o17 s18 n15". For this one signification, I will have to send an email to Leela creator.

tsuboniwa wrote:
So, using native Leela GUI under Wine might be an easy out-of-the-box option to analyse your games on GNU/Linux...
Haha, I am a happy user of the terminal, but there is more to it: The program I have been working on is intended to help me replay/analyse my games and operates this way:

It takes a sgf file, and for each game positions, it asks Leela to play its best move, and parses from the terminal the different sequences proposed by Leela. It also gets the score estimate from Gnugo, as this score estimation is more "human friendly" than what is proposed by Leela (because Gnugo does not try to win by a very small margin/high win probability, Gnugo gives a real territory estimation). With all that information, it builds a new sgf file with all variations in the game tree, and territory estimations/win rates in the comments.

As I give a long thinking time to Leela (30s or 60s per moves) it takes a (very) long time, and I usually run that during the night or while I am not busy at the computer. For that reason, working from the terminal is best to make it an automatic process, and in this case, it seems there is more information available from the terminal than from the interface (I did not try Leela 0.7.0 yet, maybe the console message are different?).

The sgf file I get is huge, with too many variations to be worked with using usual sgf editor. I usually use Quarry for that, and there may be better editors for huge game trees; but still, I think it is too much variations and too much "computer mouse work" to be able to enjoy the review. Have a try with the one enclosed, result of analysis from the original example to get a feeling.

So next, I built a visual interface to help work with that sgf file. It displays two gobans: on the left side goban, the game position with actual game move represented with the letter A (and score estimate by gnugo); on the right side goban, the different variations proposed by Leela, A, B, C, D... from the best win probability to the lowest one:
Attachment:
Screenshot1.png
Screenshot1.png [ 23.63 KiB | Viewed 1229 times ]


Move the mouse pointer over one of those letters and the complete variation is displayed with Leela win rate estimation (or actual game 5 next moves for the left side goban):
Attachment:
Screenshot2.png
Screenshot2.png [ 31.88 KiB | Viewed 1229 times ]


That's it for now, but I also plan to add the possibility to open the current game position into a third goban (like a popup) with the possibility to try other moves or variations against Gnugo or Leela. Maybe I also could use those "other potential moves" that were not searched further with MC search to display additional "hot area" the same way as Leela interface is doing. But too much information sometime is not so good. I will give it a try.

Leela is several stones stronger than I am, so reviewing my games this ways help me find missed opportunities, point out some of my recurrent mistakes, and give me some exposure to the more influence oriented Leela's way of playing. And all that with minimum "mouse work", so the review can go pretty fast.

This is all made with python, I will release the code when it's working well. Anyway, Leela is an amazing program, and having a gtp version available provides even more opportunities to harness this power.

Best regards,


Attachments:
File comment: sgf file after analysis.
8kgame.r.sgf [525.36 KiB]
Downloaded 23 times

This post by pnprog was liked by: Satorian
Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #5 Posted: Wed Oct 26, 2016 3:56 pm 
Lives with ko
User avatar

Posts: 222
Location: 中国
Liked others: 176
Was liked: 66
Rank: infant
Fantastic work, pnprog. Please do share your code once you're ready! Preferably on github or the like so others can iterate. :tmbup:

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #6 Posted: Thu Oct 27, 2016 2:53 am 
Dies in gote

Posts: 46
Liked others: 12
Was liked: 8
Rank: Total beginner
Universal go server handle: Satorian
Sounds awesome! I've thought about doing something similar in Python, so I'm happy this is being worked on already. Now let's hope Leela gets a value net at some point and gets even stronger and can provide even better analysis. A variable bias/filter between win rate and expected territory would also be a nice option for Leela.

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #7 Posted: Mon Dec 05, 2016 2:45 am 
Lives with ko

Posts: 202
Location: Santiago, Chile
Liked others: 39
Was liked: 43
Rank: EGF 1d
Universal go server handle: Jhyn
I am having some trouble to use leela in an automatic manner and would appreciate your help.
To review a game in sgf format I first turned it into a sequence of gtp commands such as (to take your example):

Code:
play black q4
play white d16
play black q17
play white d3
play black d5
play white f4
genmove black


then calling ./leela070gtp_x64 < input_moves.txt > output.txt

Leela plays through the game correctly but then starts displaying the same board again and again at a very fast rate. My guess is that it reads the end-of-file character repeatedly. Do you know how to solve this?

I am also surprised by the fact that Leela uses the standard error channel to display the board and its analysis result (instead of standard output). Is this usual?

_________________
La victoire est un hasard, la défaite une nécessité.

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #8 Posted: Mon Dec 05, 2016 2:55 am 
Lives with ko

Posts: 192
Liked others: 6
Was liked: 52
Rank: KGS 3 kyu
It's not unusual.
Note that you can redirect stderr using 2>.

Code:
./leela070gtp_x64 < input_moves.txt 2> output.txt

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #9 Posted: Mon Dec 05, 2016 7:06 am 
Lives in gote

Posts: 410
Location: Vienna, Austria
Liked others: 192
Was liked: 187
Jhyn wrote:
Leela plays through the game correctly but then starts displaying the same board again and again at a very fast rate. My guess is that it reads the end-of-file character repeatedly. Do you know how to solve this?


I saw the same effect but then tried to add a "quit" line after the "genmove" line, and it worked with:

Code:
leela_080_macOS <input-moves.txt >out 2>&1


You can get a list of supported commands with:

Code:
list_commands


So you can use the "komi" command to set the komi, for example.

_________________
http://badukspace.com - An explorer space for data related to this game


This post by Marcel Grünauer was liked by: Bonobo
Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #10 Posted: Mon Dec 05, 2016 8:51 am 
Lives with ko

Posts: 202
Location: Santiago, Chile
Liked others: 39
Was liked: 43
Rank: EGF 1d
Universal go server handle: Jhyn
Marcel Grünauer wrote:
I saw the same effect but then tried to add a "quit" line after the "genmove" line, and it worked.


Excellent; seems I hadn't done my homework properly. Thank you for the help.

_________________
La victoire est un hasard, la défaite une nécessité.

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #11 Posted: Fri Jan 13, 2017 11:36 pm 
Beginner

Posts: 17
Liked others: 0
Was liked: 8
Rank: OGS 7 kyu
Drew wrote:
Fantastic work, pnprog. Please do share your code once you're ready! Preferably on github or the like so others can iterate. :tmbup:

Hello all!

Just to say that I made some of the modifications I was mentioning above (possibility to open a position in popup to play out variations) and put the result on github.

To be honest, it is unusable for now without having to dig into the python code (to change leela executable path and other things) and running it from the command line. But at least the code is available now. In fact, I was closed to lose this project data due to a miss manipulation on my computer, so it urged me to put everything on github even if it's not usable yet.

I'm new to all this git/github thing, so I will need some time to get comfortable with this way of working, but I plan to make the whole thing easy to use soon. All the recent developments about AlphaGo/DeepZen are very exciting, it's probably a matter of years before pro level bots become available for study for most players.

Best regards,


This post by pnprog was liked by: Drew
Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #12 Posted: Sat Jan 28, 2017 8:39 pm 
Dies in gote

Posts: 20
Liked others: 0
Was liked: 5
OGS: roy7
Universal go server handle: roy7
pnprog wrote:
So the "Win%" column probably corresponds to the "U:" variable in console and the "Net Prob%" column is "N:" variable.

I still do not understand the "R:" variable meaning, and I now guess it is probably linked with "from" in the last console message "28871 visits, score 51.35% (from 51.29%) PV: n16 o15 o13 o16 o17 s18 n15". For this one signification, I will have to send an email to Leela creator.


I did email the author about some other things and while I was at it, I asked about this. The reply was:

Quote:
(R: x% y) means y RAVE traversals with an average winrate of x%. RAVE (Rapid Action Value Estimation) is an older Go programming technique. Leela doesn't actually use it any more in 19x19. It may disappear from the output eventually.

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #13 Posted: Sun Jan 29, 2017 6:32 am 
Lives in sente

Posts: 827
Liked others: 0
Was liked: 128
Jhyn wrote:

I am also surprised by the fact that Leela uses the standard error channel to display the board and its analysis result (instead of standard output). Is this usual?


You are confusing name with function. In this case, the language provides for two output files that you do not have to open as files (always available). I lost my C books in the 2006 house fire, so just going by memory, but I think all that is different is "buffering*. So standard error releases each record written to it immediately (with a "buffered" file, the physical write takes place only when the buffer has been filled or at the end (the last partial gets written). You don't send error messages to standard output because in a case of abnormal termination, you might not get to see them.

In other words, error messages should be sent to standard error but that does not mean the only things you can send there are error messages.

You can of course have more outputs, but those would be files you had to define and explicitly open.

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #14 Posted: Thu Feb 02, 2017 4:51 am 
Beginner

Posts: 17
Liked others: 0
Was liked: 8
Rank: OGS 7 kyu
roy7 wrote:
pnprog wrote:
So the "Win%" column probably corresponds to the "U:" variable in console and the "Net Prob%" column is "N:" variable.

I still do not understand the "R:" variable meaning, and I now guess it is probably linked with "from" in the last console message "28871 visits, score 51.35% (from 51.29%) PV: n16 o15 o13 o16 o17 s18 n15". For this one signification, I will have to send an email to Leela creator.


I did email the author about some other things and while I was at it, I asked about this. The reply was:

Quote:
(R: x% y) means y RAVE traversals with an average winrate of x%. RAVE (Rapid Action Value Estimation) is an older Go programming technique. Leela doesn't actually use it any more in 19x19. It may disappear from the output eventually.

Thanks for asking.

So I guess I will just ignore this one.

Best regards,

Top
 Profile  
 
Offline
 Post subject: Re: Understanding Leela console message
Post #15 Posted: Mon Feb 20, 2017 3:37 am 
Dies in gote

Posts: 27
Liked others: 0
Was liked: 7
Leela's GTP replies come on stdout, and the diagnostic output comes over stderr. The diagnostic output is not valid GTP so it would confuse GTP interfaces if sent over the same channel.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 15 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Hane and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group