It is currently Tue Oct 22, 2019 7:58 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 5 posts ] 
Author Message
Offline
 Post subject: Analysis of Pro vs Elf with LeelaElf
Post #1 Posted: Tue Jun 26, 2018 12:29 pm 
Judan

Posts: 6128
Location: Cambridge, UK
Liked others: 351
Was liked: 3300
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
When Facebook released their Elf OpenGo bot they also included 12 games versus 4 top Korean pros. These seem to have been rather neglected, so I thought I'd analyse one using the LeelaElf converted weights. As well as the go instruction this can see how close our LeelaElf conversion is to the true Elf: does it play the same moves and give the same win% that were included in the released games (Elf record uses a -1 to +1 scale whereas LeelaZero/Lizze a 0% to 100% so Elf pro record mistake of 0.05 to 0.09 is a 2% LZ mistake).

The games (except first 2) "were played using the v0 pretrained model (publicly available for download). For each move, ELF OpenGo used 2 threads with 40000 rollouts per thread (grouped into batches of 16). This took around 50 seconds per move on a V100 GPU.". For comparison my GeForce 1060 to do 80k playouts with LeelaElf takes 380 seconds (8 times). To enable a decent review pace I generally only did ~10k playouts, but if it looked like another choice might overtake the current top would do more.



And the winrate graph. White's biggest mistakes to 20% were:
- bottom right kick
- hoshi pincer
- hanging connection instead of cut tesuji
- top left approach
- double approach
- triple approach/surround
Attachment:
g11 leela elf winrate.PNG
g11 leela elf winrate.PNG [ 128.14 KiB | Viewed 1629 times ]


As for comparing Elf and LeelaElf, the only significant good move Elf found that LeelaElf didn't in the part I analuysed was the q12 pincer, though LeelaElf agreed it was good when shown it. They both shared the strange blindspot of not seeing the monkey jump kill at top left (which does cast some doubt on the previous evaluations: black could live if didn't solid connect before and fall back, but then white would capture the one stone in sente which is quite a big difference to strength of groups). They were in pretty good agreement at finding the pros mistakes, though exact win% drops varied somewhat.


Attachments:
Elf vs pro game11 - Uberdude review.sgf [14.44 KiB]
Downloaded 520 times

This post by Uberdude was liked by 3 people: Bill Spight, Gomoto, Waylon
Top
 Profile  
 
Offline
 Post subject: Re: Analysis of Pro vs Elf with LeelaElf
Post #2 Posted: Fri Jun 29, 2018 12:17 pm 
Judan

Posts: 6128
Location: Cambridge, UK
Liked others: 351
Was liked: 3300
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
About LeelaElf not finding q12 on its own, I thought this could be because of significant differences between the LeelaZero and Elf engines running similar weights, but according to someone more familiar with them they should actually be very similar, but with usual non-deterministic algorithm (and rotation symmetry) caveats. So I repeatedly ran LeelaElf on the position before q12 for 80k nodes to see what it would choose:
1) for review q6 (or q11 I forget)
2) q12
3) q12
4) q12 from 2k
5) q12 (q6 until 10k, q11 starts to look promising before q12, q12 #1 at 18k)
6) q12 (q6 to 1k, then q11 #1, q6 back to #1 around 15k, q12 #1 at 35k and remains)
7) q12

So seems I got an unluckly roll of the LeelaElf dice when I was reviewing, averaging over multiple trials LeelaElf does find and want to play q12 like real Elf did. (Assuming that my exploration of the game tree didn't somehow bias the search when I got to q12, maybe it did).

Top
 Profile  
 
Offline
 Post subject: Re: Analysis of Pro vs Elf with LeelaElf
Post #3 Posted: Fri Jun 29, 2018 12:38 pm 
Honinbo

Posts: 8913
Liked others: 2678
Was liked: 3033
One thing that I do when considering strategic plays recommended by programs (i.e., with no other stone adjacent or diagonally adjacent to it) is to also look at the adjacent plays. With programs doing a whole board search they may miss local improvements.

A good strategic heuristic is to consider the whole board and come up with a general area of play, and then to consider where to play within that area. In effect (or actuality) you have two searches, a global search followed by a local one. That works pretty well for humans. :)

Edit: If you are trying to create a program that plays well, a dual search heuristic might be inefficient, consuming too much time by comparison with a single global search. But if you are analyzing a position you do not have the same kind of time constraints to worry about. Different goals. :)

_________________
The Adkins Principle:

At some point, doesn't thinking have to go on?

— Winona Adkins

The race is not to the swift, nor the battle to the strong, but that's the way to bet. ;)

Top
 Profile  
 
Offline
 Post subject: Re: Analysis of Pro vs Elf with LeelaElf
Post #4 Posted: Fri Jun 29, 2018 1:24 pm 
Judan

Posts: 6128
Location: Cambridge, UK
Liked others: 351
Was liked: 3300
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Here's a Lizzie picture of LeelaElf thinking about q12, it does consider it, but only has a few playouts and lower win% than q11 and q6 so doesn't devote many new playouts to it so doesn't realise the true win% is higher. So it seems LZ is quite sensitive to the initial (and high error) win% assigned to the other candidate moves, if you are unlucky and they are too low compared to the current #1 it won't put enough effort into investigating them and discovering they are actually good. This test it did actually come to prefer q12 by 50k playouts, but I remember when I was reviewing q12 started at 48% or 49% and never overtook q6/11.
Attachment:
q12.PNG
q12.PNG [ 304.57 KiB | Viewed 1352 times ]

Top
 Profile  
 
Offline
 Post subject: Re: Analysis of Pro vs Elf with LeelaElf
Post #5 Posted: Fri Jun 29, 2018 2:36 pm 
Lives in sente

Posts: 1191
Location: Earth
Liked others: 408
Was liked: 196
I think it is funny how lifelike the AI response is.

Their answer always also depends on how you ask. ;-)

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: jlt, Knotwilg, Shenoute and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group