It is currently Wed Oct 23, 2019 2:40 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 13 posts ] 
Author Message
Offline
 Post subject: Did Go Seigen cheat with LeelaZero?
Post #1 Posted: Wed Sep 26, 2018 11:32 am 
Oza

Posts: 2382
Liked others: 15
Was liked: 3456
Some recent posts motivated me to do at last something I've thought about doing for quite a while.

Mainly I was wondering how Go Seigen fared when compared to AI moves. But I was also still wondering about the alleged cheating case where a claim was made that someone used LZ by picking one of its three top moves. I wondered specifically how Go would fare when his move was compared to the top three.

For lack of time I did not let LZ run very long, but since I was also curious how feasible it was to run and use LZ in parallel with fairly quick time limits typical of servers, I treated that as a plus.

None of this is at all scientific, of course, but I was intrigued by a couple of unexpected things thrown up.

The game was Go Seigen as White against Rin Kaiho, chosen mainly because it had komi (but only 5.5. - GSG never used 6.5) but also because it was a significant game (Meijin; 1963-11-18). I looked only at the first 100 moves, i.e. 50 for each player.

Go had 37 of his 50 moves in LZ's top three; Rin had 34. There were no huge mistakes. Go won after 192 moves.

But that close parity masked some trends. Go was clearly more likely than Rin to choose better moves out of the three, because his winning ratio gradually climbed. Starting at 54%, he was at 64% by move 30 and by move 100 was at 84%.

The most fascinating thing for me was that quite often Go's move did not appear at all in LZ's first sweep of the board. But as time progressed his move did appear, and also typically move up to a high score. This phenomenon occurred just once for Rin. I tentatively infer from this that Go was seeing things at a deeper level. At any rate, his performance impressed me.

A further oddity was that on one occasion, Go's move did not appear at all in LZ's many candidates, even when I let it run a bit longer than usual, but once his actual move was played his winning ratio increased (I assume this means LZ approved). On another similar occasion Go's move was scored equal by LZ.

In addition, even when Go's move did not appear in the top three, he was almost always in the top four or five, and also, when his actual move was played, his winning ratio did not fall huge (max. about 4%). On a couple of occasions it instantly showed a big dip (10%) but as LZ searched more it re-adjusted that to 4% or less. Again a sign of Go seeing deeper?

I forgot to make a record but I think there were only two other occasions apart from the one above when Go's move did not appear at all. For Rin there were between 5 and 10.

The only thing this actually proved was that it is quite possible to cheat with LZ in a server game (not that I believed otherwise beforehand) even with fastish time limits, but there was quite a bit more that was suggestive. For example, if you want to play like Rin Kaiho, take LZ's early suggestions. If you want to play like GSG, let it run a bit longer. :)

Has anyone playing on the servers had a strong feeling yet that they are really playing Leela?


This post by John Fairbairn was liked by 3 people: Bill Spight, Drew, Elom
Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #2 Posted: Wed Sep 26, 2018 12:06 pm 
Lives in sente

Posts: 807
Location: UK
Liked others: 546
Was liked: 82
Rank: OGS 9kyu
Universal go server handle: WindnWater, Elom
Thank you, this is evidence for what I've thought for a while; that Go Seigen is one of the most AI-like professionals to have played.

It seems to show that humans can understand the board the way AI do, as you could say Go Seigen was on the way there.

_________________
On Go proverbs:
"A fine Gotation is a diamond in the hand of a dan of wit and a pebble in the hand of a kyu" —Joseph Raux misquoted.


Last edited by Elom on Wed Sep 26, 2018 2:04 pm, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #3 Posted: Wed Sep 26, 2018 1:44 pm 
Lives in sente
User avatar

Posts: 836
Liked others: 180
Was liked: 150
Rank: 3d
GD Posts: 422
KGS: komi
I repeated the above exercise with Leela 11 (the software at the center of all the fuss in the Pandanet League). The results were very different: RK achieved a 40/50 (80%) match with the top 3 moves, while GSG achieved 33/50 (66%). Four of GSG's moves were not on Leela 11's radar (34/36/40/48), while two of RK's moves were not on the radar (29/37).

In terms of assessment, Leela 11 had RK ahead from the start, and at 60% as late as move 73. But by move 100, after that sequence in the corner, GSG was up to 70%.

I used Leela 11 on an i3 PC with a reasonably powerful graphics card. I tried to ensure that the top move had at least 50,000 playouts and that the top 3 moves had at least 10,000 playouts, but this wasn't always possible.

Top
 Profile  
 
Online
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #4 Posted: Wed Sep 26, 2018 3:05 pm 
Lives in sente

Posts: 1194
Location: Earth
Liked others: 409
Was liked: 196
I think this is the game:


Top
 Profile  
 
Online
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #5 Posted: Wed Sep 26, 2018 3:11 pm 
Lives in sente

Posts: 1194
Location: Earth
Liked others: 409
Was liked: 196
both players missed to press down 3-4 at :w8: and :b15: (10% and 15% loss)

(I learned to appreciate the true size of the press down by sensei Hayashi Kozo before the advent of AI)

Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #6 Posted: Thu Sep 27, 2018 12:34 am 
Oza

Posts: 2382
Liked others: 15
Was liked: 3456
Quote:
I repeated the above exercise with Leela 11 (the software at the center of all the fuss in the Pandanet League). The results were very different: RK achieved a 40/50 (80%) match with the top 3 moves, while GSG achieved 33/50 (66%).


Thank you for doing this instructive alternative run. I certainly did not match your number of playouts, and from chess experience I'd always expect even bots to be different from each other, with some being significantly stronger.

But this raises questions about how strong Leela Zero is in the kind of setups most of us here seem to have. I have mixed views. On the one hand I could claim it is weak. I played LZ twice. The first time, I lost by 5 points. I expected to lose. I went in to the game with the sole purpose of defending my own groups and territories resolutely so that nothing died. I wanted to see how LZ coped. I was surprised at the result, but assumed LZ was acting in the old Monte Carlo way, just doing enough to win, not caring about the margin. It did occur to me also that I may have accidentally stumbled across a better way to play go and had become stronger in a trice. But yet another possibility is that my version of LZ was actually weak. My second game, which I won because I made a very long ladder LZ couldn't read (and which it stupidly played out to the bitter end), suggests this could be the case.

I have, however, tended to discount the latter possibility (I favour the MC-type explanation) because the results I get when looking at other games or positions discussed on L19 is that I seem to get similar sorts of results to other setups and machines, and (maybe above all) my LZ is coming up with all the sorts of moves that are typical of the AIs - early 3-3s, shoulder hits, disinclination to pincer, etc.

Also, and this was one reason I ran the Go Seigen test, my LZ seems to make similar sorts of moves to the best pros lots of the time. I was looking to see how GSG matched up to LZ (I thought rather well), but of course it works the other way round: we can see how well LZ matches up to GSG.

I infer therefore that even my short-run version of LZ, while clearly not the strongest around, may, in the absence of ladders, nevertheless be playing at pro level. Does this sort of reasoning make sense?

Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #7 Posted: Thu Sep 27, 2018 1:12 am 
Lives in gote

Posts: 429
Liked others: 1
Was liked: 121
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
Quote:
Thank you for doing this instructive alternative run. I certainly did not match your number of playouts, and from chess experience I'd always expect even bots to be different from each other, with some being significantly stronger.

LZ (with current networks) is much (much) stronger than Leela 11 (and the playouts numbers are not comparable). With enough playouts, LZ is already stronger than humans.

What network did you use ? And how many playouts did you get when running at your time setting ?

Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #8 Posted: Thu Sep 27, 2018 3:14 am 
Oza

Posts: 2382
Liked others: 15
Was liked: 3456
Quote:
What network did you use ? And how many playouts did you get when running at your time setting ?


I don't know how to find networks but I haven't updated after installation. Playouts here I kept in low thousands because I wanted to match the typical speeds of playing on servers. Even when I'm investigating more leisurely I use a max time of 15 minutes per move. But I find that not too much changes and so I assume that my LZ set to play fast is getting 80% or more of the moves found at higher settings. Maybe a rash assumption... In short, I'm at the Mickey Mouse end of the scale :)

Top
 Profile  
 
Online
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #9 Posted: Thu Sep 27, 2018 5:03 am 
Lives in sente

Posts: 1194
Location: Earth
Liked others: 409
Was liked: 196
LZ with low playouts has a very good intuition, but is not good at reading.


This post by Gomoto was liked by: John Fairbairn
Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #10 Posted: Thu Sep 27, 2018 12:02 pm 
Lives in sente
User avatar

Posts: 836
Liked others: 180
Was liked: 150
Rank: 3d
GD Posts: 422
KGS: komi
Tryss wrote:
LZ (with current networks) is much (much) stronger than Leela 11 (and the playouts numbers are not comparable). With enough playouts, LZ is already stronger than humans.


I'm sure this is true. I tend to stick to Leela 11 for two reasons - the integrated GUI, and the fact that the moves it suggest are way more understandable to me. In fact, playing through this particular game, I found GSG's moves quite bewildering and I was not surprised that Leela 11 often didn't consider them. John, or anyone else with Leela Zero, how well did Leela Zero match on moves 29, 34, 36, 37, 40 and 48 - the games moves that didn't even occur to Leela 11?

Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #11 Posted: Thu Sep 27, 2018 12:26 pm 
Lives in gote

Posts: 429
Liked others: 1
Was liked: 121
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
Quote:
how well did Leela Zero match on moves 29, 34, 36, 37, 40 and 48 - the games moves that didn't even occur to Leela 11?


The following is LZ #175 opinion after roughly 20k-25k playouts :

:b29: : Not on LZ radar, and she dislike this move ( -7% and still dropping)
:w34: : Not on LZ radar, and she dislike this move ( -4.2% but dislike it less with more playouts)
:w36: : Barely explored, and she think it's fine ( -0.2% so in the margin of error)
:b37: : Not on LZ radar, and she's not a fan ( -2.2% )
:w40: : LZ 4th choice (with 20k+ playouts), but she like it more after being played with +1.8%
:w48: : Barely explored, and she dislike this move (-6.5% )

Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #12 Posted: Thu Sep 27, 2018 3:22 pm 
Honinbo

Posts: 8676
Liked others: 1489
Was liked: 1473
KGS: Kirby
Tygem: 커비라고해
Quote:
I tend to stick to Leela 11 for two reasons - the integrated GUI, ...


Lizzie seems to work well with Elf network weights. Does Leela 11 have some other GUI?

_________________
it's be happy, not achieve happiness

Top
 Profile  
 
Offline
 Post subject: Re: Did Go Seigen cheat with LeelaZero?
Post #13 Posted: Fri Sep 28, 2018 6:29 am 
Gosei

Posts: 1407
Liked others: 699
Was liked: 459
Rank: AGA 3k KGS 1k Fox 1d
GD Posts: 61
KGS: dfan
Kirby wrote:
Lizzie seems to work well with Elf network weights. Does Leela 11 have some other GUI?
Yes, Leela 0.11 looks like the screenshot here: https://sjeng.org/leela.html

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group