It is currently Wed Feb 26, 2020 9:44 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 11 posts ] 
Author Message
Offline
 Post subject: katago Handicap games
Post #1 Posted: Tue Jan 21, 2020 9:27 am 
Lives with ko

Posts: 261
Liked others: 7
Was liked: 64
Katago 1.3.1 wins every time at H9 against GnuGo (~5k), so, I tried higher handicaps.
At H10, it won by 20+
At H11 it won by resignation
At H12 it won by 2
at H13 (!) it lost by 10, then lost by 15, and won the 3rd game (below)

(Sabaki 0.43.3, laptop with GTX 1660Ti, komi=0, time_settings 0 7 1 (7sec/move) network=g170-b20c256x2-s1039565568-d285739972.txt.gz )

Against CrazyStoneDL (set at 5 dan)
it won at H6, H7, and H8 (below)
woaw !!!


Attachments:
CS5d_H8_kata131_7s.sgf [2.98 KiB]
Downloaded 557 times
gnu_k131H13_Kw.sgf [2.91 KiB]
Downloaded 557 times

This post by Vargo was liked by 2 people: Bill Spight, iopq
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #2 Posted: Wed Jan 22, 2020 1:49 am 
Lives with ko

Posts: 261
Liked others: 7
Was liked: 64
Same kind of results with gtp4zen7
Sabaki 0.43.3, laptop with GTX 1660Ti, komi=0
time_settings 0 7 1 (7sec/move) network=g170-b20c256x2-s1039565568-d285739972.txt.gz for katago 1.3.1

According to this site , gtp4zen parameters are :

zen7 5K : -t1 -r5 -s800 -n1 -o2.4 -p0.3
zen7 1D : -t1 -s1800 -n1 -o1.3 -p0.35 (and -r5)
zen7 5D : -t1 -s2700 -n2 -o1.0 -p0.55 (and -r5)

with these settings,
zen5k loses by resignation at H12
zen1d loses by resignation at H9
zen5d loses by resignation at H8

The games :
I'll try higher handicaps, and upload lost games too.


Attachments:
k131z71DH9.sgf [2.63 KiB]
Downloaded 515 times
k131z7_5kH12.sgf [2.29 KiB]
Downloaded 510 times
k131z75DH8.sgf [3.58 KiB]
Downloaded 511 times
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #3 Posted: Wed Jan 22, 2020 7:37 am 
Lives with ko

Posts: 261
Liked others: 7
Was liked: 64
Two handicap games with free placement of the handicap stones :
Gnugo plays n consecutive moves before the first white move.

GnuGo seems stronger if it can choose the places for the handicap stones.
At H12, GnuGo wins 3 times, with katago resigning (too?) soon. So I changed the resignThreshold to -0.9999
But katago lost this game :

At H10, with free placement by GnuGo, katago wins :


Attachments:
gnu_k131H10_Kwins15.sgf [2.77 KiB]
Downloaded 488 times
gnu_k131H12_Gwins13.sgf [3.26 KiB]
Downloaded 486 times
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #4 Posted: Fri Jan 24, 2020 6:48 am 
Lives with ko

Posts: 261
Liked others: 7
Was liked: 64
After several 20-game matches at very high handicap (katago v1.3.1 v. Gnugo 3.8)
I find the results very surprising...
For example, at H17 (!) Katago wins ~20% of the games (twogtp 1.5.1, 3sec/move and resignThreshold = -0.9999 for Katago, desktop with GTX 1080)
Here is one :

At H15, on a laptop, katago wins about 50% of its games on Sabaki (7 sec/move with a 1660 Ti)

Is Katago absolutely incredible, is Gnugo getting weaker, or is it me, doing something stupid :scratch: ?
Could someone please try it, thank you.

If you want the same places as I had for the handicap stones, download h15.sgf
1) In Sabaki, "File", "Open..." (h15.sgf)
2) "Engines" "Attach..." , Gnugo for B, Katago for W , and "OK"


Attachments:
h15.sgf [126 Bytes]
Downloaded 7 times
kata131gnu_h17-11.sgf [1.87 KiB]
Downloaded 382 times
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #5 Posted: Sat Jan 25, 2020 4:13 am 
Lives with ko

Posts: 219
Liked others: 77
Was liked: 56
Rank: EGF 3d
KGS: gennan
Tygem: gennan
OGS: gennan
Kaya handle: gennan
If GnuGo is 5k EGF, then 15 stones handicap for 50% looks quite reasonable to me. It suggests KataGo is 10-11d EGF, meaning it should be able to give strong pros 2-3 stones handicap, which is consistent with actual game results of strong pros vs strong AI.

I'm even pleasently surprised by this consistency.

Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #6 Posted: Sat Jan 25, 2020 4:35 am 
Lives with ko

Posts: 219
Liked others: 77
Was liked: 56
Rank: EGF 3d
KGS: gennan
Tygem: gennan
OGS: gennan
Kaya handle: gennan
Experimenting with KataGo 1.3.1, I updated my table of point values of handicaps (without komi). (previous instance: https://lifein19x19.com/viewtopic.php?f=15&t=16992&p=249931&hilit=handicap#p249931)
Also in this respect, I'm pleasantly surprised by the improved linearity even including higher than 9 stones handicap.

Code:
Handicap  Estimated  Ideal
1           6          7
2          20         21
3          34         35
4          48         49
5          60         63
6          74         77
7          87         91
8         103        105
9         115        119
10        127        133
11        139        147
12        153        161
13        170        175
14        186        189
15        206        203
16        219        217
17        222        232

The estimated points value is based on KataGo's evaluation.
The ideal points value is based on the assumption that perfect komi is 7 points.
I used traditional fixed handicap placement and for handicaps higher than 9 I used 3-3 points for handicaps 10-13 and 10-3 points for handicaps 14-17.

The discrepancy from perfect linearity is less than 6% (except for the 1st entry, but there it's only 1 point).

To me this suggests there exists a nice and simple probability distribution of errors made by players of a specific rank. This might be useful to create a nice statistical model for rating systems based on ranks. It may even be possible to estimate a player's rank by analyzing a few of his games to find his typical error distribution.

The anchor of such a rating system would be perfect play (no errors). We don't know exactly which rank that is, but if we assume that "God" is barely able to give strong AI a 2 stone handicap, she would be about 12d-13d EGF (EGF rating 3250).

Actually I've been working on such a statisical model lately.


This post by gennan was liked by 2 people: Bill Spight, Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #7 Posted: Tue Jan 28, 2020 3:35 am 
Lives with ko

Posts: 261
Liked others: 7
Was liked: 64
According to this site, there are several patterns for very large handicaps.
With the Japanese pattern, Katago can win some games up to 20 stones :o against Gnugo.
At 21 stones , even with different parameters, I've not seen it win.

Katago 1.3.1 , Gnugo 3.8 , Gogui-twogtp 1.5.1, maxVisits=1000 and resignThreshold=-0.999 for Katatgo

Two games won by Katago :

H19 :
H20 :


Attachments:
kata131gnu_h20japan-14.sgf [1.94 KiB]
Downloaded 277 times
kata131gnu_h19japan-1.sgf [2.2 KiB]
Downloaded 278 times
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #8 Posted: Tue Jan 28, 2020 6:57 am 
Lives with ko

Posts: 277
Liked others: 0
Was liked: 35
Rank: 2d
gennan wrote:
To me this suggests there exists a nice and simple probability distribution of errors made by players of a specific rank. This might be useful to create a nice statistical model for rating systems based on ranks. It may even be possible to estimate a player's rank by analyzing a few of his games to find his typical error distribution.

The anchor of such a rating system would be perfect play (no errors). We don't know exactly which rank that is, but if we assume that "God" is barely able to give strong AI a 2 stone handicap, she would be about 12d-13d EGF (EGF rating 3250).

You may also be interested in this, your idea sounds similar.

Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #9 Posted: Tue Jan 28, 2020 9:04 am 
Lives with ko

Posts: 237
Liked others: 35
Was liked: 24
Katago 1.3.1 "g170 20 block s1.42G" (maxVisits=300, resignThreshold=-0.999) & AmiGo1.8 (~11k) H25:

https://sourceforge.net/projects/amigogtp/


Attachments:
amigo - katago.sgf [1.83 KiB]
Downloaded 236 times
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #10 Posted: Sun Feb 23, 2020 2:26 am 
Lives with ko

Posts: 261
Liked others: 7
Was liked: 64
A chain of H9 games (komi 0.5, Sabaki 0.43.3)

KataGo 1.3.2 (16k visits, 20bs191) wins against KataGo 1.3.2 (1 playout, 20bs191) W+R

KataGo 1.3.2 (1 playout, 20bs191) wins against GnuGo 3.8 W+20.5

GnuGo 3.8 wins against AmiGo W+194.5
And maybe a very weak bot would lose at H9 against Amigo ;-)


Attachments:
kata132_16kv_kata132_1pH9.sgf [2.48 KiB]
Downloaded 76 times
gnu_kata132H9.sgf [1.56 KiB]
Downloaded 75 times
GnuGo-AmiGoH9.sgf [1.94 KiB]
Downloaded 76 times
Top
 Profile  
 
Offline
 Post subject: Re: katago Handicap games
Post #11 Posted: Sun Feb 23, 2020 9:15 am 
Lives with ko

Posts: 237
Liked others: 35
Was liked: 24
Vargo wrote:
And maybe a very weak bot would lose at H9 against Amigo ;-)

may be LZ 008 0.1k? (https://github.com/breakwa11/GoAIRatings), but against him LZ 004 0.01k?

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: explo, Kirby and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group