Hi,

I am quite new in Computer Go, in fact I learned Go to be able to follow the AlphaGo vs. Lee Sedol match. And I really like the game and play a lot since that time.

In the past I spent a lot of time with computer chess, but this became boring with the huge amount of super strong engines and a sometimes unfriendly community. Computer Go seems to be much more interesting and I wonder why there isn't a similar community as in computer chess. In the internet I found some sites, but most of them seem to be incomplete or outdated. For example I found no information about Hirataku somewhere else in the net.

First I wanted to know the strength of the different engines and played some matches with twogtp and GoGui. I sorted out the stupid and buggy bots, and here is the result of my 19x19 engine tournaments, please let me know if there is a nice gtp-engine missing:

League A:



League B:



1h/game; 1GB/engine; 1Thread/engine; Ponder off

The next project will be to sort out the ranks, maybe I start a webblog with links and informations.

Best wishes,
Alex

That is because the computer chess community has had useful/reasonable programs and computers for about 35 years now, while Go programs only have become useful for anything else but complete beginners in the last 15 years. The computer chess community is much older, and research has been much more focused on chess in the past. Now that chess is almost 'cracked' (30+ move opening books, 7 piece endgame table bases, 50 years of research into algorithms), research has been moving to Go for the last 10-12 years, with impressive results (or disastrous, depending on the way you look at it).



Well done. I'm about as strong as GNU Go on level 10 at this point, so now I know that everything weaker than that is not worth bothering with unless I give some handicaps. I didn't know that Mogo was still available as a Windows binary. On the site, there's only a Linux one, and no source. It looks like I have to find it somewhere, because it seems to be the next step up after I can beat GNU Go decisively (Which for me, is: defeat it >= 50% of the time when giving 2 stones handicap.)

If you want to find out the ranks: GNU Go is always KGS 5k at level 10. IT only plays faster on faster hardware, not stronger. You can thus use GNU Go as a benchmark. If an engine can beat GNU Go while giving it 3 stones handicap, it would be a 2K engine.

Aya, according to the readme, has a rank of KGS 8k or 9k when run at the default settings.

But...



Dude... I patented that idea!

Everything below GNU Go basically is 7k or less. As matches between these engines are not too useful for serious/more experienced players, I think I'll only run matches for the strongest engines (if I run any at all, apart from benchmarking against GNU Go.) For the rest, I'll just do what I planned to, and that is to put up a small site that provides compiles of engines that aren't normally available.

Did you use 'my' version of Pachi 11.99, which I posted in this forum?

For the sake of comparison, I would also include the 'official' Pachi 11.0:
http://asmoda.net/files/pachi_11_0_win32_win64.zip

And the newest version of Fuego, being 1.1_r2029:
http://asmoda.net/files/fuego_1_1_r2029_win32_win64.zip

If you are interested in the engine Oakfoam, you can get an older binary there, for version 0.13. I have created the 0.20 version, but I have yet to test it and write a configuration HowTo for it (it looks a lot like Fuego), and then I'll put that one online as well. Maybe you can then also use Oakfoam 0.13 and 0.20 in your tournament run as well.

If you have a multi-core computer (4 cores), you'd be able to run 2 threads per engine, and leave ponder on. That would improve the strength by a noticeable amount.

Maybe the Go player is more easy going than the typical chess player. There are people who run thousands of chess games to find out which engine is 5 ELO stronger than the other. In Go you have a few ranks, handicap matches are common and it is not so important for Go players to be "better" than the opponent. They just enjoy the game.


I am on Ubuntu and use the binary from the Mogo site but here is a windows binary: http://www.godrago.net/Engines.htm


I'll try to start with the strongest (3-4dan), it will be interesting to see if GNU Go will end up with 5k in my testing.


I used the one from the Pachi site. Do you think there is a measurable difference between 11 and 11.99? I doubt this few games will show such small differences I would expect from this two versions.
I saw your Fuego after I started tournament, I'll take a look at it.


Thanks


For now I decided to run 1 thread/engine to be able to run 3 matches at the same time. I know that some engines supporting more threads and pondering will have a disadvantage, but for now I just want to have a rough estimate about the strength.

Do you know about http://www.computer-go.info/ ?

I think that could be the wrong way around. There is a reason why I mentioned GNU Go as a benchmark.

Most engines use a set time interval to play a move, such as 10 seconds, at their default settings. Their strength depends on the speed of the computer (and sometimes, the memory available). On a faster computer, they become stronger. They don't have a set strength.

GNU Go, when playing at level 10, is always 5-6k, independent of the hardware. On slow hardware, it just takes longer to make the same moves (give or take a bit of randomness, to make it not play exactly the same each time) as it would on fast hardware.

Because GNU Go's strength does not change, it is useful as a benchmark point to test other engines against. After you find out, for example, that Mogo is 3 stones stronger than GNU Go on your computer, at the given settings, then you can use Mogo as a benchmark for an even stronger engine, such as Fuego; and so on.

If you do have a way to use a different engine as a benchmark, such as by setting Pachi in such a way that it always plays at 1d strength independet of hardware, I'd certainly like to hear about it

edit: Where did you find Indigo 2009? The version linked from Sensei's Library is version 2006.

I don't think it is that easy, don't forget the timecontrol. GNU Go will perform better at fast timecontrol compared to other engines. If it is even in a 1h/game against an certain opponent, it will also be even against this opponent with 2 times faster hardware in a 30min/game.

The first step will be to find the differences between all engines, (no matter where to start ) and then calibrate it with the reference engines of known strength. The 2nd step could be to do the same in faster games, then one could create a matrix of strength on different hardware.


Sorry, that was a typo, it is Indigo 2006


Yes, I've seen this site. It seems to be complete, and has some interesting informations but sometimes it's hard to find what I am searching for

That would be a good way to do it. I wonder how much computing power would really make a difference. In a chess engine, going from 1 to 4 cores, often only makes a 35 ELO difference. Would a boost from 1 to 4 cores + pondering add more than 1 stone to an engine such as Pachi? I don't know.



And there's only a Linux binary there, right?

Also, if you used Pachi from the website, is there an 11.99 binary there for Linux? If so, I've missed it. AFAIK, my own version is the only one around for Windows. The website still shows the 11.0 binary for Windows.

RIght.


I compiled the binary from the sources, worked without problems.

I had no success with Oakfoam 0.2.0 so far, it wants to have some libraries installed. For now I'll try v0.1.3...

Yes, Oakfoam requires Boost. It works when using the latest version. If your distribution has 1.61 available, you can install them, or compile them from source and link them statically if you want to.

I can only provide a Windows version of Oakfoam 0.20, if you want it. I'll test it this weekend, and write the HowTo as well, but I can provide a download link tomorrow, if you wish.

No need to make an effort, I'll try your windows binary when it is ready.

Interesting, didn't know Leela. Haha, that guy did chess programming before, AFAIR. That guy! I knew him from the hydrogenaudio.org community/foobar20000...
Leela cannot use opening books I guess. Did you use opening books for Hira, Pachi and Fuego? I guess you didn't.
Also you should publish the used command lines.

Fuego and Patchi played with book, Hira without. I didn't know out that it supports books.

Leela has no book support afaik.

Here is the configuration:

fuego.exe –config fuego.cfg
---
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
taskset -c 0 Hiratuka-19×19.exe -po 15000
leela –gtp –threads 1 –noponder
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
gnugo –mode gtp –level 10
Indigo.exe -gtp

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

I lately tried to find out the Rank differences between the engines. It is quite time consumeing because you have to play a lot of games and you have to set them up manually. So in a first step I selected only 3 engines with a low number of playouts to play fast games.

The engines are:

GNU Go Level 10
Hiratuka with 1000 po
Pachi with 6500 po

Pachi and Hiratuka are calculating nearly the same time with this settings, Hiratuka is 6 stones ahead Pachi, and Pachi is 5 stones ahead GNU go.

GNU go is supposed to have a 5k rank, that would mean Pachi with 6500po is 1d, and Hiratuka with 1000po is 7d. That is complete nonsense of course so I suppose the rank system can not be adapted to engine matches, at least at this level.

After that I wanted to find out how the playing strength increases with more playouts. After doubling it's speed Pachi was 4-5 stones behind Hiratuka and 6-7 stones ahead GNU go. So doubling the speed increases the strength by more than 1.5 ranks in engine matches which is probably like 1/2 rank in real games.

How is this differ from computer go server?

There are some engines with strong parameters of compilation and board for them (see manual in Russian with rating).
There and there are some games with strong parameters of these engines start.

Hello Alex,

you could have a look into BayesElo here: http://www.remi-coulom.fr/Bayesian-Elo/. It is a widely used tool in chess programming to determine strength differences between engines. I haven't worked with it much, but basically, you feed it game results in PGN format and receive ELO ratings calibrated around an initial ELO rating. It will also calculate a confidence interval (called the error bar), which will become narrower the more games are available.

Because BayesELO doesn't care about the chess moves but only for the metadata, you could write a script to convert your SGFs into PGNs without moves.

I don't know if GoGUI has good support for engine tournaments. In chess, there are tools that can run tournaments automatically - so you could set up a dedicated machine, configure the tournament, press "Start" and come back a week later to view at the BayesElo results...

I think for meaningful ELO ratings you would need much more games, especially because there are some huge gaps in strength between some of the go engines.

SmartGo is a nice GUI for engine tournaments, but there are some flaws under Linux with wine. With GoGui you need a tool called twogtp which is something like cutechessqli. Unfortunately it only supports engine matches so you have to setup the rounds of a tournament manually. You have to fiddle with command line options, but once you created the strings it is easy to use.

Update with Ray 8.0.1:

League A:



League B:




fuego.exe –config fuego.cfg
---
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
taskset -c 0 Hiratuka-19×19.exe -po 15000
leela –gtp –threads 1 –noponder
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
gnugo –mode gtp –level 10
Indigo.exe -gtp
ray --time 3600 --thread 1 --no-debug

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Update with Hara 0.9 (with hacked time management):

https://github.com/antoniogarro/Hara

I had to modify it a little bit because it used too much time.

League A:



League B:




amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
fuego.exe –config fuego.cfg
---
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
gnugo –mode gtp –level 10
hara
taskset -c 0 Hiratuka-19×19.exe -po 15000
Indigo.exe -gtp
leela –gtp –threads 1 –noponder
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
ray --time 3600 --thread 1 --no-debug

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

New engine Matilda by Gonçalo

https://github.com/gonmf/matilda