Engine Tournament

[LetterRip]

Any idea how much time per move and how many playouts were being used by LZ?

[as0770]

Any idea how much time per move and how many playouts were being used by LZ?

1h/game on one thread. Starting with 50sec/move what is around 7000 playouts.

[as0770]

The results (with pondering - without pondering):
MoGo 3 - 1;
Pachi 3 - 1;
Ray 3 - 1;
Leela 3 - 1;
in all 12 - 4 (details).
I don't know, what about quantitatively results (in ELO), but definitely there is qualitative effect, and in sparrings of equivalent strength Go engines the same with pondering may pass in rating engine without pondering.

One more thing about this test: If you play with one engine against itself the ponder hits are close to 100%, That means the pondering side will benefit a lot more than against other engines.

I did also some testing:

In my League A there are two engines that do not ponder: Hiratuka and DarkGo. DarkGo plays instandly, so it doesn't matter if the opponent is pondering because he don't get any time to ponder. So the only Engine that may be affected is Hiratuka, so I testet Hiratuka against Pachi in 30min games with ponder on and ponder off.

First Round:

Ponder Off: Pachi vs. Hiratuka 5:11
Ponder On: Pachi vs. Hiratuka 8:8

This could indeed be a hint that Pachi becomes significantly stronger with pondering. Because I don't believe this I did the same match again with the same conditions. Now I got:

Ponder Off: Pachi vs. Hiratuka 6:10
Ponder On: Pachi vs. Hiratuka 2:14

That is exactly what I expected: The statistical fluctuation when playing matches between engines with similar strength is very high, just like rolling a dice. The difference between pondering and not pondering is simply not meassurable with such a small amount of games.

[q30]

... and in sparrings of equivalent strength Go engines the same with pondering may pass in rating engine without pondering.

So we agree that the question is only relevant in matches between engines where one is able to ponder and the other engine is not? ...
So for both the question of the absolute timecontrol and CPU power is much more relevant than the question of running it in ponder on or ponder off matches...
So where is your point always claiming others as "synthetic" results? ...

If both can ponder too, because one may ponder more effectively, than other...
Yes.
Only on results, obtained from tests with non realistic parameters, for example, time control. On engines with no big difference in strength the results may vary from real control parameters tests.

[q30]

...

That is exactly what I expected: The statistical fluctuation when playing matches between engines with similar strength is very high, just like rolling a dice. The difference between pondering and not pondering is simply not meassurable with such a small amount of games.

Try to use time control equivalent to 2 min per move. In this case fluctuations will be much smaller and difference between pondering and not pondering will be measurable...

[as0770]

Try to use time control equivalent to 2 min per move. In this case fluctuations will be much smaller and difference between pondering and not pondering will be measurable...

You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.

[as0770]

New entry in League B is Dream Go, update in League C Mathilda 1.25

Leela vs. AQ

Code: Select all

    1. AQ 2.0.3                     12/16
    2. Leela 0.11.0 Beta 11          4/16

Configuration:

2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

AQ: AQ

Code: Select all

aq_config.txt:
-main　time[sec] =7200
-time controll =true
-japanese rule =true

Leela: leela_gtp_opencl –gtp –threads 4 –noponder

twogtp: gogui-twogtp -auto -white "leela_gtp_opencl --gtp --threads 4 --noponder" -black "AQ" -games 4 -size 19 -time 120 -verbose -sgffile xxxx

League A:

Code: Select all

    1. Leela 0.10.0                 22/24
    2. Rayon 4.6.0                  19/24
    3. Oakfoam 0.2.1 NG-06          18/24
    4. Hiratuka 10.37B (CPU)         9/24
    5. DarkForest v2 MCTS 1.0        7/24
    6  DarkGo 1.0                    5/24
    7. Pachi DCNN 11.99              4/24

Configuration:

2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 2500 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
Hiratuka: Hiratuka-19×19.exe -po 175000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Oakfoam: oakfoam -c nicego-cnn-06.gtp

Code: Select all

nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

Pachi: pachidcnn -f pachibook.dat threads=4,max_tree_size=2048,pondering=0
Rayon: rayon --thread 4 --no-debug

twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 120 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League B:

Code: Select all

    1. Ray 9.0.1                    31/36
    2. Pachi DCNN 11.99             30/36
    3. Dream Go 0.5.0               29/36
    4. Leela Zero 0.9 (2018.01.01)  21/36
    5. MoGo 4.86                    18/36
    6. deltaGo 1.0.0                18/36
    7. Fuego 1.1                    15/36
    8. Michi C-2 1.4.2               8/36
    9. Orego 7.08                    8/36
   10. GNU Go 3.8                    2/36

Configuration:

1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
Fuego: fuego.exe –config fuego.cfg

Code: Select all

fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
Leela Zero: leelaz --gtp --threads 1 --w /path/to/Leelaz_best-network_2018_01_01 --noponder
Michi C-2: michi gtp

Code: Select all

ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
Ray: ray --time 3600 --thread 1 --no-debug
twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League C:

Code: Select all

    1. GNU Go 3.8                   25/28
    2. Hara 0.9                     18/28
    3. Matilda 1.25                 16/28
    4. Indigo 2009                  16/28
    5. Dariush 3.1.5.7              15/28
    6. Aya 6.34                     13/28
    7. Fudo Go 3.0                   7/28
    8. JrefBot 081016-2022           2/28

Configuration:

1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

Aya: Aya.exe –mode gtp –level max
Dariush: DarGTP.exe –level 10
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
GnuGo: gnugo --mode gtp --level 10 --resign-allowed
Hara: hara
IndiGo: Indigo.exe -gtp
JrefBot: java -jar jrefgo.jar 10000
Matilda: matilda

Code: Select all

matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1

twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League D:

Code: Select all

    1. JrefBot 081016-2022          16/18             
    2. Iomrascálaí 0.3.2            15/18
    3. Crazy Patterns 0008-13       13/18
    4. Marcos Go 1.0                13/18
    5. AmiGo 1.8                    13/18
    6. Beancounter 0.1               8/18
    7. Stop 0.9-005                  5/18
    8. GoTraxx 1.4.2                 3/18
    0. CopyBot 0.1                   2/18
   10. Brown 1.0                     2/18

Configuration:

1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

Amigo: amigogtp
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
GoTraxx: GoTraxx.exe
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 10000
Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp

twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Links:

Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/w ... 73259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/w ... 88349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: https://github.com/gcp/leela-zero
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
Orego: https://sites.google.com/a/lclark.edu/d ... arch/orego
Pachi: http://pachi.or.cz/
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/

Best,
Alex

[lemonsqueez]

Thanks for running these tournaments, impressive lineup !

Right now the leagues are based on strength more or less if i understand correctly.
Just an idea: how about a gpu league and a cpu league for the top programs ?
This is already the case mostly, what i mean is that for programs like Leela which can do both, it'd be interesting to see how the cpu version fares. Not sure how practical this would be. Maybe you don't want to have to unplug your graphic card to prevent it from using the gpu =)

[as0770]

Thanks for running these tournaments, impressive lineup !

Right now the leagues are based on strength more or less if i understand correctly.

You do

One or two engines play in the upper and lower League to have a virtual connection between the Leagues.

Just an idea: how about a gpu league and a cpu league for the top programs ?
This is already the case mostly, what i mean is that for programs like Leela which can do both, it'd be interesting to see how the cpu version fares. Not sure how practical this would be. Maybe you don't want to have to unplug your graphic card to prevent it from using the gpu =)

I want to have a comparison between GPU and CPU engines so I don't want to make different Leagues.

In fact I tried to run the GPU engines also in CPU mode, this works well with Leela. It would play in the same League as Leela GPU but I don't want to have one engine playing twice in one League. This would strain the results. But you can find Leela CPU in the history

Rayon CPU is basically Ray 9.0.1 afaik. AQ won't work as CPU engine here and for Oakfoam as CPU engine I have to adjust some parameters and there where problems running it. And btw it is very, very weak. For other engines I would have to change the system configuration, but I don't want to mess up my system like this. So after all I would like to run the engines in both modes but I would face too many problems...

[as0770]

This time I downsized the Leagues to get space for new entries.

In League B Leela Zero is updated to v0.11 and the last 5x64 network (2018.01.17). After that they changed to a 6x128 network.

Also there is a new Engine in League E: SimpleGo 0.4.3


Leela vs. AQ

Code: Select all

    1. AQ 2.0.3                        12/16
    2. Leela 0.11.0 Beta 11             4/16

League A:

Code: Select all

    1. Leela 0.10.0                    22/24
    2. Rayon 4.6.0                     19/24
    3. Oakfoam 0.2.1 NG-06             18/24
    4. Hiratuka 10.37B (CPU)            9/24
    5. DarkForest v2 MCTS 1.0           7/24
    6  DarkGo 1.0                       5/24
    7. Pachi DCNN 11.99                 4/24

League B:

Code: Select all

    1. Leela Zero 0.11 (2018.01.17)    15/20
    2. Pachi DCNN 11.99                13/20
    3. DarkGo 1.0                      12/20
    4. Dream Go 0.5.0                  11/20
    5. Ray 9.0.1                        7/20
    6. Mogo 4.86                        2/20

League C:

Code: Select all

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:

Code: Select all

    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:

Code: Select all

    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:

Code: Select all

    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:

League A: 2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time xx -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League B-F: 1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time xx -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Amigo: amigogtp
AQ: AQ

Code: Select all

aq_config.txt:
-main　time[sec] =7200
-time controll =true
-japanese rule =true

Aya: Aya.exe –mode gtp –level max
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
Dariush: DarGTP.exe –level 10
DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 2500 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
Fuego: fuego.exe –config fuego.cfg

Code: Select all

fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
GoTraxx: GoTraxx.exe
Hara: hara
Hiratuka: Hiratuka-19×19.exe -po 175000
IndiGo: Indigo.exe -gtp
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 10000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 1 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder
Matilda: matilda

Code: Select all

matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1

Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Michi C-2: michi gtp

Code: Select all

ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Oakfoam: oakfoam -c nicego-cnn-06.gtp

Code: Select all

nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachidcnn -f pachibook.dat threads=4,max_tree_size=2048,pondering=0
Pachi: pachidcnn -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
Ray: ray --time 3600 --thread 1 --no-debug
Rayon: rayon --thread 4 --no-debug
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp

Links:

Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/w ... 73259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/w ... 88349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: https://github.com/gcp/leela-zero
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
Orego: https://sites.google.com/a/lclark.edu/d ... arch/orego
Pachi: http://pachi.or.cz/
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/

Best,
Alex

[q30]

You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.

It depends on game randomness, that changes with the time control...

[Cyan]

Some strong bots have been updated:
AQ v2.1.1
Leela 0.11.0
Ray 4.32
Pachi 12.00

[as0770]

You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.

It depends on game randomness, that changes with the time control...

Then we have to rewrite basic mathematical principles.

[as0770]

Some strong bots have been updated:
AQ v2.1.1
Leela 0.11.0
Ray 4.32
Pachi 12.00

Leela 0.11 is running and I'll wait for the official release of Pachi 12.

I'll take a look at the others, thank you.

Edit: Tried to compile Ray 4.32 without success. First I had to get some libs of cntk 2.1 although in the readme the cntk version is 2.3. Then I ran into the next error messages. The author is not much interested in making the installation easier, not even in keeping the readme up to date. That's no problem, but I'll leave Rn until it is easier to install.

[pnprog]

Edit: Tried to compile Ray 4.32 without success. First I had to get some libs of cntk 2.1 although in the readme the cntk version is 2.3. Then I ran into the next error messages. The author is not much interested in making the installation easier, not even in keeping the readme up to date. That's no problem, but I'll leave Rn until it is easier to install.

Yes, and sadly it's doing prejudice to this otherwise strong bot

It would be nice to have somebody who is well versed in unbuntu/apt/ppa/compilation/deb to implement a dedicated ppa for ubuntu and all usual go software. This ppa would include updated deb files for commonly used go program (Leela, Sabaki...). Those hard to compile/install programs like Ray would come in sort of containers or snap applications (with all dependencies included).

Not quite a come back of Hikarunix, but still a big improvement.