Engine Tournament

[Revilo]

The first step will be to find the differences between all engines, (no matter where to start ) and then calibrate it with the reference engines of known strength. The 2nd step could be to do the same in faster games, then one could create a matrix of strength on different hardware.

Hello Alex,

you could have a look into BayesElo here: http://www.remi-coulom.fr/Bayesian-Elo/. It is a widely used tool in chess programming to determine strength differences between engines. I haven't worked with it much, but basically, you feed it game results in PGN format and receive ELO ratings calibrated around an initial ELO rating. It will also calculate a confidence interval (called the error bar), which will become narrower the more games are available.

Because BayesELO doesn't care about the chess moves but only for the metadata, you could write a script to convert your SGFs into PGNs without moves.

I don't know if GoGUI has good support for engine tournaments. In chess, there are tools that can run tournaments automatically - so you could set up a dedicated machine, configure the tournament, press "Start" and come back a week later to view at the BayesElo results...

[as0770]

The first step will be to find the differences between all engines, (no matter where to start ) and then calibrate it with the reference engines of known strength. The 2nd step could be to do the same in faster games, then one could create a matrix of strength on different hardware.

Hello Alex,

you could have a look into BayesElo here: http://www.remi-coulom.fr/Bayesian-Elo/. It is a widely used tool in chess programming to determine strength differences between engines. I haven't worked with it much, but basically, you feed it game results in PGN format and receive ELO ratings calibrated around an initial ELO rating. It will also calculate a confidence interval (called the error bar), which will become narrower the more games are available.

Because BayesELO doesn't care about the chess moves but only for the metadata, you could write a script to convert your SGFs into PGNs without moves.

I don't know if GoGUI has good support for engine tournaments. In chess, there are tools that can run tournaments automatically - so you could set up a dedicated machine, configure the tournament, press "Start" and come back a week later to view at the BayesElo results...

I think for meaningful ELO ratings you would need much more games, especially because there are some huge gaps in strength between some of the go engines.

SmartGo is a nice GUI for engine tournaments, but there are some flaws under Linux with wine. With GoGui you need a tool called twogtp which is something like cutechessqli. Unfortunately it only supports engine matches so you have to setup the rounds of a tournament manually. You have to fiddle with command line options, but once you created the strings it is easy to use.

[as0770]

Update with Ray 8.0.1:

League A:

Code: Select all

    1. Leela 0.6.2                  22:2
    2. Hiratuka 10.37               21:3
    3. PachiUCT 11.99               15:9
    4. Ray 8.0.1                    13:11
    5. Fuego 1.1                     7:17
    6. MoGo 3                        5:19
    7. GNU Go 3.8                    1:23

League B:

Code: Select all

    1. GNU Go 3.8                   18:2
    2. Indigo 2009                  12:8
    3. Dariush 3.1.5.7              11:9
    4. Aya 6.34                     10:10
    5. Fudo Go 3.0                   9:11
    6. AmiGoGtp 1.5                  0:20

fuego.exe –config fuego.cfg
---
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
taskset -c 0 Hiratuka-19×19.exe -po 15000
leela –gtp –threads 1 –noponder
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
gnugo –mode gtp –level 10
Indigo.exe -gtp
ray --time 3600 --thread 1 --no-debug

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

[as0770]

Update with Hara 0.9 (with hacked time management):

https://github.com/antoniogarro/Hara

I had to modify it a little bit because it used too much time.

League A:

Code: Select all

    1. Leela 0.6.2                  22:2
    2. Hiratuka 10.37               21:3
    3. PachiUCT 11.99               15:9
    4. Ray 8.0.1                    13:11
    5. Fuego 1.1                     7:17
    6. MoGo 3                        5:19
    7. GNU Go 3.8                    1:23

League B:

Code: Select all

    1. GNU Go 3.8                   21:3
    2. Hara 0.9                     16:8
    3. Indigo 2009                  14:10
    4. Dariush 3.1.5.7              13:11
    5. Aya 6.34                     11:13
    6. Fudo Go 3.0                   9:15
    7. AmiGoGtp 1.5                  0:24

amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
fuego.exe –config fuego.cfg
---
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
gnugo –mode gtp –level 10
hara
taskset -c 0 Hiratuka-19×19.exe -po 15000
Indigo.exe -gtp
leela –gtp –threads 1 –noponder
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
ray --time 3600 --thread 1 --no-debug

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

[Hane]

New engine Matilda by Gonçalo

https://github.com/gonmf/matilda

[as0770]

New engine Matilda by Gonçalo

https://github.com/gonmf/matilda

League A:

Code: Select all

    1. Leela 0.6.2                  22:2
    2. Hiratuka 10.37               21:3
    3. PachiUCT 11.99               15:9
    4. Ray 8.0.1                    13:11
    5. Fuego 1.1                     7:17
    6. MoGo 3                        5:19
    7. GNU Go 3.8                    1:23

League B:

Code: Select all

    1. GNU Go 3.8                   25:3
    2. Hara 0.9                     17:11
    3. Dariush 3.1.5.7              16:12
    4. Indigo 2009                  16:12
    5. Matilda 1.20                 14:14
    6. Aya 6.34                     13:15
    7. Fudo Go 3.0                  11:17
    8. AmiGoGtp 1.8                  0:28

Configuration:

amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
fuego.exe –config fuego.cfg
---
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
gnugo –mode gtp –level 10
hara
taskset -c 0 Hiratuka-19×19.exe -po 15000
Indigo.exe -gtp
leela –gtp –threads 1 –noponder
matilda
---
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1
---
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
ray --time 3600 --thread 1 --no-debug

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

[Hane]

Leela 0.7.0

http://www.sjeng.org/leela.html

[as0770]

Leela 0.7.0

http://www.sjeng.org/leela.html

Leela vs. Hiratuka:

Code: Select all

    1. Leela 0.7.0                  18/24
    2. Hiratuka 10.37B               6/24

League A:

Code: Select all

    1. Leela 0.7.0                  22/24
    2. Hiratuka 10.37B              21/24
    3. PachiUCT 11.99               15/24
    4. Ray 8.0.1                    13/24
    5. Fuego 1.1                     7/24
    6. MoGo 3                        5/24
    7. GNU Go 3.8                    1/24

League B:

Code: Select all

    1. GNU Go 3.8                   25/28
    2. Hara 0.9                     17/28
    3. Dariush 3.1.5.7              16/28
    4. Indigo 2009                  16/28
    5. Matilda 1.20                 14/28
    6. Aya 6.34                     13/28
    7. Fudo Go 3.0                  11/28
    8. AmiGoGtp 1.8                  0/28

Configuration:

amigogtp.exe
Aya.exe –mode gtp –level max
DarGTP.exe –level 10
taskset -c 1 fudo –boardsize=19 –komi=6.5
fuego.exe –config fuego.cfg
---
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1
---
gnugo –mode gtp –level 10
hara
taskset -c 0 Hiratuka-19×19.exe -po 15000
Indigo.exe -gtp
leela –gtp –threads 1 –noponder
matilda
---
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1
---
mogo –19 –pondering 0 –nbThreads 1
pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
ray --time 3600 --thread 1 --no-debug

TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

[gonmf]

Hello, I am the author of the Matilda bot, this is my first post.

I noticed you are launching matilda without any options, and this will by default use as many threads as available cores. I noticed that the other bots are playing on one thread and as such they may be playing in unfair ground.

Best wishes,
Gonçalo

[as0770]

Hello, I am the author of the Matilda bot, this is my first post.

I noticed you are launching matilda without any options, and this will by default use as many threads as available cores. I noticed that the other bots are playing on one thread and as such they may be playing in unfair ground.

Best wishes,
Gonçalo

Thanks for the hint. Matilda was playing on one core, in my post above the info was missing and I now added the changes in matilda.h.

Is it possible to configure it at the command line or with a configuration file?

Best wishes and thanks for sharing your bot,
Alex

[q30]

Go engines rate (sparrings by GTP on one computer with a half of processor and memory limits for each engine):
1. Ray
2&3. Pachi & Leela
4. MoGo
5&6. Many Faces of Go & FueGo


Go engines non default compiling parameters:
Pachi 32a031c - DOUBLE_FLOATING=1; prefix=/usr; spatial_hash_bits 30; gogod-handikgspachi-iter patterns (copying in engine dir. prob and spat files)
MoGo 4.86 - noMpi
FueGo 1.1.SVN [fix_stage] - prefix=/usr


Go engines start up parameters:

Ray 8.0.1 - --komi 7.5 --const-time 120 --thread 2 --pondering --tree-size 262144

Pachi 32a031c (working dir. with patterns and opening book) - -f book.dat -t 120 threads=2,max_tree_size=4096,prior=pattern=160

leela_062_linux_x64 - --gtp --threads 2

MoGo 4.86 (working dir. with patterns and opening book) - --19 --time 120 --pondering 1 --nbThreads 2 --collectorLimitTreeSize 800000

MoGo 3.0.0[mod] (working dir. with patterns) - --19 --time 120 --pondering 1 --nbThreads 2 --collectorLimitTreeSize 800000 --loadOB totalOB.ob --limitTreeSize 1000000 --randomMode 44 --saveMode on --firstPlayUrgency 1.1 --cfgMode 2 --metaBlockMode 6 --blockMode 3 --distance1 3 --distance2 2 --freeLocationSize 30 --zoneMode 1 --zoneModeStopNumber 200 --undoMode 0 --accelerateMode 1 --firstMoveFixed 4 --shishoCheckMode 1 --uctUpdateMode -1

MoGo 3.0.0[mod] 32bits (working dir. with patterns and opening book) - --19 --time 120 --pondering 1 --nbThreads 2 --collectorLimitTreeSize 400000 --loadOB totalOB.ob --limitTreeSize 1000000 --randomMode 44 --saveMode on --firstPlayUrgency 1.1 --cfgMode 2 --metaBlockMode 6 --blockMode 3 --distance1 3 --distance2 2 --freeLocationSize 30 --zoneMode 1 --zoneModeStopNumber 200 --undoMode 0 --accelerateMode 1 --firstMoveFixed 4 --shishoCheckMode 1 --uctUpdateMode -1

FueGo 1.1.SVN [fix_stage] (working dir. with opening book) - --config $HOME/FueGo_server.cfg
where FueGo_server.cfg file content:
uct_param_search number_threads 2
uct_param_search lock_free 1
uct_max_memory 7000000000
uct_param_player reuse_subtree 1
uct_param_player ponder 1
go_param timelimit 120
uct_param_globalsearch live_gfx 1
uct_param_globalsearch territory_statistics 1
uct_param_search live_gfx sequence
uct_param_search live_gfx_interval 9000

Many Faces of Go 12 - New Game windows content:
Ranked Game
Board Size: 19
Handicap: 0
Komi: 7.5
Chinese
Periods: 4444
Stones: 1
Time: 0:04
Configure...
Folder: Z:\usr\share\gogui\lib
Command: java -jar gogui-client.jar localhost 55555
White: Base Time 00:00; 2 Kyu
Black: Base Time 44:44; GTP Player
or
White: Base Time 44:44; GTP Player
Black: Base Time: 00:00; 2 Kyu

Engines. Games.

May be it will be another rate, if someone will let know, how to change Leela's thinking time...

[q30]

Hello, I am the author of the Matilda bot, this is my first post.

I noticed you are launching matilda without any options, and this will by default use as many threads as available cores. I noticed that the other bots are playing on one thread and as such they may be playing in unfair ground.

Best wishes,
Gonçalo

I shall sparring Matilda with other engines above. Can You recommend today any game strengthening compilation and start up parameters for 2 min/move, 2 threads and max 8 GB of memory use?

Thanks in advance.

[as0770]

May be it will be another rate, if someone will let know, how to change Leela's thinking time...

You can set Leela's (and most other gtp engines) thinking time via gtp.

I use twogtp with this options TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

The gtp command looks like this: time_settings 3600 0 0

Best,
Alex

[q30]

You can set Leela's (and most other gtp engines) thinking time via gtp.

I use twogtp with this options TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

The gtp command looks like this: time_settings 3600 0 0

Best,
Alex

Yes, I have found, that it is possible to change Leela's thinking time by adding to gogui-twogtp.jar time control start up parameter, and I have use: "-time 1s+2/1" (details are in Games link or this message).

Go engines rate (sparrings by GTP on one computer with a half of processor and memory limits for each engine):

League A

• 1. Leela

League B

• 2. Ray
• 3. Pachi
• 4. MoGo

Go engines non default compiling and start up parameters.

Engines. Games.

[q30]

Hello, I am the author of the Matilda bot, this is my first post.

I noticed you are launching matilda without any options, and this will by default use as many threads as available cores. I noticed that the other bots are playing on one thread and as such they may be playing in unfair ground.

Best wishes,
Gonçalo

Matilda - Pachi: 0 - 4;
Matilda - MoGo: 0 - 4;
Matilda - FueGo: 0 - 4.

Matilda start up parameters:

Code: Select all

 -t 0+2m/1 --think_in_opt_time --memory 8000 --threads 2 -l ewi --log_dest o

Matilda precompilation set up file (./src/inc/matilda.h) content:

Code: Select all

#define BOARD_SIZ 19
#define DEFAULT_KOMI 15
#define DEFAULT_UCT_MEMORY 7000
#define LATENCY_COMPENSATION 270
#define DEFAULT_DATA_PATH "./data/"
#define MATILDA_RELEASE_MODE 1
#define DEFAULT_NUM_THREADS 0
#define MAXIMUM_NUM_THREADS 64
#define CAN_RESIGN 1
#define UCT_EXPANSION_DELAY 1
#define DEFAULT_TIME_PER_TURN 100000

Changes in ./src/inc/mcts.h:

Code: Select all

#define UCT_RESIGN_WINRATE 0.25

Go engines non default compiling and start up parameters.

Matilda is very weak engine.
Recommendation: take MoGo (it was made for dissertation too and its development was left many years ago) and upgrade it!
May be it will be enough to update OB and patterns files and set up start up parameters:

--gobanSize
--playerName
--time

--resignThreshold

--undoMode
--useServer
--useSimpleServerVersion
--nbSimulationsByNodeServer
--serverPort

--bufferSizeForEachMachine
--nbThreads

--test
--limitTreeSize
--collectorLimitTreeSize
--openingTestMode
--openingTestThreshold
--randomThreshold
--gobanRandomThreshold
--thresholdSelfAtari
--debugMode
--pondering

--useColor
--dontDisplay
--showBigGobanStringNumber

--randomMode
--randomMode2
--srandByTime
--mixRandomModeRatio
--mixRandomMode1
--mixRandomMode2

--openingModes
--openingMode
--openingMoveNumber
--useOpeningDatabase
--firstMoveFixed
--minimumDatabaseUse
--databaseName
--name
--openTestThreshold

--imitateMode
--imitateMoveNumber

--logShift
--invShift
--alpha
--beta
--mixThreshold
--alpha2
--meta
--correlatedMode
--alphaSqrt

--addingOwnSimulations
--multiplyingOwnSimulations
--typeOfComputation

--p1
--p2
--numUpdateFormulaeMoveSelector

--firstPlayUrgency
--urgencySelectorMode
--firstPlanUrgencyByGrandFather


--allowDoublePassInTree
--nbChildsAlreadyDone
--firstPlayUrgencyByGrandFather
--earlyCut
--relativeDepthMax

--uctUpdateMode

--keepTreeIfPossible

--fastRand
--earlyStop
--stopDifference
--recheckEyeMode
--isSelfAtariMode
--shishoLikeMoveForced
--nbTotalSimulationsForEachNode
--nbSimulationsForEachNode
--fillRangeFirstMode
--uselessMode

--accelerateMode
--nbTotalSimulations
--shishoCheckMode
--komi2

--zoneModeStopNumber
--blockMode
--metaBlockMode
--zoneMode
--freeLocationSize
--distance1
--distance2
--grDistance1
--grDistance2
--cfgMode
--nearMode
--sizeNear
--nearMode2

--defaultNew19
--defaultNew9
--defaultNew13
--19
--9
--13
--displayGames
--initiateByFiles
--player0
--player1
--minNumberOfSimulationsOfResign
--mercyThresholdDivider
--scoreMode
--factorForSide

--loadOB

Engines. Games.