It is currently Thu May 23, 2019 2:34 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 266 posts ]  Go to page Previous  1 ... 8, 9, 10, 11, 12, 13, 14  Next
Author Message
Offline
 Post subject: Re: Engine Tournament
Post #201 Posted: Mon May 21, 2018 7:49 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
abcd_z wrote:
as0770 wrote:
I won't test every network, so I have to do a cut somewhere.
That's reasonable, but if you're going to do a tournament that tests the strength of Leela Zero it should include, at minimum, the most recent LZ network. Cut out other LZ networks if you have to.


Other networks have been played already, I don't save time and computing resources when I cut them out.

But you are right, it has been a long time since the last test so I currently run a new one.

Because of the training with ELF games the current 192x15 network will likely make progress for a while.

abcd_z wrote:
as0770 wrote:
When ELF came up Leela won 7% against it. Ten Networks later it wins 10%. I think this says much more about the progress than a 16 game tournament.
I have no idea what you mean by this.


When ELF was published Leelas best network won 7% against the ELF network. Yesterday, 11 days later, Leelas best network won 10% against the same ELF network.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #202 Posted: Mon May 21, 2018 1:08 pm 
Beginner

Posts: 12
Liked others: 5
Was liked: 0
Rank: 15k
as0770 wrote:
Other networks have been played already, I don't save time and computing resources when I cut them out.
Wait, are you reusing the results between tournaments?!

Quote:
But you are right, it has been a long time since the last test so I currently run a new one.

Because of the training with ELF games the current 192x15 network will likely make progress for a while.

Oh my god, you are reusing the results. That's why you don't want to use the results from a constantly-changing network.

Look, if you're running a tournament between game engines, you can't just reuse previous results. You have to use all-new matches, or the results aren't valid.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #203 Posted: Mon May 21, 2018 8:58 pm 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
abcd_z wrote:
You have to use all-new matches, or the results aren't valid.


It is not a ratinglist but a tournament. The results are exactly as valid as intended. If it doesn't match your prospects you have to watch for other sources or run your own tournament. But be aware: One run will take 20 days 24/7 if you do all-new matches.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #204 Posted: Tue May 22, 2018 3:20 am 
Beginner

Posts: 12
Liked others: 5
Was liked: 0
Rank: 15k
as0770 wrote:
It is not a ratinglist but a tournament.
Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.

To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #205 Posted: Tue May 22, 2018 6:04 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
abcd_z wrote:
as0770 wrote:
It is not a ratinglist but a tournament.
Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.

Now you've heard of one.
abcd_z wrote:
To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.

I am sorry that you feel misleaded. How can I name it in the future to avoid misleading?

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #206 Posted: Tue May 22, 2018 9:58 am 
Dies in gote

Posts: 40
Liked others: 2
Was liked: 14
Rank: EGF 1 kyu
KGS: finity
abcd_z wrote:
Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.
To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.


Although normal standards of politeness don't apply on internet forums, I would allow as0770 some credit (even quite a significant amount) for running this "tournament" before going into critique part. I have found this thread very interesting to follow and it has been a joy to check out newest results once they become available, even though it doesn't happen every day. However, running this kind of project on voluntary effort is a huge undertaking, so that is completely understandable. So thanks to as0770 from here!

Now it is true that I also had the impression that "tournament" was somehow plurar (like "engine tournaments", with each post representing one tournament), i.e. a new one is done every once in a while. So there is ambiguity in the term. Maybe the updates could say "here are updated results of the continuing tournament", or something to that effect.

However, despite abcd_z's vehement reaction to this, I would maybe argue, that if the engines are exact same version (like SomeBot 1.4.4 or OtherEngine 10), arranging a 10 game match between them every time is just reducing variance after a while, and not likely to uncover any deeper truths, so I would not put much effort into having GNU Go play against every other bot all the time. For Leela Zero and other actively progressing engines, however, there might be a need to re-run the newest version against the opposition every once in a while.

Of course that raises the question that if BotX won 2/10 against LZ #120, and then LZ #142 beats it 10-0, will it then be 10-0 or 18-2 for Leela vs. BotX. But if it's only a one match per opponent, then you can just overwrite the previous results.

Naturally a tournament is not an ELO comparison, as in the cases where only single game is played, it's about winning probabilities but the result is binary. But in a league, you could argue that the results are likely to even each other out.

One possible idea: Maybe do a matrix of engines, and accumulate games slowly into the matrix, with more games allotted for those where results are not as clear? With one row/column for single version, this would slowly build up into quite good comparison. And for GNU Go vs. LZ ELF you don't need to have 10 game match, one or two would be enough to see that yes indeed LZ ELF mostly wins.


This post by jokkebk was liked by 3 people: abcd_z, drmwc, Javaness2
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #207 Posted: Tue May 22, 2018 12:13 pm 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
jokkebk wrote:
abcd_z wrote:
Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.
To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.


Although normal standards of politeness don't apply on internet forums, I would allow as0770 some credit (even quite a significant amount) for running this "tournament" before going into critique part. I have found this thread very interesting to follow and it has been a joy to check out newest results once they become available, even though it doesn't happen every day. However, running this kind of project on voluntary effort is a huge undertaking, so that is completely understandable. So thanks to as0770 from here!


Thanks for your kind words. I did this kind of tournaments many years with chess engines. When AlphaGo came up I became very intersted in Go and Go engines. I know, and I stated several times, that a tournament of ~20 games for each AI won't give a statistically significant result. Beside that the strength of a Go engine depends a lot on the hardware. Some use GPU some don't. The same tournament on a better graphic card will lead to a totally different result. So my intention is not to find the truth about Go engines, simply because there is no real truth. My intention is to give an overview of the strength of gtp engines of all levels. And I want to do that without running tournaments 24/7.

In the beginning of each post I mentioned the "updates" of the tournament, I wasn't aware this can be misunderstood, I'll try to state that more clear in the future :)


jokkebk wrote:
I also had the impression that "tournament" was somehow plurar (like "engine tournaments", with each post representing one tournament)


The thread title is "Engine tournament" (singular) :)


jokkebk wrote:
For Leela Zero and other actively progressing engines, however, there might be a need to re-run the newest version against the opposition every once in a while.


New versions of an engine or networks play of course new games. I updated the Leela version quite a lot in the past. Now it is somewhere between ZEN and ELF so it don't make much sense to test every new network.


This post by as0770 was liked by: drmwc
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #208 Posted: Tue May 22, 2018 9:00 pm 
Beginner

Posts: 12
Liked others: 5
Was liked: 0
Rank: 15k
as0770 wrote:
I am sorry that you feel misleaded. How can I name it in the future to avoid misleading?
I would edit your very first post and include something like the following:
Quote:
Because the strength of go programs don't really change over time, and because it takes so long to run the matches, I reuse the results from previous tournaments. Each program runs once against all the current competitors when it's introduced, and those results are used for each subsequent tournament.

This also means that I won't be testing the current weights of Leela Zero, which changes constantly. Instead, I'm going to use the last LZ model of each size.
Feel free to copy and paste those paragraphs if you want.

Jokkebk makes a good point: go program strengths don't usually change much over time. LZ is the exception to this rule. With that in mind, I don't have any problems with you doing the tournament the way you've been doing it, as long as it's obvious to the readers that it's not exactly a standard tournament.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #209 Posted: Tue May 22, 2018 9:19 pm 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
abcd_z wrote:
as0770 wrote:
I am sorry that you feel misleaded. How can I name it in the future to avoid misleading?
I would edit your very first post and include something like the following:
Quote:
Because the strength of go programs don't really change over time, and because it takes so long to run the matches, I reuse the results from previous tournaments. Each program runs once against all the current competitors when it's introduced, and those results are used for each subsequent tournament.

This also means that I won't be testing the current weights of Leela Zero, which changes constantly. Instead, I'm going to use the last LZ model of each size.
Feel free to copy and paste those paragraphs if you want.

Jokkebk makes a good point: go program strengths don't usually change much over time. LZ is the exception to this rule. With that in mind, I don't have any problems with you doing the tournament the way you've been doing it, as long as it's obvious to the readers that it's not exactly a standard tournament.


I didn't mix different weights or different versions of one engine, in the future I'll explain updated tournaments better. But I don't see a reason to explain why I don't test all versions and networks.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #210 Posted: Tue May 22, 2018 10:38 pm 
Beginner

Posts: 12
Liked others: 5
Was liked: 0
Rank: 15k
as0770 wrote:
I didn't mix different weights or different versions of one engine
I... what? I never said you did.
as0770 wrote:
But I don't see a reason to explain why I don't test all versions and networks.
I didn't say that, either. I feel like you're reading something different from what I'm writing.

All I'm saying is that when you use the word "tournament", the natural assumption is that you're re-generating the matches each time. All you need to do to prevent this misunderstanding is to make a note at the beginning of your first post saying that you're reusing the results of each tournament. That's all.

EDIT: Wait, you mean that thing about Leela Zero? The reason I brought LZ up specifically was because, unlike all the other engines, LZ changes frequently, and it's easy for a reader to assume that the tests you're running on it are indicative of LZ's current strength.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #211 Posted: Wed May 23, 2018 7:17 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
abcd_z wrote:
as0770 wrote:
I didn't mix different weights or different versions of one engine
I... what? I never said you did.
as0770 wrote:
But I don't see a reason to explain why I don't test all versions and networks.
I didn't say that, either. I feel like you're reading something different from what I'm writing.

I know what you said. Everything is in order. From the discussion one could get the impression I would use different weights with one participant. I just stated this clear.

abcd_z wrote:
Wait, you mean that thing about Leela Zero? The reason I brought LZ up specifically was because, unlike all the other engines, LZ changes frequently, and it's easy for a reader to assume that the tests you're running on it are indicative of LZ's current strength.


No, the size of the network is specified. It is much easier to understand than the network names. Beside that when I publish the results the network will be quite up to date and one can also take a look at the date of a post to see how old it is.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #212 Posted: Thu May 24, 2018 2:37 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
For a change some results :-)

Updates:
-Leela Zero 0.15 with the 192x15 network 11057dd from 2018.05.20
-ELF OpenGo v0 now also with Leela Zero 0.15.

So the results in the Champions League are all new, other Leagues are not affected.

ELF performed better with LZ 0.15, maybe there was a problem with the first LZ version that was able to use the ELF network.

LZ 192x15 now clearly outperformes Zen 7. Beside the new network there were changes in the time management.

Zen won it's only game because of a ladder against ELF. There were no results affected by the komi of 6.5.

Champions League:
Code:
    1. ELF OpenGo v0 LZ 0.15           14/16
    2. Leela Zero 0.15 11057dd          9/16
    2. Zen 7                            1/16

League A:
Code:
    1. Zen 7                           11/16
    2. ELF OpenGo v0 LZ 0.13x (CPU)    11/16
    3. AQ 2.1.1                         8/16
    4. Leela 0.11                       6/16
    5. Leela Zero 0.13 128x10 (CPU)     4/16

League B:
Code:
    1. Leela Zero 0.13 128x10 (CPU)    22/24
    2. Leela 0.11.0                    19/24
    3. Rayon 4.6.0                     13/24
    4. Oakfoam 0.2.1 NG-06             12/24
    5. Leela Zero 128x6 (CPU)          12/24
    6. Hiratuka 10.37B (CPU)            4/24
    7. DreamGo 0.5.0                    2/24

League C:
Code:
    1. DreamGo 0.5.0                   18/24
    2. DarkForrest MCTS 1.0            16/24
    3. Pachi 12                        16/24
    4. Ray 9.0.1                       11/24
    5. DarkGo 1.0                      10/24
    6. Leela Zero 64x5 (CPU)           10/24
    7. Mogo 4.86                        3/24

League D:
Code:
    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League E:
Code:
    1. GNU Go 3.8                      18/20
    2. Hara 0.9                        11/20
    3. Dariush 3.1.5.7                  9/20
    4. Indigo 2009                      9/20
    5. Matilda 1.25                     9/20
    6. OnePunchGo 1.0                   4/20

League F:
Code:
    1. OnePunchGo 1.0                  17/20
    2. Aya 6.34                        16/20 
    3. JrefBot 081016-2022             12/20 
    4. Fudo Go 3.0                     10/20
    5. Iomrascálaí 0.3.2                4/20
    6. SimpleGo 0.4.3                   1/20

League G:
Code:
    1. SimpleGo 0.4.3                  17/20
    2. AmiGo 1.8                       15/20
    3. Crazy Patterns 0008-13          13/20
    4. Marcos Go 1.0                   11/20
    5. Beancounter 0.1                  4/20
    6. Stop 0.9-005                     0/20

League H:
Code:
    1. Stop 0.9-005                    18/20
    2. GoTraxx 1.4.2                   14/20
    3. V-Run 1                          9/20
    4. CopyBot 0.1                      8/20
    5. Leela Zero 32x4 (CPU)            7/20
    6. Brown 1.0                        4/20



Configuration:
Please note this is a continuous tournament, new versions will replace old versions, new AI's will be added. Matches in existence will not be replayed as long as the conditions remain the same.

Champions League: 1h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 120 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League A-B: 1h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League C-H: 1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Amigo: amigogtp
AQ: AQ
Code:
aq_config.txt:
-main time[sec] =3600
-time controll =true
-japanese rule =true

Aya: Aya.exe –mode gtp –level max
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
Dariush: DarGTP.exe –level 10
DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 750 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
ELF OpenGo: leelaz_015 --gtp --threads 4 --weights /path/to/network_elf --noponder
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
Fuego: fuego.exe –config fuego.cfg
Code:
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
GoTraxx: GoTraxx.exe
Hara: hara
Hiratuka: Hiratuka-19×19.exe -po 50000
IndiGo: Indigo.exe -gtp
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 50000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_xxxxxx --noponder
Matilda: matilda
Code:
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1

Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Michi C-2: michi gtp
Code:
ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Oakfoam: oakfoam -c nicego-cnn-06.gtp
Code:
nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

OnePunchGo: taskset -c 0 OPG
Code:
TimeInfo.h (Modified to play 15sec/move because OPG requires time_left command which is not supported by twogtp and Sabaki)
timeAvailable = 15;

Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachidcnn -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
PhoenixGo: leelaz_for_phoenix_go --gtp --threads 4 --weights /path/to/network_phoenix_go --noponder
Ray: ray --time 3600 --thread 1 --no-debug
Rayon: rayon --thread 4 --no-debug
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp
V-Run: python3 /path/to/V-Run
Zen: gtp4zen.exe -z 7 -t 4 -T 30 -s 10000 -i 1000

Links:
Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
ELF OpenGo: http://zero.sjeng.org/ Find the network 62b5417b and use it with the latest Leela Zero.
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/winnt/game/fh673259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/winnt/game/fh688349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: http://zero.sjeng.org/
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: https://lifein19x19.com/forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
OnePunchGo https://github.com/AlexKent3141/OnePunchGo
Orego: https://sites.google.com/a/lclark.edu/drake/research/orego
Pachi: https://github.com/pasky/pachi
PhoenixGo: https://github.com/yenw/LeelaZero_PhoenixGo
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/
V-Run: https://github.com/Fioelkais/MCTS
Zen via gtp4zen (info): https://www.lifein19x19.com/forum/viewtopic.php?f=18&t=15599

Best,
Alex


Last edited by as0770 on Sat May 26, 2018 12:15 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #213 Posted: Sat May 26, 2018 12:13 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
Update: PhoenixGo, in fact LZ 0.14 with the PhoenixGo 256x19 network, added to the Champions League.
Edith: CPU version of PhoenixGo added to League A.

Champions League:
Code:
    1. ELF OpenGo v0 LZ 0.15           20/24
    2. Leela Zero 0.15 11057dd         14/24
    3. PhoenixGo v1 LZ 0.14x           12/24
    4. Zen 7                            2/24

League A:
Code:
    1. ELF OpenGo v0 LZ 0.13x (CPU)    15/20
    2. Zen 7                           14/20
    3. AQ 2.1.1                        11/20
    4. PhoenixGo v1 LZ 0.14x (CPU)      7/20
    5. Leela 0.11                       7/20
    6. Leela Zero 0.13 128x10 (CPU)     6/20

League B:
Code:
    1. Leela Zero 0.13 128x10 (CPU)    22/24
    2. Leela 0.11.0                    19/24
    3. Rayon 4.6.0                     13/24
    4. Oakfoam 0.2.1 NG-06             12/24
    4. Leela Zero 128x6 (CPU)          12/24
    6. Hiratuka 10.37B (CPU)            4/24
    7. DreamGo 0.5.0                    2/24

League C:
Code:
    1. DreamGo 0.5.0                   18/24
    2. DarkForrest MCTS 1.0            16/24
    2. Pachi 12                        16/24
    4. Ray 9.0.1                       11/24
    5. DarkGo 1.0                      10/24
    5. Leela Zero 64x5 (CPU)           10/24
    7. Mogo 4.86                        3/24

League D:
Code:
    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League E:
Code:
    1. GNU Go 3.8                      18/20
    2. Hara 0.9                        11/20
    3. Dariush 3.1.5.7                  9/20
    3. Indigo 2009                      9/20
    5. Matilda 1.25                     9/20
    6. OnePunchGo 1.0                   4/20

League F:
Code:
    1. OnePunchGo 1.0                  17/20
    2. Aya 6.34                        16/20 
    3. JrefBot 081016-2022             12/20 
    4. Fudo Go 3.0                     10/20
    5. Iomrascálaí 0.3.2                4/20
    6. SimpleGo 0.4.3                   1/20

League G:
Code:
    1. SimpleGo 0.4.3                  17/20
    2. AmiGo 1.8                       15/20
    3. Crazy Patterns 0008-13          13/20
    4. Marcos Go 1.0                   11/20
    5. Beancounter 0.1                  4/20
    6. Stop 0.9-005                     0/20

League H:
Code:
    1. Stop 0.9-005                    18/20
    2. GoTraxx 1.4.2                   14/20
    3. V-Run 1                          9/20
    4. CopyBot 0.1                      8/20
    5. Leela Zero 32x4 (CPU)            7/20
    6. Brown 1.0                        4/20



Configuration:
Please note this is a continuous tournament, new versions will replace old versions, new bots will be added. Matches in existence will not be replayed as long as the conditions remain the same.

Champions League: 1h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 4 -komi 7.5 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League A-B: 1h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League C-H: 1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Amigo: amigogtp
AQ: AQ
Code:
aq_config.txt:
-main time[sec] =3600
-time controll =true
-japanese rule =true

Aya: Aya.exe –mode gtp –level max
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
Dariush: DarGTP.exe –level 10
DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 750 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
ELF OpenGo: leelaz_015 --gtp --threads 4 --weights /path/to/network_elf --noponder
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
Fuego: fuego.exe –config fuego.cfg
Code:
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
GoTraxx: GoTraxx.exe
Hara: hara
Hiratuka: Hiratuka-19×19.exe -po 50000
IndiGo: Indigo.exe -gtp
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 50000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_xxxxxx --noponder
Matilda: matilda
Code:
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1

Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Michi C-2: michi gtp
Code:
ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Oakfoam: oakfoam -c nicego-cnn-06.gtp
Code:
nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

OnePunchGo: taskset -c 0 OPG
Code:
TimeInfo.h (Modified to play 15sec/move because OPG requires time_left command which is not supported by twogtp and Sabaki)
timeAvailable = 15;

Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachidcnn -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
PhoenixGo: leelaz_for_phoenix_go --gtp --threads 4 --weights /path/to/network_phoenix_go --noponder
Ray: ray --time 3600 --thread 1 --no-debug
Rayon: rayon --thread 4 --no-debug
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp
V-Run: python3 /path/to/V-Run
Zen: gtp4zen.exe -z 7 -t 4 -T 30 -s 10000 -i 1000

Links:
Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
ELF OpenGo: http://zero.sjeng.org/ Find the network 62b5417b and use it with the latest Leela Zero.
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/winnt/game/fh673259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/winnt/game/fh688349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: http://zero.sjeng.org/
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: https://lifein19x19.com/forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
OnePunchGo https://github.com/AlexKent3141/OnePunchGo
Orego: https://sites.google.com/a/lclark.edu/drake/research/orego
Pachi: https://github.com/pasky/pachi
PhoenixGo: https://github.com/yenw/LeelaZero_PhoenixGo
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/
V-Run: https://github.com/Fioelkais/MCTS
Zen via gtp4zen (info): https://www.lifein19x19.com/forum/viewtopic.php?f=18&t=15599

Best,
Alex


Last edited by as0770 on Thu May 31, 2018 5:56 am, edited 2 times in total.
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #214 Posted: Sun May 27, 2018 4:16 am 
Lives with ko

Posts: 243
Location: Spain
Liked others: 149
Was liked: 24
Rank: Low
GD Posts: 10
as0770 wrote:
There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.

_________________
Sum ergo non ero.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #215 Posted: Sun May 27, 2018 8:02 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
luigi wrote:
as0770 wrote:
There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.


It happend to be a komi of 6.5 and I won't replay all games.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #216 Posted: Tue May 29, 2018 11:23 am 
Lives with ko

Posts: 243
Location: Spain
Liked others: 149
Was liked: 24
Rank: Low
GD Posts: 10
as0770 wrote:
luigi wrote:
as0770 wrote:
There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.


It happend to be a komi of 6.5 and I won't replay all games.

You can still make it 7.5 hereafter.

_________________
Sum ergo non ero.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #217 Posted: Tue May 29, 2018 1:37 pm 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
luigi wrote:
You can still make it 7.5 hereafter.


I don't want to have different settings in one tourney. One day I'll get a new computer and will start from scratch :)

The statistical fluctuation has much more influence on the result than the komi.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #218 Posted: Wed May 30, 2018 8:40 pm 
Lives with ko
User avatar

Posts: 281
Liked others: 93
Was liked: 147
Rank: OGS 7 kyu
luigi wrote:
You can still make it 7.5 hereafter.
Curious about that: how to do?

as0770 wrote:
One day I'll get a new computer and will start from scratch
You may have to implement a 6.5komi league, and a 7.5komi league. Some bots are just not supposed to play at 7.5komi (AQ is the strongest of those) and other bots are not supposed to play at 6.5komi (Leela, LeelaZero, Ray... the vast majority of value network bots). It's probably that Japanese bots are more suitable for 6.5komi, so maybe Zen also plays better on 6.5komi, but it a case by case checking.

Bots without value network can probably play in both leagues, but those are expected to be much weaker bots. So maybe 2 leagues A (for 6.5komi and 7.5komi) and others leagues (B, C, D...) of weaker bots that can stay at 6.5komi so you don't have to re-run those matches.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #219 Posted: Thu May 31, 2018 5:53 am 
Lives with ko

Posts: 156
Liked others: 15
Was liked: 23
Rank: Beginner
pnprog wrote:
luigi wrote:
You can still make it 7.5 hereafter.
Curious about that: how to do?

as0770 wrote:
One day I'll get a new computer and will start from scratch
You may have to implement a 6.5komi league, and a 7.5komi league. Some bots are just not supposed to play at 7.5komi (AQ is the strongest of those) and other bots are not supposed to play at 6.5komi (Leela, LeelaZero, Ray... the vast majority of value network bots). It's probably that Japanese bots are more suitable for 6.5komi, so maybe Zen also plays better on 6.5komi, but it a case by case checking.

Bots without value network can probably play in both leagues, but those are expected to be much weaker bots. So maybe 2 leagues A (for 6.5komi and 7.5komi) and others leagues (B, C, D...) of weaker bots that can stay at 6.5komi so you don't have to re-run those matches.


What can we expect by a change to komi 7.5?

In the Champions League there are 3 Leela networks. They don't care about the komi, all results of them are due to resignation. So playing with a komi of 7.5 would have lead to the same results, right? 4th bot is Zen, it won two games with white by resignation of the opponent in completely lost positions. The other games of Zen where lost, so also no influence by the komi. Even though I doubt it will make any difference I'll change the komi for the Champions League to 7.5 from now on since it would have lead to the same results in the previous games.

In League A it doesn't look very different. I am very sure the inaccuracy in the results is many times smaller than the statistical fluctuation of a 20 games tournament.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #220 Posted: Thu May 31, 2018 7:06 am 
Lives with ko
User avatar

Posts: 281
Liked others: 93
Was liked: 147
Rank: OGS 7 kyu
as0770 wrote:
What can we expect by a change to komi 7.5?
I think Zen may drop out of the champion league and A league if komi is 7.5pt. But we won't know until we try.

Quote:
In the Champions League there are 3 Leela networks. They don't care about the komi, all results of them are due to resignation. So playing with a komi of 7.5 would have lead to the same results, right?
Not necessarily. In fact, Leela Zero should only lose by resignation, if she lose 0.5pt, then there is definitively something wrong. Same goes for AQ, Leela and Ray.

But let's take and example, LZ (black) vs Zen (white), komi=6.5pt
20 moves before the end of the game, black (LZ) is about to win by 0.5pt, seems good for LZ. But because LZ is trained for 7.5pt komi, she believes she will loose by 0.5 (her value network gives a 23% win rate), so she starts desperate attempts at complicating the game, play weird end game, lose points here and there, then her win rate drops bellow 20% and finally she resigns.

Zen was suppose to loose, but he ends up winning, Leela Zero resigns from a completely lost position.

Quote:
4th bot is Zen, it won two games with white by resignation of the opponent in completely lost positions.
Difficult to say, maybe Zen was losing by 0.5pt 20 moves before then end, and the opponent, believing it was losing, throw the game away?

Quote:
The other games of Zen where lost, so also no influence by the komi.
For those ones, I agree!

Quote:
Even though I doubt it will make any difference I'll change the komi for the Champions League to 7.5 from now on since it would have lead to the same results in the previous games.
If you do so, I encourage you to replay all the matches of the league, but, sure, that's a lot of time, energy (and money).

Quote:
In League A it doesn't look very different. I am very sure the inaccuracy in the results is many times smaller than the statistical fluctuation of a 20 games tournament.
Well, I will be surprised if Zen can score 14 wins.

You know, even between bots that are trained at 7.5pt komi, it could make a difference. Maybe some of them are stronger "as black", and they loose that "edge" because of the 6.5pt komi? Who know?

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 266 posts ]  Go to page Previous  1 ... 8, 9, 10, 11, 12, 13, 14  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: emeraldemon, Majestic-12 [Bot] and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group