Engine Tournament

as0770 · **#161**

Updates: DarkForrest is relegated to League B and Dream Go made it into League A. Also the Leela vs AQ match was replayed with the latest versions. Surprisingly (for me) Leela 0.11 was able to strike back after Leela 0.11 Beta lost 4-12 against AQ 2.0.1.

Unfortunately AQ doesn't work with Rayon and Oakfoam. One of the engines will crash wenn running on one GPU. So for now AQ can't play in League A.

Leela vs. AQ

Code:

    1. Leela 0.11.0                     9/16
    2. AQ 2.1.1                         7/16

League A:

Code:

    1. Leela 0.11.0                    18/20
    2. Rayon 4.6.0                     15/20
    3. Oakfoam 0.2.1 NG-06             12/20
    4. Hiratuka 10.37B (CPU)            7/20
    5. Leela Zero 0.11 5773f44c         6/20
    6. DreamGo 0.5.0                    2/20

League B:

Code:

    1. DreamGo 0.5.0                   15/20
    2. DarkForrest MCTS 1.0            12/20
    3. Pachi 11.99                     12/20
    4. DarkGo 1.0                      10/20
    5. Ray 9.0.1                        9/20
    6. Mogo 4.86                        2/20

League C:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:

Code:

    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:

Code:

    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:

Code:

    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:

Links:

Best,
Alex

q30 · **#162**

as0770 wrote:

q30 wrote:

You are quite right, if there is the same engine sparring. But even if there will be 2 simple MC engines (which will in sparring demonstrate mentioned by You chances with time on move --> 0), it may be difference in strength (i.e. in chances) dependent on time control because of difference in best move choice algorithm (and especially more complex engines with more complex algorithms).
You can try to compare 2 engines (with close strength levels) results with time and thread control, that You have used for league B-F, and results of these engines sparring with 2' per move and 4 threads...

You don't get the point. The statistical fluctuation is way too high to meassure little differences in strength. I won't play hundreds of games to prove you wrong.
Once again: This are two matches with the same engines and the same conditions:

as0770 wrote:

Pachi vs. Hiratuka 8:8
Pachi vs. Hiratuka 2:14

This discussion doesn't make any sense. No more replies by me.

This result only proves, that time control was very small for these (or one of these) engines, so games were very randomness...

as0770 · **#163**

Yet another Leela Zero Update in League A with a network from last Sunday. Although its learning progress seems to decrease, it made a big step in the last two weeks, it was even able to win one of four game against Leela 0.11.0:

Leela vs. AQ

Code:

    1. Leela 0.11.0                     9/16
    2. AQ 2.1.1                         7/16

League A:

Code:

    1. Leela 0.11.0                    17/20
    2. Leela Zero 0.11 cde9c8d4        13/20
    3. Rayon 4.6.0                     13/20
    4. Oakfoam 0.2.1 NG-06             12/20
    5. Hiratuka 10.37B (CPU)            4/20
    6. DreamGo 0.5.0                    1/20

League B:

Code:

    1. DreamGo 0.5.0                   15/20
    2. DarkForrest MCTS 1.0            12/20
    3. Pachi 11.99                     12/20
    4. DarkGo 1.0                      10/20
    5. Ray 9.0.1                        9/20
    6. Mogo 4.86                        2/20

League C:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:

Code:

    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:

Code:

    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:

Code:

    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:

Links:

Best,
Alex

Vargo · **#164**

Hello,
First, let me say that I'm a huge fan of your engine tournament.

Concerning AQ 2.1.1 v. Leela, the result depends a lot on the harware, and particularly on the GPU.

A GeForce 1080Ti is roughly 2 or 2.5 times more powerful than a 840M, that means AQ is probably one stone stronger on a 1080Ti than on a 840M.
With 2 GPUs, AQ is certainly 2 stones stronger than on a 840M.
If I'm not mistaken, Leela zero can handle multiple GPUs, but Leela011 OpenCL can't. So, on a gaming PC with 2 GPUs, AQ should be 2 stones stronger than Leela OpenCL, whereas on a standard PC, your tournament has shown that they're about even.

To test this, I've run matches (16 games) between AQ 2.1.1 and Leela011 OpenCL : time_settings 900 0 0 (same as in CGOS server) pondering off for both (Sabaki 033.3 used for all the games)

Even games_____GPU: 1x1080Ti____CPU: i7 6700K____RAM: 32 GB
H2 games_______GPU: 2x1080Ti____CPU: i9 7920X____RAM: 64 GB

Results

Even games : AQ 2.1.1 v. Leela011 OpenCL --------> AQ: 13/16 , L011: 3/16
H2 games : AQ 2.1.1 (W) v. Leela011 OpenCL (B)--->AQ: 9/16 , L011: 7/16

Quote:

Unfortunately AQ doesn't work with Rayon and Oakfoam...

I've run games between AQ and Rayon or others, it works well with Sabaki. The problem is that Sabaki doesn't handle consecutive matches automatically. You have to run one game after another, I don't think you can tell Sabaki to run automatically 16 consecutive games between X and Y, save the games, and at the end, tell the score of the 16 games match. If someone knows how to do it, tell me, I'd be interested.

Thanks for your engine tournament, keep up the good work, it's very interesting :clap:

as0770 · **#165**

Vargo wrote:

Quote:

Unfortunately AQ doesn't work with Rayon and Oakfoam...

I've run games between AQ and Rayon or others, it works well with Sabaki. The problem is that Sabaki doesn't handle consecutive matches automatically. You have to run one game after another

I gonna try Sabaki, but I think it is a gpu memory conflict. Even running both engines in console makes one crash. I think I need to update my computer...

Thanks for your results. I think in Go you can't define one best engine, because, like you said, the strength depends a lot on the hardware.

almost_sente · **#166**

Nice, the next version of Leela Zero has just been officially released!

https://github.com/gcp/leela-zero/releases

Will be interesting to see how much the improved search algorithm and speed help it. How many playouts did they usually get with your settings?

Quote:

Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder

as0770 · **#167**

almost_sente wrote:

Nice, the next version of Leela Zero has just been officially released!

https://github.com/gcp/leela-zero/releases

Will be interesting to see how much the improved search algorithm and speed help it. How many playouts did they usually get with your settings?

Quote:

Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder

Thanks for the hint

Leela Zero 0.12 starts a 1h/game with 50sec/move what is around 17500 playouts.

almost_sente · **#168**

Quote:

Thanks for the hint

Leela Zero 0.12 starts a 1h/game with 50sec/move what is around 17500 playouts.

Nice, I just tested Leela Zero 0.12's fixed time settings and

Code:

kgs-time_settings absolute 3600 0 0

does indeed start off with 50s per move. I hope it correctly scales down as the game moves on :lol:

Not sure how other engines do it or how fancy their time management gets. Probably it doesn't give that many extra Elo, compared to the strength gained almost every day anyway :clap:

.

as0770 · **#169**

almost_sente wrote:

Quote:

Thanks for the hint

Leela Zero 0.12 starts a 1h/game with 50sec/move what is around 17500 playouts.

Nice, I just tested Leela Zero 0.12's fixed time settings and

Code:

kgs-time_settings absolute 3600 0 0

does indeed start off with 50s per move. I hope it correctly scales down as the game moves on :lol:

Not sure how other engines do it or how fancy their time management gets. Probably it doesn't give that many extra Elo, compared to the strength gained almost every day anyway :clap:

.

It's the same as with v0.11. I'd say the new version does sometimes interrupt the search earlier and spares some extra time for the next moves.

as0770 · **#170**

Yet another Leela Zero Update in League A.

v0.12 is around 25% faster on my system and the Network is from 2018.02.18. I thought this might be enough to get 1st, but we will likely have to wait some more weeks.

Leela vs. AQ

Code:

    1. Leela 0.11.0                     9/16
    2. AQ 2.1.1                         7/16

League A:

Code:

    1. Leela 0.11.0                    17/20
    2. Rayon 4.6.0                     14/20
    3. Leela Zero 0.12 0fb68ccf        13/20
    4. Oakfoam 0.2.1 NG-06             11/20
    5. Hiratuka 10.37B (CPU)            4/20
    6. DreamGo 0.5.0                    1/20

League B:

Code:

    1. DreamGo 0.5.0                   15/20
    2. DarkForrest MCTS 1.0            12/20
    3. Pachi 11.99                     12/20
    4. DarkGo 1.0                      10/20
    5. Ray 9.0.1                        9/20
    6. Mogo 4.86                        2/20

League C:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:

Code:

    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:

Code:

    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:

Code:

    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:

Links:

Best,
Alex

almost_sente · **#171**

10 block Leela Zero is getting pretty strong now. Not sure if it's quite at AQ's level (on good hardware) but it is beating regular Leela now.

Vargo · **#172**

The strength of LZ (v. AQ) depends on the graphic card(s)

With a weak graphic card, LZ is already stronger than AQ.

With a GeForce 840M, they're about even (see the results on this thread).

With 1x GTX1080Ti, AQ is about 2 stones stronger

With 2x GTX1080Ti , AQ is around 3 stones stronger.
I've never (yet) seen LZ win an H2 game, but the newest networks win about half of the H3 games.

LZ's strength is constantly increasing, see here

But with 2x1080Ti, AQ is pro level, here is what AQ's author says :

Quote:

AQ has been officially running at Yugen no ma, which is a Go server managed Nihon Ki-in, since last week and the result of 100 games against Japanese professional players was 95-5 (win-loss). The hardware is dual GPUs (GTX 1080Ti) and i7-8700K. If you use a single GTX 1080Ti, the elo rating will be reduced by 100-120.

I've run a 6 games match between AQ211 and LZ12_545ca6 with H3, AQ being white.
545ca6 performed very well, winning half of the games.

(Using Sabaki, both engines at CGOS time setting, no pondering, using 2x GTX1080Ti)

The games :

AQ wins
AQ wins
AQ wins
LZ wins
LZ wins
LZ wins

LZ's progression is very interesting !

as0770 · **#173**

Now Leela Zero is the Leader in League A. It lost only one game against Leela by 0.5 points. Unfortunately I can't run it against AQ, but there are others playing this match in this thread :-)

Leela vs. AQ

Code:

    1. Leela 0.11.0                     9/16
    2. AQ 2.1.1                         7/16

League A:

Code:

    1. Leela Zero 0.12 545ca6d6        19/20
    2. Leela 0.11.0                    15/20
    3. Rayon 4.6.0                     11/20
    4. Oakfoam 0.2.1 NG-06             10/20
    5. Hiratuka 10.37B (CPU)            4/20
    6. DreamGo 0.5.0                    1/20

League B:

Code:

    1. DreamGo 0.5.0                   15/20
    2. DarkForrest MCTS 1.0            12/20
    3. Pachi 11.99                     12/20
    4. DarkGo 1.0                      10/20
    5. Ray 9.0.1                        9/20
    6. Mogo 4.86                        2/20

League C:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:

Code:

    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:

Code:

    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:

Code:

    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:

Links:

Best,
Alex

as0770 · **#174**

as0770 wrote:

It lost only one game against Leela by 0.5 points.

I learned that this might be due to Leela Zero is learning with a komi of 7.5 and my matches are done with a komi of 6.5.

luigi · **#175**

I think using 6.5 komi for computer games is a bad idea in general, as they all use area scoring internally.

pnprog · **#176**

as0770 wrote:

It lost only one game against Leela by 0.5 points.

I learned that this might be due to Leela Zero is learning with a komi of 7.5 and my matches are done with a komi of 6.5.

Yes, normally Leela Zero (and other bots) won't lose a game by 0.5pt or 1.5pt, they will throw the game away and resign.

It is most probably that Leela Zero believes she was winning by 0.5pt in that game.

as0770 · **#177**

luigi wrote:

I think using 6.5 komi for computer games is a bad idea in general, as they all use area scoring internally.

From the AQ website:

Quote:

!!!Caution!!! This version was trained in Komi = 6.5 for the Japanese rule. So, AQ often loses 0.5 point at Black, but that is inevitable.

IMHO bots should be able to handle different komi.

dfan · **#178**

as0770 wrote:

IMHO bots should be able to handle different komi.

I think most bot authors would agree. It turns out to be a tricky business, though.

lightvector · **#179**

Yep. I had exactly this issue in mind when I posted this topic:
forum/viewtopic.php?f=45&t=15314

As should be obvious from the discussion, there might indeed be a way, or indeed several ways, but it's not a trivial issue.

as0770 · **#180**

This time I took the CPU only version of Leela Zero with the network from 2018.03.25. It is around 5x slower than the GPU version but I can follow the progress a little bit longer and play fair games against AQ. I guess also the CPU version of Leela Zero will soon surpass AQ on my system.

Leela vs. AQ

Code:

    1. AQ 2.1.1                         5/8
    2. Leela Zero 0.13 CPU 85c6f2ad     3/8

League A:

Code:

    1. Leela Zero CPU 0.13 85c6f2ad    18/20
    2. Leela 0.11.0                    15/20
    3. Oakfoam 0.2.1 NG-06             11/20
    4. Rayon 4.6.0                     11/20
    5. Hiratuka 10.37B (CPU)            4/20
    6. DreamGo 0.5.0                    1/20

League B:

Code:

    1. DreamGo 0.5.0                   15/20
    2. DarkForrest MCTS 1.0            12/20
    3. Pachi 11.99                     12/20
    4. DarkGo 1.0                      10/20
    5. Ray 9.0.1                        9/20
    6. Mogo 4.86                        2/20

League C:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:

Code:

    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:

Code:

    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:

Code:

    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:

Links:

Best,
Alex

Engine Tournament

Who is online