Engine Tournament

as0770 · Post by **as0770** » Wed May 23, 2018 7:17 am

abcd_z wrote:
as0770 wrote:I didn't mix different weights or different versions of one engine
I... what? I never said you did.
as0770 wrote:But I don't see a reason to explain why I don't test all versions and networks.
I didn't say that, either. I feel like you're reading something different from what I'm writing.

I know what you said. Everything is in order. From the discussion one could get the impression I would use different weights with one participant. I just stated this clear.

abcd_z wrote:Wait, you mean that thing about Leela Zero? The reason I brought LZ up specifically was because, unlike all the other engines, LZ changes frequently, and it's easy for a reader to assume that the tests you're running on it are indicative of LZ's current strength.

No, the size of the network is specified. It is much easier to understand than the network names. Beside that when I publish the results the network will be quite up to date and one can also take a look at the date of a post to see how old it is.

as0770 · Post by **as0770** » Thu May 24, 2018 2:37 am

For a change some results

Updates:
-Leela Zero 0.15 with the 192x15 network 11057dd from 2018.05.20
-ELF OpenGo v0 now also with Leela Zero 0.15.

So the results in the Champions League are all new, other Leagues are not affected.

ELF performed better with LZ 0.15, maybe there was a problem with the first LZ version that was able to use the ELF network.

LZ 192x15 now clearly outperformes Zen 7. Beside the new network there were changes in the time management.

Zen won it's only game because of a ladder against ELF. There were no results affected by the komi of 6.5.

Champions League:

Code: Select all

    1. ELF OpenGo v0 LZ 0.15           14/16
    2. Leela Zero 0.15 11057dd          9/16
    2. Zen 7                            1/16

League A:

Code: Select all

    1. Zen 7                           11/16
    2. ELF OpenGo v0 LZ 0.13x (CPU)    11/16
    3. AQ 2.1.1                         8/16
    4. Leela 0.11                       6/16
    5. Leela Zero 0.13 128x10 (CPU)     4/16

League B:

Code: Select all

    1. Leela Zero 0.13 128x10 (CPU)    22/24
    2. Leela 0.11.0                    19/24
    3. Rayon 4.6.0                     13/24
    4. Oakfoam 0.2.1 NG-06             12/24
    5. Leela Zero 128x6 (CPU)          12/24
    6. Hiratuka 10.37B (CPU)            4/24
    7. DreamGo 0.5.0                    2/24

League C:

Code: Select all

    1. DreamGo 0.5.0                   18/24
    2. DarkForrest MCTS 1.0            16/24
    3. Pachi 12                        16/24
    4. Ray 9.0.1                       11/24
    5. DarkGo 1.0                      10/24
    6. Leela Zero 64x5 (CPU)           10/24
    7. Mogo 4.86                        3/24

League D:

Code: Select all

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League E:

Code: Select all

    1. GNU Go 3.8                      18/20
    2. Hara 0.9                        11/20
    3. Dariush 3.1.5.7                  9/20
    4. Indigo 2009                      9/20
    5. Matilda 1.25                     9/20
    6. OnePunchGo 1.0                   4/20

League F:

Code: Select all

    1. OnePunchGo 1.0                  17/20
    2. Aya 6.34                        16/20  
    3. JrefBot 081016-2022             12/20  
    4. Fudo Go 3.0                     10/20
    5. Iomrascálaí 0.3.2                4/20
    6. SimpleGo 0.4.3                   1/20

League G:

Code: Select all

    1. SimpleGo 0.4.3                  17/20
    2. AmiGo 1.8                       15/20
    3. Crazy Patterns 0008-13          13/20
    4. Marcos Go 1.0                   11/20
    5. Beancounter 0.1                  4/20
    6. Stop 0.9-005                     0/20

League H:

Code: Select all

    1. Stop 0.9-005                    18/20
    2. GoTraxx 1.4.2                   14/20
    3. V-Run 1                          9/20
    4. CopyBot 0.1                      8/20
    5. Leela Zero 32x4 (CPU)            7/20
    6. Brown 1.0                        4/20

Configuration:

Links:

Best,
Alex

as0770 · Post by **as0770** » Sat May 26, 2018 12:13 am

Update: PhoenixGo, in fact LZ 0.14 with the PhoenixGo 256x19 network, added to the Champions League.
Edith: CPU version of PhoenixGo added to League A.

Champions League:

Code: Select all

    1. ELF OpenGo v0 LZ 0.15           20/24
    2. Leela Zero 0.15 11057dd         14/24
    3. PhoenixGo v1 LZ 0.14x           12/24
    4. Zen 7                            2/24

League A:

Code: Select all

    1. ELF OpenGo v0 LZ 0.13x (CPU)    15/20
    2. Zen 7                           14/20
    3. AQ 2.1.1                        11/20
    4. PhoenixGo v1 LZ 0.14x (CPU)      7/20
    5. Leela 0.11                       7/20
    6. Leela Zero 0.13 128x10 (CPU)     6/20

League B:

Code: Select all

    1. Leela Zero 0.13 128x10 (CPU)    22/24
    2. Leela 0.11.0                    19/24
    3. Rayon 4.6.0                     13/24
    4. Oakfoam 0.2.1 NG-06             12/24
    4. Leela Zero 128x6 (CPU)          12/24
    6. Hiratuka 10.37B (CPU)            4/24
    7. DreamGo 0.5.0                    2/24

League C:

Code: Select all

    1. DreamGo 0.5.0                   18/24
    2. DarkForrest MCTS 1.0            16/24
    2. Pachi 12                        16/24
    4. Ray 9.0.1                       11/24
    5. DarkGo 1.0                      10/24
    5. Leela Zero 64x5 (CPU)           10/24
    7. Mogo 4.86                        3/24

League D:

Code: Select all

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League E:

Code: Select all

    1. GNU Go 3.8                      18/20
    2. Hara 0.9                        11/20
    3. Dariush 3.1.5.7                  9/20
    3. Indigo 2009                      9/20
    5. Matilda 1.25                     9/20
    6. OnePunchGo 1.0                   4/20

League F:

Code: Select all

    1. OnePunchGo 1.0                  17/20
    2. Aya 6.34                        16/20  
    3. JrefBot 081016-2022             12/20  
    4. Fudo Go 3.0                     10/20
    5. Iomrascálaí 0.3.2                4/20
    6. SimpleGo 0.4.3                   1/20

League G:

Code: Select all

    1. SimpleGo 0.4.3                  17/20
    2. AmiGo 1.8                       15/20
    3. Crazy Patterns 0008-13          13/20
    4. Marcos Go 1.0                   11/20
    5. Beancounter 0.1                  4/20
    6. Stop 0.9-005                     0/20

League H:

Code: Select all

    1. Stop 0.9-005                    18/20
    2. GoTraxx 1.4.2                   14/20
    3. V-Run 1                          9/20
    4. CopyBot 0.1                      8/20
    5. Leela Zero 32x4 (CPU)            7/20
    6. Brown 1.0                        4/20

Configuration:

Links:

Best,
Alex

luigi · Post by **luigi** » Sun May 27, 2018 4:16 am

as0770 wrote:There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.

as0770 · Post by **as0770** » Sun May 27, 2018 8:02 am

luigi wrote:
as0770 wrote:There were no results affected by the komi of 6.5.
It doesn't make sense to use 6.5 komi with area scoring.

It happend to be a komi of 6.5 and I won't replay all games.

luigi · Post by **luigi** » Tue May 29, 2018 11:23 am

as0770 wrote:
luigi wrote:
as0770 wrote:There were no results affected by the komi of 6.5.
It doesn't make sense to use 6.5 komi with area scoring.
It happend to be a komi of 6.5 and I won't replay all games.

You can still make it 7.5 hereafter.

as0770 · Post by **as0770** » Tue May 29, 2018 1:37 pm

luigi wrote:You can still make it 7.5 hereafter.

I don't want to have different settings in one tourney. One day I'll get a new computer and will start from scratch

The statistical fluctuation has much more influence on the result than the komi.

pnprog · Post by **pnprog** » Wed May 30, 2018 8:40 pm

luigi wrote:You can still make it 7.5 hereafter.

Curious about that: how to do?

as0770 wrote:One day I'll get a new computer and will start from scratch

You may have to implement a 6.5komi league, and a 7.5komi league. Some bots are just not supposed to play at 7.5komi (AQ is the strongest of those) and other bots are not supposed to play at 6.5komi (Leela, LeelaZero, Ray... the vast majority of value network bots). It's probably that Japanese bots are more suitable for 6.5komi, so maybe Zen also plays better on 6.5komi, but it a case by case checking.

Bots without value network can probably play in both leagues, but those are expected to be much weaker bots. So maybe 2 leagues A (for 6.5komi and 7.5komi) and others leagues (B, C, D...) of weaker bots that can stay at 6.5komi so you don't have to re-run those matches.

as0770 · Post by **as0770** » Thu May 31, 2018 5:53 am

pnprog wrote:
luigi wrote:You can still make it 7.5 hereafter.
Curious about that: how to do?

as0770 wrote:One day I'll get a new computer and will start from scratch
You may have to implement a 6.5komi league, and a 7.5komi league. Some bots are just not supposed to play at 7.5komi (AQ is the strongest of those) and other bots are not supposed to play at 6.5komi (Leela, LeelaZero, Ray... the vast majority of value network bots). It's probably that Japanese bots are more suitable for 6.5komi, so maybe Zen also plays better on 6.5komi, but it a case by case checking.

Bots without value network can probably play in both leagues, but those are expected to be much weaker bots. So maybe 2 leagues A (for 6.5komi and 7.5komi) and others leagues (B, C, D...) of weaker bots that can stay at 6.5komi so you don't have to re-run those matches.

What can we expect by a change to komi 7.5?

In the Champions League there are 3 Leela networks. They don't care about the komi, all results of them are due to resignation. So playing with a komi of 7.5 would have lead to the same results, right? 4th bot is Zen, it won two games with white by resignation of the opponent in completely lost positions. The other games of Zen where lost, so also no influence by the komi. Even though I doubt it will make any difference I'll change the komi for the Champions League to 7.5 from now on since it would have lead to the same results in the previous games.

In League A it doesn't look very different. I am very sure the inaccuracy in the results is many times smaller than the statistical fluctuation of a 20 games tournament.

pnprog · Post by **pnprog** » Thu May 31, 2018 7:06 am

as0770 wrote:What can we expect by a change to komi 7.5?

I think Zen may drop out of the champion league and A league if komi is 7.5pt. But we won't know until we try.

In the Champions League there are 3 Leela networks. They don't care about the komi, all results of them are due to resignation. So playing with a komi of 7.5 would have lead to the same results, right?

Not necessarily. In fact, Leela Zero should only lose by resignation, if she lose 0.5pt, then there is definitively something wrong. Same goes for AQ, Leela and Ray.

But let's take and example, LZ (black) vs Zen (white), komi=6.5pt
20 moves before the end of the game, black (LZ) is about to win by 0.5pt, seems good for LZ. But because LZ is trained for 7.5pt komi, she believes she will loose by 0.5 (her value network gives a 23% win rate), so she starts desperate attempts at complicating the game, play weird end game, lose points here and there, then her win rate drops bellow 20% and finally she resigns.

Zen was suppose to loose, but he ends up winning, Leela Zero resigns from a completely lost position.

4th bot is Zen, it won two games with white by resignation of the opponent in completely lost positions.

Difficult to say, maybe Zen was losing by 0.5pt 20 moves before then end, and the opponent, believing it was losing, throw the game away?

The other games of Zen where lost, so also no influence by the komi.

For those ones, I agree!

Even though I doubt it will make any difference I'll change the komi for the Champions League to 7.5 from now on since it would have lead to the same results in the previous games.

If you do so, I encourage you to replay all the matches of the league, but, sure, that's a lot of time, energy (and money).

In League A it doesn't look very different. I am very sure the inaccuracy in the results is many times smaller than the statistical fluctuation of a 20 games tournament.

Well, I will be surprised if Zen can score 14 wins.

You know, even between bots that are trained at 7.5pt komi, it could make a difference. Maybe some of them are stronger "as black", and they loose that "edge" because of the 6.5pt komi? Who know?

as0770 · Post by **as0770** » Thu May 31, 2018 8:30 am

pnprog wrote:I will be surprised if Zen can score 14 wins.

Against the CPU version of Leela Zero with different networks?

I think I'll keep my results private in the future. This discussions take more time than to play the games.

Bye

q30 · Post by **q30** » Sat Jun 02, 2018 12:08 am

Last version of Pachi became weaker (details).

q30 · Post by **q30** » Mon Jun 11, 2018 12:51 am

Last version of Ray became a bit stronger (details).
The rating of available for usual users Go engines without GPU support is next:

Highest level
1. Leela;

High level
2. Hiratuka;

Middle level
3. Pachi_DCNN;
4. Ray;
5. Pachi;
6. MoGo.

q30 · Post by **q30** » Sat Jun 23, 2018 2:01 am

I had sparred last version of Ray with other this level engines (details).
The rating of available for usual users Go engines without GPU support is next:

Highest level
1. Leela;

High level
2. Hiratuka;

Middle level
3. Pachi_DCNN;
4&5. Ray & Pachi;
6. MoGo.

q30 · Post by **q30** » Sat Jun 30, 2018 2:07 am

I had tested LeelaZero (details).
The rating of Go engines without using GPU is next:

Super level
1. LeelaZero;

Highest level
2. Leela;

High level
3. Hiratuka;

Middle level
4. Pachi_DCNN;
5&6. Ray & Pachi;
7. MoGo.

Life In 19x19

Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament

Re: Engine Tournament