Engine Tournament

as0770 · **#201**

abcd_z wrote:

as0770 wrote:

I won't test every network, so I have to do a cut somewhere.

That's reasonable, but if you're going to do a tournament that tests the strength of Leela Zero it should include, at minimum, the most recent LZ network. Cut out other LZ networks if you have to.

Other networks have been played already, I don't save time and computing resources when I cut them out.

But you are right, it has been a long time since the last test so I currently run a new one.

Because of the training with ELF games the current 192x15 network will likely make progress for a while.

abcd_z wrote:

as0770 wrote:

When ELF came up Leela won 7% against it. Ten Networks later it wins 10%. I think this says much more about the progress than a 16 game tournament.

I have no idea what you mean by this.

When ELF was published Leelas best network won 7% against the ELF network. Yesterday, 11 days later, Leelas best network won 10% against the same ELF network.

abcd_z · **#202**

as0770 wrote:

Other networks have been played already, I don't save time and computing resources when I cut them out.

Wait, are you reusing the results between tournaments?!

Quote:

But you are right, it has been a long time since the last test so I currently run a new one.

Because of the training with ELF games the current 192x15 network will likely make progress for a while.

Oh my god, you are reusing the results. That's why you don't want to use the results from a constantly-changing network.

Look, if you're running a tournament between game engines, you can't just reuse previous results. You have to use all-new matches, or the results aren't valid.

as0770 · **#203**

abcd_z wrote:

You have to use all-new matches, or the results aren't valid.

It is not a ratinglist but a tournament. The results are exactly as valid as intended. If it doesn't match your prospects you have to watch for other sources or run your own tournament. But be aware: One run will take 20 days 24/7 if you do all-new matches.

abcd_z · **#204**

as0770 wrote:

It is not a ratinglist but a tournament.

Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.

To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.

as0770 · **#205**

abcd_z wrote:

as0770 wrote:

It is not a ratinglist but a tournament.

Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.

Now you've heard of one.

abcd_z wrote:

To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.

I am sorry that you feel misleaded. How can I name it in the future to avoid misleading?

jokkebk · **#206**

abcd_z wrote:

Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.
To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.

Although normal standards of politeness don't apply on internet forums, I would allow as0770 some credit (even quite a significant amount) for running this "tournament" before going into critique part. I have found this thread very interesting to follow and it has been a joy to check out newest results once they become available, even though it doesn't happen every day. However, running this kind of project on voluntary effort is a huge undertaking, so that is completely understandable. So thanks to as0770 from here!

Now it is true that I also had the impression that "tournament" was somehow plurar (like "engine tournaments", with each post representing one tournament), i.e. a new one is done every once in a while. So there is ambiguity in the term. Maybe the updates could say "here are updated results of the continuing tournament", or something to that effect.

However, despite abcd_z's vehement reaction to this, I would maybe argue, that if the engines are exact same version (like SomeBot 1.4.4 or OtherEngine 10), arranging a 10 game match between them every time is just reducing variance after a while, and not likely to uncover any deeper truths, so I would not put much effort into having GNU Go play against every other bot all the time. For Leela Zero and other actively progressing engines, however, there might be a need to re-run the newest version against the opposition every once in a while.

Of course that raises the question that if BotX won 2/10 against LZ #120, and then LZ #142 beats it 10-0, will it then be 10-0 or 18-2 for Leela vs. BotX. But if it's only a one match per opponent, then you can just overwrite the previous results.

Naturally a tournament is not an ELO comparison, as in the cases where only single game is played, it's about winning probabilities but the result is binary. But in a league, you could argue that the results are likely to even each other out.

One possible idea: Maybe do a matrix of engines, and accumulate games slowly into the matrix, with more games allotted for those where results are not as clear? With one row/column for single version, this would slowly build up into quite good comparison. And for GNU Go vs. LZ ELF you don't need to have 10 game match, one or two would be enough to see that yes indeed LZ ELF mostly wins.

as0770 · **#207**

jokkebk wrote:

abcd_z wrote:

Every tournament I have ever heard of uses the results of current matches. I have never heard of a tournament that reused results from previous tournaments.
To describe what you are doing as an engine tournament is very misleading, because it implies you're using new results for each new round of testing.

Although normal standards of politeness don't apply on internet forums, I would allow as0770 some credit (even quite a significant amount) for running this "tournament" before going into critique part. I have found this thread very interesting to follow and it has been a joy to check out newest results once they become available, even though it doesn't happen every day. However, running this kind of project on voluntary effort is a huge undertaking, so that is completely understandable. So thanks to as0770 from here!

Thanks for your kind words. I did this kind of tournaments many years with chess engines. When AlphaGo came up I became very intersted in Go and Go engines. I know, and I stated several times, that a tournament of ~20 games for each AI won't give a statistically significant result. Beside that the strength of a Go engine depends a lot on the hardware. Some use GPU some don't. The same tournament on a better graphic card will lead to a totally different result. So my intention is not to find the truth about Go engines, simply because there is no real truth. My intention is to give an overview of the strength of gtp engines of all levels. And I want to do that without running tournaments 24/7.

In the beginning of each post I mentioned the "updates" of the tournament, I wasn't aware this can be misunderstood, I'll try to state that more clear in the future

jokkebk wrote:

I also had the impression that "tournament" was somehow plurar (like "engine tournaments", with each post representing one tournament)

The thread title is "Engine tournament" (singular)

jokkebk wrote:

For Leela Zero and other actively progressing engines, however, there might be a need to re-run the newest version against the opposition every once in a while.

New versions of an engine or networks play of course new games. I updated the Leela version quite a lot in the past. Now it is somewhere between ZEN and ELF so it don't make much sense to test every new network.

abcd_z · **#208**

as0770 wrote:

I am sorry that you feel misleaded. How can I name it in the future to avoid misleading?

I would edit your very first post and include something like the following:

Quote:

Because the strength of go programs don't really change over time, and because it takes so long to run the matches, I reuse the results from previous tournaments. Each program runs once against all the current competitors when it's introduced, and those results are used for each subsequent tournament.

This also means that I won't be testing the current weights of Leela Zero, which changes constantly. Instead, I'm going to use the last LZ model of each size.

Feel free to copy and paste those paragraphs if you want.

Jokkebk makes a good point: go program strengths don't usually change much over time. LZ is the exception to this rule. With that in mind, I don't have any problems with you doing the tournament the way you've been doing it, as long as it's obvious to the readers that it's not exactly a standard tournament.

as0770 · **#209**

abcd_z wrote:

as0770 wrote:

I am sorry that you feel misleaded. How can I name it in the future to avoid misleading?

I would edit your very first post and include something like the following:

Quote:

Because the strength of go programs don't really change over time, and because it takes so long to run the matches, I reuse the results from previous tournaments. Each program runs once against all the current competitors when it's introduced, and those results are used for each subsequent tournament.

This also means that I won't be testing the current weights of Leela Zero, which changes constantly. Instead, I'm going to use the last LZ model of each size.

Feel free to copy and paste those paragraphs if you want.

Jokkebk makes a good point: go program strengths don't usually change much over time. LZ is the exception to this rule. With that in mind, I don't have any problems with you doing the tournament the way you've been doing it, as long as it's obvious to the readers that it's not exactly a standard tournament.

I didn't mix different weights or different versions of one engine, in the future I'll explain updated tournaments better. But I don't see a reason to explain why I don't test all versions and networks.

abcd_z · **#210**

as0770 wrote:

I didn't mix different weights or different versions of one engine

I... what? I never said you did.

as0770 wrote:

But I don't see a reason to explain why I don't test all versions and networks.

I didn't say that, either. I feel like you're reading something different from what I'm writing.

All I'm saying is that when you use the word "tournament", the natural assumption is that you're re-generating the matches each time. All you need to do to prevent this misunderstanding is to make a note at the beginning of your first post saying that you're reusing the results of each tournament. That's all.

EDIT: Wait, you mean that thing about Leela Zero? The reason I brought LZ up specifically was because, unlike all the other engines, LZ changes frequently, and it's easy for a reader to assume that the tests you're running on it are indicative of LZ's current strength.

as0770 · **#211**

abcd_z wrote:

as0770 wrote:

I didn't mix different weights or different versions of one engine

I... what? I never said you did.

as0770 wrote:

But I don't see a reason to explain why I don't test all versions and networks.

I didn't say that, either. I feel like you're reading something different from what I'm writing.

I know what you said. Everything is in order. From the discussion one could get the impression I would use different weights with one participant. I just stated this clear.

abcd_z wrote:

Wait, you mean that thing about Leela Zero? The reason I brought LZ up specifically was because, unlike all the other engines, LZ changes frequently, and it's easy for a reader to assume that the tests you're running on it are indicative of LZ's current strength.

No, the size of the network is specified. It is much easier to understand than the network names. Beside that when I publish the results the network will be quite up to date and one can also take a look at the date of a post to see how old it is.

as0770 · **#212**

For a change some results :-)

Updates:
-Leela Zero 0.15 with the 192x15 network 11057dd from 2018.05.20
-ELF OpenGo v0 now also with Leela Zero 0.15.

So the results in the Champions League are all new, other Leagues are not affected.

ELF performed better with LZ 0.15, maybe there was a problem with the first LZ version that was able to use the ELF network.

LZ 192x15 now clearly outperformes Zen 7. Beside the new network there were changes in the time management.

Zen won it's only game because of a ladder against ELF. There were no results affected by the komi of 6.5.

Champions League:

Code:

    1. ELF OpenGo v0 LZ 0.15           14/16
    2. Leela Zero 0.15 11057dd          9/16
    2. Zen 7                            1/16

League A:

Code:

    1. Zen 7                           11/16
    2. ELF OpenGo v0 LZ 0.13x (CPU)    11/16
    3. AQ 2.1.1                         8/16
    4. Leela 0.11                       6/16
    5. Leela Zero 0.13 128x10 (CPU)     4/16

League B:

Code:

    1. Leela Zero 0.13 128x10 (CPU)    22/24
    2. Leela 0.11.0                    19/24
    3. Rayon 4.6.0                     13/24
    4. Oakfoam 0.2.1 NG-06             12/24
    5. Leela Zero 128x6 (CPU)          12/24
    6. Hiratuka 10.37B (CPU)            4/24
    7. DreamGo 0.5.0                    2/24

League C:

Code:

    1. DreamGo 0.5.0                   18/24
    2. DarkForrest MCTS 1.0            16/24
    3. Pachi 12                        16/24
    4. Ray 9.0.1                       11/24
    5. DarkGo 1.0                      10/24
    6. Leela Zero 64x5 (CPU)           10/24
    7. Mogo 4.86                        3/24

League D:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League E:

Code:

    1. GNU Go 3.8                      18/20
    2. Hara 0.9                        11/20
    3. Dariush 3.1.5.7                  9/20
    4. Indigo 2009                      9/20
    5. Matilda 1.25                     9/20
    6. OnePunchGo 1.0                   4/20

League F:

Code:

    1. OnePunchGo 1.0                  17/20
    2. Aya 6.34                        16/20  
    3. JrefBot 081016-2022             12/20  
    4. Fudo Go 3.0                     10/20
    5. Iomrascálaí 0.3.2                4/20
    6. SimpleGo 0.4.3                   1/20

League G:

Code:

    1. SimpleGo 0.4.3                  17/20
    2. AmiGo 1.8                       15/20
    3. Crazy Patterns 0008-13          13/20
    4. Marcos Go 1.0                   11/20
    5. Beancounter 0.1                  4/20
    6. Stop 0.9-005                     0/20

League H:

Code:

    1. Stop 0.9-005                    18/20
    2. GoTraxx 1.4.2                   14/20
    3. V-Run 1                          9/20
    4. CopyBot 0.1                      8/20
    5. Leela Zero 32x4 (CPU)            7/20
    6. Brown 1.0                        4/20

Configuration:

Links:

Best,
Alex

as0770 · **#213**

Update: PhoenixGo, in fact LZ 0.14 with the PhoenixGo 256x19 network, added to the Champions League.
Edith: CPU version of PhoenixGo added to League A.

Champions League:

Code:

    1. ELF OpenGo v0 LZ 0.15           20/24
    2. Leela Zero 0.15 11057dd         14/24
    3. PhoenixGo v1 LZ 0.14x           12/24
    4. Zen 7                            2/24

League A:

Code:

    1. ELF OpenGo v0 LZ 0.13x (CPU)    15/20
    2. Zen 7                           14/20
    3. AQ 2.1.1                        11/20
    4. PhoenixGo v1 LZ 0.14x (CPU)      7/20
    5. Leela 0.11                       7/20
    6. Leela Zero 0.13 128x10 (CPU)     6/20

League B:

Code:

    1. Leela Zero 0.13 128x10 (CPU)    22/24
    2. Leela 0.11.0                    19/24
    3. Rayon 4.6.0                     13/24
    4. Oakfoam 0.2.1 NG-06             12/24
    4. Leela Zero 128x6 (CPU)          12/24
    6. Hiratuka 10.37B (CPU)            4/24
    7. DreamGo 0.5.0                    2/24

League C:

Code:

    1. DreamGo 0.5.0                   18/24
    2. DarkForrest MCTS 1.0            16/24
    2. Pachi 12                        16/24
    4. Ray 9.0.1                       11/24
    5. DarkGo 1.0                      10/24
    5. Leela Zero 64x5 (CPU)           10/24
    7. Mogo 4.86                        3/24

League D:

Code:

    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League E:

Code:

    1. GNU Go 3.8                      18/20
    2. Hara 0.9                        11/20
    3. Dariush 3.1.5.7                  9/20
    3. Indigo 2009                      9/20
    5. Matilda 1.25                     9/20
    6. OnePunchGo 1.0                   4/20

League F:

Code:

    1. OnePunchGo 1.0                  17/20
    2. Aya 6.34                        16/20  
    3. JrefBot 081016-2022             12/20  
    4. Fudo Go 3.0                     10/20
    5. Iomrascálaí 0.3.2                4/20
    6. SimpleGo 0.4.3                   1/20

League G:

Code:

    1. SimpleGo 0.4.3                  17/20
    2. AmiGo 1.8                       15/20
    3. Crazy Patterns 0008-13          13/20
    4. Marcos Go 1.0                   11/20
    5. Beancounter 0.1                  4/20
    6. Stop 0.9-005                     0/20

League H:

Code:

    1. Stop 0.9-005                    18/20
    2. GoTraxx 1.4.2                   14/20
    3. V-Run 1                          9/20
    4. CopyBot 0.1                      8/20
    5. Leela Zero 32x4 (CPU)            7/20
    6. Brown 1.0                        4/20

Configuration:

Links:

Best,
Alex

luigi · **#214**

as0770 wrote:

There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.

as0770 · **#215**

luigi wrote:

as0770 wrote:

There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.

It happend to be a komi of 6.5 and I won't replay all games.

luigi · **#216**

as0770 wrote:

luigi wrote:

as0770 wrote:

There were no results affected by the komi of 6.5.

It doesn't make sense to use 6.5 komi with area scoring.

It happend to be a komi of 6.5 and I won't replay all games.

You can still make it 7.5 hereafter.

as0770 · **#217**

luigi wrote:

You can still make it 7.5 hereafter.

I don't want to have different settings in one tourney. One day I'll get a new computer and will start from scratch

The statistical fluctuation has much more influence on the result than the komi.

pnprog · **#218**

luigi wrote:

You can still make it 7.5 hereafter.

Curious about that: how to do?

as0770 wrote:

One day I'll get a new computer and will start from scratch

You may have to implement a 6.5komi league, and a 7.5komi league. Some bots are just not supposed to play at 7.5komi (AQ is the strongest of those) and other bots are not supposed to play at 6.5komi (Leela, LeelaZero, Ray... the vast majority of value network bots). It's probably that Japanese bots are more suitable for 6.5komi, so maybe Zen also plays better on 6.5komi, but it a case by case checking.

Bots without value network can probably play in both leagues, but those are expected to be much weaker bots. So maybe 2 leagues A (for 6.5komi and 7.5komi) and others leagues (B, C, D...) of weaker bots that can stay at 6.5komi so you don't have to re-run those matches.

as0770 · **#219**

pnprog wrote:

luigi wrote:

You can still make it 7.5 hereafter.

Curious about that: how to do?

as0770 wrote:

One day I'll get a new computer and will start from scratch

You may have to implement a 6.5komi league, and a 7.5komi league. Some bots are just not supposed to play at 7.5komi (AQ is the strongest of those) and other bots are not supposed to play at 6.5komi (Leela, LeelaZero, Ray... the vast majority of value network bots). It's probably that Japanese bots are more suitable for 6.5komi, so maybe Zen also plays better on 6.5komi, but it a case by case checking.

Bots without value network can probably play in both leagues, but those are expected to be much weaker bots. So maybe 2 leagues A (for 6.5komi and 7.5komi) and others leagues (B, C, D...) of weaker bots that can stay at 6.5komi so you don't have to re-run those matches.

What can we expect by a change to komi 7.5?

In the Champions League there are 3 Leela networks. They don't care about the komi, all results of them are due to resignation. So playing with a komi of 7.5 would have lead to the same results, right? 4th bot is Zen, it won two games with white by resignation of the opponent in completely lost positions. The other games of Zen where lost, so also no influence by the komi. Even though I doubt it will make any difference I'll change the komi for the Champions League to 7.5 from now on since it would have lead to the same results in the previous games.

In League A it doesn't look very different. I am very sure the inaccuracy in the results is many times smaller than the statistical fluctuation of a 20 games tournament.

pnprog · **#220**

as0770 wrote:

What can we expect by a change to komi 7.5?

I think Zen may drop out of the champion league and A league if komi is 7.5pt. But we won't know until we try.

Quote:

In the Champions League there are 3 Leela networks. They don't care about the komi, all results of them are due to resignation. So playing with a komi of 7.5 would have lead to the same results, right?

Not necessarily. In fact, Leela Zero should only lose by resignation, if she lose 0.5pt, then there is definitively something wrong. Same goes for AQ, Leela and Ray.

But let's take and example, LZ (black) vs Zen (white), komi=6.5pt
20 moves before the end of the game, black (LZ) is about to win by 0.5pt, seems good for LZ. But because LZ is trained for 7.5pt komi, she believes she will loose by 0.5 (her value network gives a 23% win rate), so she starts desperate attempts at complicating the game, play weird end game, lose points here and there, then her win rate drops bellow 20% and finally she resigns.

Zen was suppose to loose, but he ends up winning, Leela Zero resigns from a completely lost position.

Quote:

4th bot is Zen, it won two games with white by resignation of the opponent in completely lost positions.

Difficult to say, maybe Zen was losing by 0.5pt 20 moves before then end, and the opponent, believing it was losing, throw the game away?

Quote:

The other games of Zen where lost, so also no influence by the komi.

For those ones, I agree!

Quote:

Even though I doubt it will make any difference I'll change the komi for the Champions League to 7.5 from now on since it would have lead to the same results in the previous games.

If you do so, I encourage you to replay all the matches of the league, but, sure, that's a lot of time, energy (and money).

Quote:

In League A it doesn't look very different. I am very sure the inaccuracy in the results is many times smaller than the statistical fluctuation of a 20 games tournament.

Well, I will be surprised if Zen can score 14 wins.

You know, even between bots that are trained at 7.5pt komi, it could make a difference. Maybe some of them are stronger "as black", and they loose that "edge" because of the 6.5pt komi? Who know?

Engine Tournament

Who is online