Life In 19x19 http://lifein19x19.com/ |
|
Engine Tournament http://lifein19x19.com/viewtopic.php?f=18&t=13322 |
Page 9 of 20 |
Author: | as0770 [ Fri Feb 09, 2018 2:41 am ] |
Post subject: | Re: Engine Tournament |
Updates: DarkForrest is relegated to League B and Dream Go made it into League A. Also the Leela vs AQ match was replayed with the latest versions. Surprisingly (for me) Leela 0.11 was able to strike back after Leela 0.11 Beta lost 4-12 against AQ 2.0.1. Unfortunately AQ doesn't work with Rayon and Oakfoam. One of the engines will crash wenn running on one GPU. So for now AQ can't play in League A. Leela vs. AQ Code: 1. Leela 0.11.0 9/16 2. AQ 2.1.1 7/16 League A: Code: 1. Leela 0.11.0 18/20 2. Rayon 4.6.0 15/20 3. Oakfoam 0.2.1 NG-06 12/20 4. Hiratuka 10.37B (CPU) 7/20 5. Leela Zero 0.11 5773f44c 6/20 6. DreamGo 0.5.0 2/20 League B: Code: 1. DreamGo 0.5.0 15/20 2. DarkForrest MCTS 1.0 12/20 3. Pachi 11.99 12/20 4. DarkGo 1.0 10/20 5. Ray 9.0.1 9/20 6. Mogo 4.86 2/20 League C: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League D: Code: 1. GNU Go 3.8 25/28 2. Hara 0.9 18/28 3. Matilda 1.25 16/28 4. Indigo 2009 16/28 5. Dariush 3.1.5.7 15/28 6. Aya 6.34 13/28 7. Fudo Go 3.0 7/28 8. JrefBot 081016-2022 2/28 League E: Code: 1. JrefBot 081016-2022 16/20 2. Iomrascálaí 0.3.2 12/20 3. SimpleGo 0.4.3 11/20 4. Crazy Patterns 0008-13 7/20 5. Marcos Go 1.0 7/20 6. AmiGo 1.8 7/20 League F: Code: 1. AmiGo 1.8 19/20 2. Beancounter 0.1 15/20 3. Stop 0.9-005 10/20 4. GoTraxx 1.4.2 7/20 5. CopyBot 0.1 6/20 6. Brown 1.0 3/20 Configuration: Links: Best, Alex |
Author: | q30 [ Sat Feb 10, 2018 9:35 am ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: q30 wrote: You are quite right, if there is the same engine sparring. But even if there will be 2 simple MC engines (which will in sparring demonstrate mentioned by You chances with time on move --> 0), it may be difference in strength (i.e. in chances) dependent on time control because of difference in best move choice algorithm (and especially more complex engines with more complex algorithms). You can try to compare 2 engines (with close strength levels) results with time and thread control, that You have used for league B-F, and results of these engines sparring with 2' per move and 4 threads... You don't get the point. The statistical fluctuation is way too high to meassure little differences in strength. I won't play hundreds of games to prove you wrong. Once again: This are two matches with the same engines and the same conditions: as0770 wrote: Pachi vs. Hiratuka 8:8 Pachi vs. Hiratuka 2:14 This discussion doesn't make any sense. No more replies by me. This result only proves, that time control was very small for these (or one of these) engines, so games were very randomness... |
Author: | as0770 [ Tue Feb 13, 2018 10:43 pm ] |
Post subject: | Re: Engine Tournament |
Yet another Leela Zero Update in League A with a network from last Sunday. Although its learning progress seems to decrease, it made a big step in the last two weeks, it was even able to win one of four game against Leela 0.11.0: Leela vs. AQ Code: 1. Leela 0.11.0 9/16 2. AQ 2.1.1 7/16 League A: Code: 1. Leela 0.11.0 17/20 2. Leela Zero 0.11 cde9c8d4 13/20 3. Rayon 4.6.0 13/20 4. Oakfoam 0.2.1 NG-06 12/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League B: Code: 1. DreamGo 0.5.0 15/20 2. DarkForrest MCTS 1.0 12/20 3. Pachi 11.99 12/20 4. DarkGo 1.0 10/20 5. Ray 9.0.1 9/20 6. Mogo 4.86 2/20 League C: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League D: Code: 1. GNU Go 3.8 25/28 2. Hara 0.9 18/28 3. Matilda 1.25 16/28 4. Indigo 2009 16/28 5. Dariush 3.1.5.7 15/28 6. Aya 6.34 13/28 7. Fudo Go 3.0 7/28 8. JrefBot 081016-2022 2/28 League E: Code: 1. JrefBot 081016-2022 16/20 2. Iomrascálaí 0.3.2 12/20 3. SimpleGo 0.4.3 11/20 4. Crazy Patterns 0008-13 7/20 5. Marcos Go 1.0 7/20 6. AmiGo 1.8 7/20 League F: Code: 1. AmiGo 1.8 19/20 2. Beancounter 0.1 15/20 3. Stop 0.9-005 10/20 4. GoTraxx 1.4.2 7/20 5. CopyBot 0.1 6/20 6. Brown 1.0 3/20 Configuration: Links: Best, Alex |
Author: | Vargo [ Fri Feb 16, 2018 11:54 pm ] |
Post subject: | Re: Engine Tournament |
Hello, First, let me say that I'm a huge fan of your engine tournament. Concerning AQ 2.1.1 v. Leela, the result depends a lot on the harware, and particularly on the GPU. A GeForce 1080Ti is roughly 2 or 2.5 times more powerful than a 840M, that means AQ is probably one stone stronger on a 1080Ti than on a 840M. With 2 GPUs, AQ is certainly 2 stones stronger than on a 840M. If I'm not mistaken, Leela zero can handle multiple GPUs, but Leela011 OpenCL can't. So, on a gaming PC with 2 GPUs, AQ should be 2 stones stronger than Leela OpenCL, whereas on a standard PC, your tournament has shown that they're about even. To test this, I've run matches (16 games) between AQ 2.1.1 and Leela011 OpenCL : time_settings 900 0 0 (same as in CGOS server) pondering off for both (Sabaki 033.3 used for all the games) Even games_____GPU: 1x1080Ti____CPU: i7 6700K____RAM: 32 GB H2 games_______GPU: 2x1080Ti____CPU: i9 7920X____RAM: 64 GB Results Even games : AQ 2.1.1 v. Leela011 OpenCL --------> AQ: 13/16 , L011: 3/16 H2 games : AQ 2.1.1 (W) v. Leela011 OpenCL (B)--->AQ: 9/16 , L011: 7/16 Quote: Unfortunately AQ doesn't work with Rayon and Oakfoam... I've run games between AQ and Rayon or others, it works well with Sabaki. The problem is that Sabaki doesn't handle consecutive matches automatically. You have to run one game after another, I don't think you can tell Sabaki to run automatically 16 consecutive games between X and Y, save the games, and at the end, tell the score of the 16 games match. If someone knows how to do it, tell me, I'd be interested.Thanks for your engine tournament, keep up the good work, it's very interesting The games : Even games, AQ wins : http://eidogo.com/#43Z2SOX69 http://eidogo.com/#xiY7CYBH http://eidogo.com/#ffgQmCy6 http://eidogo.com/#wvNnI0Cd http://eidogo.com/#4o8Bt2DDi http://eidogo.com/#3qqrHBgd1 http://eidogo.com/#3wls4hDC http://eidogo.com/#FmaQUvCk http://eidogo.com/#3U0RMegVA http://eidogo.com/#PYCapAdC http://eidogo.com/#1yvV62zbX http://eidogo.com/#2CBYiDi0a http://eidogo.com/#3iPHUyV1M Even games, L011 wins : http://eidogo.com/#43ay1QnF7 http://eidogo.com/#4o0OQTvU1 http://eidogo.com/#u6ovKvXV H2 games, AQ wins : http://eidogo.com/#CChOLmfN http://eidogo.com/#12Zf9BM93 http://eidogo.com/#kKbAYDgl http://eidogo.com/#27oUZlYWC http://eidogo.com/#20Wfekxy4 http://eidogo.com/#y68JWKeu http://eidogo.com/#3Pt5MiuZ1 http://eidogo.com/#1zxCWuspk http://eidogo.com/#gPw8oYNv H2 games, L011 wins : http://eidogo.com/#2PwvZRi3Y http://eidogo.com/#2PwvZRi3Y http://eidogo.com/#3rfaaJ92X http://eidogo.com/#ybotc5bG http://eidogo.com/#AW7GziNr http://eidogo.com/#3qM7UfC8R |
Author: | as0770 [ Sat Feb 17, 2018 12:13 am ] |
Post subject: | Re: Engine Tournament |
Vargo wrote: Quote: Unfortunately AQ doesn't work with Rayon and Oakfoam... I've run games between AQ and Rayon or others, it works well with Sabaki. The problem is that Sabaki doesn't handle consecutive matches automatically. You have to run one game after anotherI gonna try Sabaki, but I think it is a gpu memory conflict. Even running both engines in console makes one crash. I think I need to update my computer... Thanks for your results. I think in Go you can't define one best engine, because, like you said, the strength depends a lot on the hardware. |
Author: | almost_sente [ Mon Feb 19, 2018 7:23 am ] |
Post subject: | Re: Engine Tournament |
Nice, the next version of Leela Zero has just been officially released! https://github.com/gcp/leela-zero/releases Will be interesting to see how much the improved search algorithm and speed help it. How many playouts did they usually get with your settings? Quote: Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder |
Author: | as0770 [ Mon Feb 19, 2018 11:48 am ] |
Post subject: | Re: Engine Tournament |
almost_sente wrote: Nice, the next version of Leela Zero has just been officially released! https://github.com/gcp/leela-zero/releases Will be interesting to see how much the improved search algorithm and speed help it. How many playouts did they usually get with your settings? Quote: Leela: leela_gtp_opencl –gtp –threads 4 –noponder Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder Thanks for the hint Leela Zero 0.12 starts a 1h/game with 50sec/move what is around 17500 playouts. |
Author: | almost_sente [ Mon Feb 19, 2018 2:43 pm ] |
Post subject: | Re: Engine Tournament |
Quote: Thanks for the hint Leela Zero 0.12 starts a 1h/game with 50sec/move what is around 17500 playouts. Nice, I just tested Leela Zero 0.12's fixed time settings and Code: kgs-time_settings absolute 3600 0 0 does indeed start off with 50s per move. I hope it correctly scales down as the game moves on Not sure how other engines do it or how fancy their time management gets. Probably it doesn't give that many extra Elo, compared to the strength gained almost every day anyway .
|
Author: | as0770 [ Tue Feb 20, 2018 9:15 am ] |
Post subject: | Re: Engine Tournament |
almost_sente wrote: Quote: Thanks for the hint Leela Zero 0.12 starts a 1h/game with 50sec/move what is around 17500 playouts. Nice, I just tested Leela Zero 0.12's fixed time settings and Code: kgs-time_settings absolute 3600 0 0 does indeed start off with 50s per move. I hope it correctly scales down as the game moves on Not sure how other engines do it or how fancy their time management gets. Probably it doesn't give that many extra Elo, compared to the strength gained almost every day anyway .It's the same as with v0.11. I'd say the new version does sometimes interrupt the search earlier and spares some extra time for the next moves. |
Author: | as0770 [ Wed Feb 21, 2018 10:47 pm ] |
Post subject: | Re: Engine Tournament |
Yet another Leela Zero Update in League A. v0.12 is around 25% faster on my system and the Network is from 2018.02.18. I thought this might be enough to get 1st, but we will likely have to wait some more weeks. Leela vs. AQ Code: 1. Leela 0.11.0 9/16 2. AQ 2.1.1 7/16 League A: Code: 1. Leela 0.11.0 17/20 2. Rayon 4.6.0 14/20 3. Leela Zero 0.12 0fb68ccf 13/20 4. Oakfoam 0.2.1 NG-06 11/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League B: Code: 1. DreamGo 0.5.0 15/20 2. DarkForrest MCTS 1.0 12/20 3. Pachi 11.99 12/20 4. DarkGo 1.0 10/20 5. Ray 9.0.1 9/20 6. Mogo 4.86 2/20 League C: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League D: Code: 1. GNU Go 3.8 25/28 2. Hara 0.9 18/28 3. Matilda 1.25 16/28 4. Indigo 2009 16/28 5. Dariush 3.1.5.7 15/28 6. Aya 6.34 13/28 7. Fudo Go 3.0 7/28 8. JrefBot 081016-2022 2/28 League E: Code: 1. JrefBot 081016-2022 16/20 2. Iomrascálaí 0.3.2 12/20 3. SimpleGo 0.4.3 11/20 4. Crazy Patterns 0008-13 7/20 5. Marcos Go 1.0 7/20 6. AmiGo 1.8 7/20 League F: Code: 1. AmiGo 1.8 19/20 2. Beancounter 0.1 15/20 3. Stop 0.9-005 10/20 4. GoTraxx 1.4.2 7/20 5. CopyBot 0.1 6/20 6. Brown 1.0 3/20 Configuration: Links: Best, Alex |
Author: | almost_sente [ Sat Mar 10, 2018 5:03 pm ] |
Post subject: | Re: Engine Tournament |
10 block Leela Zero is getting pretty strong now. Not sure if it's quite at AQ's level (on good hardware) but it is beating regular Leela now. |
Author: | Vargo [ Sun Mar 11, 2018 2:54 am ] |
Post subject: | Re: Engine Tournament |
The strength of LZ (v. AQ) depends on the graphic card(s) With a weak graphic card, LZ is already stronger than AQ. With a GeForce 840M, they're about even (see the results on this thread). With 1x GTX1080Ti, AQ is about 2 stones stronger With 2x GTX1080Ti , AQ is around 3 stones stronger. I've never (yet) seen LZ win an H2 game, but the newest networks win about half of the H3 games. LZ's strength is constantly increasing, see here But with 2x1080Ti, AQ is pro level, here is what AQ's author says : Quote: AQ has been officially running at Yugen no ma, which is a Go server managed Nihon Ki-in, since last week and the result of 100 games against Japanese professional players was 95-5 (win-loss). The hardware is dual GPUs (GTX 1080Ti) and i7-8700K. If you use a single GTX 1080Ti, the elo rating will be reduced by 100-120. I've run a 6 games match between AQ211 and LZ12_545ca6 with H3, AQ being white. 545ca6 performed very well, winning half of the games. (Using Sabaki, both engines at CGOS time setting, no pondering, using 2x GTX1080Ti) The games : AQ wins AQ wins AQ wins LZ wins LZ wins LZ wins LZ's progression is very interesting ! |
Author: | as0770 [ Sun Mar 11, 2018 8:50 am ] |
Post subject: | Re: Engine Tournament |
Now Leela Zero is the Leader in League A. It lost only one game against Leela by 0.5 points. Unfortunately I can't run it against AQ, but there are others playing this match in this thread Leela vs. AQ Code: 1. Leela 0.11.0 9/16 2. AQ 2.1.1 7/16 League A: Code: 1. Leela Zero 0.12 545ca6d6 19/20 2. Leela 0.11.0 15/20 3. Rayon 4.6.0 11/20 4. Oakfoam 0.2.1 NG-06 10/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League B: Code: 1. DreamGo 0.5.0 15/20 2. DarkForrest MCTS 1.0 12/20 3. Pachi 11.99 12/20 4. DarkGo 1.0 10/20 5. Ray 9.0.1 9/20 6. Mogo 4.86 2/20 League C: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League D: Code: 1. GNU Go 3.8 25/28 2. Hara 0.9 18/28 3. Matilda 1.25 16/28 4. Indigo 2009 16/28 5. Dariush 3.1.5.7 15/28 6. Aya 6.34 13/28 7. Fudo Go 3.0 7/28 8. JrefBot 081016-2022 2/28 League E: Code: 1. JrefBot 081016-2022 16/20 2. Iomrascálaí 0.3.2 12/20 3. SimpleGo 0.4.3 11/20 4. Crazy Patterns 0008-13 7/20 5. Marcos Go 1.0 7/20 6. AmiGo 1.8 7/20 League F: Code: 1. AmiGo 1.8 19/20 2. Beancounter 0.1 15/20 3. Stop 0.9-005 10/20 4. GoTraxx 1.4.2 7/20 5. CopyBot 0.1 6/20 6. Brown 1.0 3/20 Configuration: Links: Best, Alex |
Author: | as0770 [ Wed Mar 14, 2018 10:01 am ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: It lost only one game against Leela by 0.5 points. I learned that this might be due to Leela Zero is learning with a komi of 7.5 and my matches are done with a komi of 6.5. |
Author: | luigi [ Wed Mar 14, 2018 10:26 am ] |
Post subject: | Re: Engine Tournament |
I think using 6.5 komi for computer games is a bad idea in general, as they all use area scoring internally. |
Author: | pnprog [ Tue Mar 20, 2018 3:34 am ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: as0770 wrote: It lost only one game against Leela by 0.5 points. I learned that this might be due to Leela Zero is learning with a komi of 7.5 and my matches are done with a komi of 6.5. Yes, normally Leela Zero (and other bots) won't lose a game by 0.5pt or 1.5pt, they will throw the game away and resign. It is most probably that Leela Zero believes she was winning by 0.5pt in that game. |
Author: | as0770 [ Sun Mar 25, 2018 11:48 am ] |
Post subject: | Re: Engine Tournament |
luigi wrote: I think using 6.5 komi for computer games is a bad idea in general, as they all use area scoring internally. From the AQ website: Quote: !!!Caution!!! This version was trained in Komi = 6.5 for the Japanese rule. So, AQ often loses 0.5 point at Black, but that is inevitable. IMHO bots should be able to handle different komi. |
Author: | dfan [ Sun Mar 25, 2018 2:20 pm ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: IMHO bots should be able to handle different komi. I think most bot authors would agree. It turns out to be a tricky business, though. |
Author: | lightvector [ Mon Mar 26, 2018 6:00 pm ] |
Post subject: | Re: Engine Tournament |
Yep. I had exactly this issue in mind when I posted this topic: forum/viewtopic.php?f=45&t=15314 As should be obvious from the discussion, there might indeed be a way, or indeed several ways, but it's not a trivial issue. |
Author: | as0770 [ Thu Mar 29, 2018 1:26 pm ] |
Post subject: | Re: Engine Tournament |
This time I took the CPU only version of Leela Zero with the network from 2018.03.25. It is around 5x slower than the GPU version but I can follow the progress a little bit longer and play fair games against AQ. I guess also the CPU version of Leela Zero will soon surpass AQ on my system. Leela vs. AQ Code: 1. AQ 2.1.1 5/8 2. Leela Zero 0.13 CPU 85c6f2ad 3/8 League A: Code: 1. Leela Zero CPU 0.13 85c6f2ad 18/20 2. Leela 0.11.0 15/20 3. Oakfoam 0.2.1 NG-06 11/20 4. Rayon 4.6.0 11/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League B: Code: 1. DreamGo 0.5.0 15/20 2. DarkForrest MCTS 1.0 12/20 3. Pachi 11.99 12/20 4. DarkGo 1.0 10/20 5. Ray 9.0.1 9/20 6. Mogo 4.86 2/20 League C: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League D: Code: 1. GNU Go 3.8 25/28 2. Hara 0.9 18/28 3. Matilda 1.25 16/28 4. Indigo 2009 16/28 5. Dariush 3.1.5.7 15/28 6. Aya 6.34 13/28 7. Fudo Go 3.0 7/28 8. JrefBot 081016-2022 2/28 League E: Code: 1. JrefBot 081016-2022 16/20 2. Iomrascálaí 0.3.2 12/20 3. SimpleGo 0.4.3 11/20 4. Crazy Patterns 0008-13 7/20 5. Marcos Go 1.0 7/20 6. AmiGo 1.8 7/20 League F: Code: 1. AmiGo 1.8 19/20 2. Beancounter 0.1 15/20 3. Stop 0.9-005 10/20 4. GoTraxx 1.4.2 7/20 5. CopyBot 0.1 6/20 6. Brown 1.0 3/20 Configuration: Links: Best, Alex |
Page 9 of 20 | All times are UTC - 8 hours [ DST ] |
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |