It is currently Sun Sep 24, 2017 12:20 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 13 posts ] 
Author Message
Offline
 Post subject: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #1 Posted: Wed Sep 13, 2017 3:06 pm 
Lives in gote

Posts: 443
Location: Vienna, Austria
Liked others: 203
Was liked: 201
I'm running Leela 0.10.0 and 0.11.0 Beta 3 on a MacBook Air 2013. It doesn't have a dedicated GPU, so I'm using the normal, non-OpenCL version of Leela.

As with Beta 1 and Beta 2, the Beta 3 version seems to be quite a bit slower than 0.10.0.

Leela 0.10.0:

Code:
benchmark:

200000 games in 25.16 seconds -> 7949 g/s (1987 g/s per thread)
Avg Len: 451.63 Score: -1.487315

netbench:

2000 predictions in 43.64 seconds -> 45 p/s
10000 evaluations in 50.85 seconds -> 196 p/s


Leela 0.11.0 Beta 3:

Code:
benchmark:

200000 games in 36.63 seconds -> 5460 g/s (1365 g/s per thread)
Avg Len: 418.62 Score: -2.488070

netbench:

2000 predictions in 43.87 seconds -> 45 p/s
10000 evaluations in 75.07 seconds -> 133 p/s


Can you confirm this; do you have an explanation?

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #2 Posted: Wed Sep 13, 2017 3:13 pm 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 4
Rank: KGS 6 dan
I 've been having the exact same issue on my average laptop.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #3 Posted: Wed Sep 13, 2017 3:49 pm 
Lives in gote

Posts: 685
Location: Littleton, CO
Liked others: 240
Was liked: 194
Rank: KGS 3k
Universal go server handle: jeromie
I haven't tried running it, but I wonder if it's this item from the change log:

Quote:
New, much stronger Monte Carlo evaluation by combining Policy Gradient Reinforcement Learning and Simulation Balancing.


That seems consistent with extra time being in the evaluation process.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #4 Posted: Thu Sep 14, 2017 12:49 am 
Lives in gote

Posts: 443
Location: Vienna, Austria
Liked others: 203
Was liked: 201
I've had Leela 0.10.0 play five games against 0.11.0 Beta 1 a few days ago and the beta version only won three of those, so I'm not sure how the playout improvements affect strength.

For reference, the command I've used it

Code:
twogtp.py --verbose 2 --black '/usr/local/bin/leela_0110b1_macOS --gtp --noponder' --white '/usr/local/bin/leela_0100_macOS --gtp --noponder' --komi 6.5 --games 5


Here is a game I just ran with the "beta 3" version. That's on a Linux server with 6 Intel Xeon X5650 2.67 GHz, 24 threads. The beta lost because it misread a life-and-death problem. (The 0.10.0 version's life-and-death behaviour is no less weird.) So the increased evaluation strength is not reflected in its life-and-death or semeai either.



I'll try to run more games with the newest beta.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #5 Posted: Thu Sep 14, 2017 4:45 am 
Beginner

Posts: 3
Liked others: 0
Was liked: 0
is twogtp.py available opensource? I can post results for comparison from a machine with a large gpu and linux.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #6 Posted: Thu Sep 14, 2017 6:16 am 
Lives in gote

Posts: 443
Location: Vienna, Austria
Liked others: 203
Was liked: 201
jin2983 wrote:
is twogtp.py available opensource? I can post results for comparison from a machine with a large gpu and linux.


https://github.com/pasky/pachi/blob/mas ... /twogtp.py

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #7 Posted: Thu Sep 14, 2017 9:25 am 
Dies with sente

Posts: 79
Liked others: 23
Was liked: 12
Rank: Total beginner
Universal go server handle: Satorian
Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #8 Posted: Fri Sep 15, 2017 12:46 pm 
Dies in gote

Posts: 29
Liked others: 0
Was liked: 7
Satorian wrote:
Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.


From the same release notes:
"More accurate, larger value network for 19x19."

Larger network would also take longer to compute.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #9 Posted: Sat Sep 16, 2017 7:48 pm 
Lives in gote

Posts: 514
Liked others: 35
Was liked: 164
GD Posts: 10
This person said that Leela 11 beta is 2 stones stronger than Leela 10

https://www.reddit.com/r/baduk/comments ... nger_than/



By the way, Beta 5 is out, the changelog is

Windows version is now compiled with Clang/LLVM 5.0 instead of MSVC2017. This makes the Monte Carlo evaluations about 15% faster.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #10 Posted: Sun Sep 17, 2017 1:58 pm 
Lives in gote

Posts: 443
Location: Vienna, Austria
Liked others: 203
Was liked: 201
I ran 40 games on the above-mentioned Xeon server. Results:

Code:
0.10.0 (W) - 0.11.0 Beta 3 (B)
Beta wins 8, stable 2

0.11.0 Beta 3 (W) - 0.10.0 (B)
Beta wins 7, stable 3

0.10.0 (W) - 0.11.0 Beta 5 (B)
Beta wins 7, White 3

0.11.0 Beta 5 (W) - 0.10.0 (B)
Beta wins 9, stable 1

Total 40 games.
Beta wins 31, stable 9.


Based on this table from Sensei's Library, winning three out of four games in dan ranks means a difference of one stone, right?

Here are benchmarks for that server:

Leela 0.10.0:

Code:
benchmark
200000 games in  3.23 seconds -> 61919 g/s (2579 g/s per thread)
Avg Len: 451.57 Score: -1.819890

netbench
2000 predictions in 17.35 seconds -> 115 p/s
10000 evaluations in 18.31 seconds -> 546 p/s


Leela 0.11.0 Beta 5:

Code:
benchmark
200000 games in  4.62 seconds -> 43290 g/s (1803 g/s per thread)
Avg Len: 418.63 Score: -2.596610

netbench
2000 predictions in 17.00 seconds -> 117 p/s
10000 evaluations in 24.63 seconds -> 406 p/s

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #11 Posted: Tue Sep 19, 2017 8:29 am 
Beginner

Posts: 3
Liked others: 0
Was liked: 0
I started on a 40 game series but 0.11 won the first 12 of them so I'll restart with 0.5 komi to get into 0-100% range.

Different from above, I used 7.5 komi as leela plays by chinese rules.

command:
Code:
~/pachi/tools/twogtp.py --black '0.10.0/leela_0100_linux_x64_opencl --gtp --quiet --noponder' --white '0.11.0b5/leela_0110_linux_x64_opencl --gtp --quiet --noponder' --komi 7.5 --games 40


0.10 openCL benchmarks:
Code:
Leela: benchmark
200000 games in  7.55 seconds -> 26490 g/s (3311 g/s per thread)
Avg Len: 451.69 Score: -1.961805

Leela: netbench
2000 predictions in  1.45 seconds -> 1379 p/s
10000 evaluations in  3.44 seconds -> 2906 p/s


0.11b5 openCL benchmarks:
Code:
Leela: benchmark
200000 games in 10.76 seconds -> 18587 g/s (2323 g/s per thread)
Avg Len: 418.58 Score: -2.588590

Leela: netbench
2000 predictions in  3.11 seconds -> 643 p/s
10000 evaluations in  3.40 seconds -> 2941 p/s


Looking at benchmark results predictions are slower(explained by larger value net?) while evaluations are roughly equal(same sized policy net?).

As for playing strength / game quality, having a GPU seems almost a must.

First 5 games attached below:









Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #12 Posted: Wed Sep 20, 2017 8:41 am 
Beginner

Posts: 3
Liked others: 0
Was liked: 0
so far 0.11b5 as white has won 7 out of 8 games on 0.5 komi

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #13 Posted: Wed Sep 20, 2017 9:57 am 
Lives in gote

Posts: 514
Liked others: 35
Was liked: 164
GD Posts: 10
beta 9 is out and changelog didn't mention any strength-related improvement
Judging by the number, if nothing wrong happen the completed version 11 will release soon.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group