It is currently Mon Jan 22, 2018 5:05 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2  Next
Author Message
Offline
 Post subject: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #1 Posted: Wed Sep 13, 2017 3:06 pm 
Lives in gote

Posts: 453
Location: Vienna, Austria
Liked others: 211
Was liked: 206
I'm running Leela 0.10.0 and 0.11.0 Beta 3 on a MacBook Air 2013. It doesn't have a dedicated GPU, so I'm using the normal, non-OpenCL version of Leela.

As with Beta 1 and Beta 2, the Beta 3 version seems to be quite a bit slower than 0.10.0.

Leela 0.10.0:

Code:
benchmark:

200000 games in 25.16 seconds -> 7949 g/s (1987 g/s per thread)
Avg Len: 451.63 Score: -1.487315

netbench:

2000 predictions in 43.64 seconds -> 45 p/s
10000 evaluations in 50.85 seconds -> 196 p/s


Leela 0.11.0 Beta 3:

Code:
benchmark:

200000 games in 36.63 seconds -> 5460 g/s (1365 g/s per thread)
Avg Len: 418.62 Score: -2.488070

netbench:

2000 predictions in 43.87 seconds -> 45 p/s
10000 evaluations in 75.07 seconds -> 133 p/s


Can you confirm this; do you have an explanation?

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #2 Posted: Wed Sep 13, 2017 3:13 pm 
Dies with sente

Posts: 77
Liked others: 17
Was liked: 7
Rank: KGS 6 dan
I 've been having the exact same issue on my average laptop.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #3 Posted: Wed Sep 13, 2017 3:49 pm 
Lives in sente

Posts: 774
Location: Littleton, CO
Liked others: 257
Was liked: 217
Rank: KGS 3k
Universal go server handle: jeromie
I haven't tried running it, but I wonder if it's this item from the change log:

Quote:
New, much stronger Monte Carlo evaluation by combining Policy Gradient Reinforcement Learning and Simulation Balancing.


That seems consistent with extra time being in the evaluation process.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #4 Posted: Thu Sep 14, 2017 12:49 am 
Lives in gote

Posts: 453
Location: Vienna, Austria
Liked others: 211
Was liked: 206
I've had Leela 0.10.0 play five games against 0.11.0 Beta 1 a few days ago and the beta version only won three of those, so I'm not sure how the playout improvements affect strength.

For reference, the command I've used it

Code:
twogtp.py --verbose 2 --black '/usr/local/bin/leela_0110b1_macOS --gtp --noponder' --white '/usr/local/bin/leela_0100_macOS --gtp --noponder' --komi 6.5 --games 5


Here is a game I just ran with the "beta 3" version. That's on a Linux server with 6 Intel Xeon X5650 2.67 GHz, 24 threads. The beta lost because it misread a life-and-death problem. (The 0.10.0 version's life-and-death behaviour is no less weird.) So the increased evaluation strength is not reflected in its life-and-death or semeai either.



I'll try to run more games with the newest beta.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #5 Posted: Thu Sep 14, 2017 4:45 am 
Beginner

Posts: 3
Liked others: 0
Was liked: 0
is twogtp.py available opensource? I can post results for comparison from a machine with a large gpu and linux.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #6 Posted: Thu Sep 14, 2017 6:16 am 
Lives in gote

Posts: 453
Location: Vienna, Austria
Liked others: 211
Was liked: 206
jin2983 wrote:
is twogtp.py available opensource? I can post results for comparison from a machine with a large gpu and linux.


https://github.com/pasky/pachi/blob/mas ... /twogtp.py

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #7 Posted: Thu Sep 14, 2017 9:25 am 
Dies with sente

Posts: 81
Liked others: 24
Was liked: 12
Rank: Total beginner
Universal go server handle: Satorian
Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #8 Posted: Fri Sep 15, 2017 12:46 pm 
Dies in gote

Posts: 32
Liked others: 0
Was liked: 8
Satorian wrote:
Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.


From the same release notes:
"More accurate, larger value network for 19x19."

Larger network would also take longer to compute.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #9 Posted: Sat Sep 16, 2017 7:48 pm 
Lives in gote

Posts: 683
Liked others: 43
Was liked: 210
GD Posts: 10
This person said that Leela 11 beta is 2 stones stronger than Leela 10

https://www.reddit.com/r/baduk/comments ... nger_than/



By the way, Beta 5 is out, the changelog is

Windows version is now compiled with Clang/LLVM 5.0 instead of MSVC2017. This makes the Monte Carlo evaluations about 15% faster.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #10 Posted: Sun Sep 17, 2017 1:58 pm 
Lives in gote

Posts: 453
Location: Vienna, Austria
Liked others: 211
Was liked: 206
I ran 40 games on the above-mentioned Xeon server. Results:

Code:
0.10.0 (W) - 0.11.0 Beta 3 (B)
Beta wins 8, stable 2

0.11.0 Beta 3 (W) - 0.10.0 (B)
Beta wins 7, stable 3

0.10.0 (W) - 0.11.0 Beta 5 (B)
Beta wins 7, White 3

0.11.0 Beta 5 (W) - 0.10.0 (B)
Beta wins 9, stable 1

Total 40 games.
Beta wins 31, stable 9.


Based on this table from Sensei's Library, winning three out of four games in dan ranks means a difference of one stone, right?

Here are benchmarks for that server:

Leela 0.10.0:

Code:
benchmark
200000 games in  3.23 seconds -> 61919 g/s (2579 g/s per thread)
Avg Len: 451.57 Score: -1.819890

netbench
2000 predictions in 17.35 seconds -> 115 p/s
10000 evaluations in 18.31 seconds -> 546 p/s


Leela 0.11.0 Beta 5:

Code:
benchmark
200000 games in  4.62 seconds -> 43290 g/s (1803 g/s per thread)
Avg Len: 418.63 Score: -2.596610

netbench
2000 predictions in 17.00 seconds -> 117 p/s
10000 evaluations in 24.63 seconds -> 406 p/s

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #11 Posted: Tue Sep 19, 2017 8:29 am 
Beginner

Posts: 3
Liked others: 0
Was liked: 0
I started on a 40 game series but 0.11 won the first 12 of them so I'll restart with 0.5 komi to get into 0-100% range.

Different from above, I used 7.5 komi as leela plays by chinese rules.

command:
Code:
~/pachi/tools/twogtp.py --black '0.10.0/leela_0100_linux_x64_opencl --gtp --quiet --noponder' --white '0.11.0b5/leela_0110_linux_x64_opencl --gtp --quiet --noponder' --komi 7.5 --games 40


0.10 openCL benchmarks:
Code:
Leela: benchmark
200000 games in  7.55 seconds -> 26490 g/s (3311 g/s per thread)
Avg Len: 451.69 Score: -1.961805

Leela: netbench
2000 predictions in  1.45 seconds -> 1379 p/s
10000 evaluations in  3.44 seconds -> 2906 p/s


0.11b5 openCL benchmarks:
Code:
Leela: benchmark
200000 games in 10.76 seconds -> 18587 g/s (2323 g/s per thread)
Avg Len: 418.58 Score: -2.588590

Leela: netbench
2000 predictions in  3.11 seconds -> 643 p/s
10000 evaluations in  3.40 seconds -> 2941 p/s


Looking at benchmark results predictions are slower(explained by larger value net?) while evaluations are roughly equal(same sized policy net?).

As for playing strength / game quality, having a GPU seems almost a must.

First 5 games attached below:









Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #12 Posted: Wed Sep 20, 2017 8:41 am 
Beginner

Posts: 3
Liked others: 0
Was liked: 0
so far 0.11b5 as white has won 7 out of 8 games on 0.5 komi

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #13 Posted: Wed Sep 20, 2017 9:57 am 
Lives in gote

Posts: 683
Liked others: 43
Was liked: 210
GD Posts: 10
beta 9 is out and changelog didn't mention any strength-related improvement
Judging by the number, if nothing wrong happen the completed version 11 will release soon.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #14 Posted: Mon Sep 25, 2017 1:14 pm 
Lives in gote

Posts: 436
Liked others: 1
Was liked: 38
Rank: KGS 5 kyu
I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #15 Posted: Mon Sep 25, 2017 3:38 pm 
Dies with sente

Posts: 77
Liked others: 17
Was liked: 7
Rank: KGS 6 dan
Krama wrote:
I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

I feel you. Too strong... even now (Leela 0.11 beta 10)...

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #16 Posted: Tue Sep 26, 2017 5:52 am 
Beginner

Posts: 2
Location: Vietnam
Liked others: 5
Was liked: 0
I think that Leela is stronger is the most important, slower is not a problem.

_________________
http://toannguyenthanh.rf.gd/


Last edited by toannguyenthanh on Sun Oct 22, 2017 2:15 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #17 Posted: Tue Sep 26, 2017 11:51 am 
Lives in gote

Posts: 436
Liked others: 1
Was liked: 38
Rank: KGS 5 kyu
johnsmith wrote:
Krama wrote:
I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

I feel you. Too strong... even now (Leela 0.11 beta 10)...


You are a 6 dan on kgs? So that is like 4 dan EGF right?

How well can you fare against gpu version on unlimited simulations (let's say 30 min)

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #18 Posted: Wed Sep 27, 2017 1:43 am 
Tengen

Posts: 4377
Location: Cambridge, UK
Liked others: 172
Was liked: 2156
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Krama wrote:
You are a 6 dan on kgs? So that is like 4 dan EGF right?

As an aside, I don't think many 4d EGF are 6d KGS, I'm certainly not. There are 6d EGF who are 6d KGS like Lukas Kraemer, or Cornel Burzo was 6d for a long time and just got back to 7d. KGS dan ranks are pretty tough these days. Ilya and Pavol (7d/1p) are only 7d KGS now.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #19 Posted: Wed Sep 27, 2017 3:59 am 
Dies with sente

Posts: 77
Liked others: 17
Was liked: 7
Rank: KGS 6 dan
Krama wrote:
You are a 6 dan on kgs? So that is like 4 dan EGF right?
How well can you fare against gpu version on unlimited simulations (let's say 30 min)


Well, I don't play on KGS, only on Tygem. And I'm 7d over there. I guess that's something like solid 5d EGF :)

Anyways, LEELA(B) on Tygem has 50:50 score against pros. That surely tells something about its strength. And it only plays against pros.
Personally, I like to play with RoyalLeela on OGS (thx to roy777 who has Ryzen 1700 CPU and 1080 Ti GPU), and I have 50:50 against Leela on 3 handicap stones; winrate on 4 handi is 100% though I struggle a little bit :) In even games RoyalLeela just smashes. Like Hulk.
On my personal average laptop, in even games I don't stand a chance either, though I have 100% score against 3 handi. And so far I won 2 games in Sunjung Baduk.

Top
 Profile  
 
Offline
 Post subject: Re: Leela 0.11.0 Beta version slower than 0.10.0 ?
Post #20 Posted: Wed Sep 27, 2017 4:05 am 
Lives with ko

Posts: 213
Liked others: 0
Was liked: 37
Rank: kgs 6 kyu
KGS: Pio2001
One of our club members is 7d on Tygem, but 1 kyu FFG (EGF).

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group