It is currently Tue Sep 17, 2019 3:52 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 8 posts ] 
Author Message
Offline
 Post subject: How can I let Leela Zero use two rtx gpus ???
Post #1 Posted: Tue May 14, 2019 2:19 pm 
Dies in gote

Posts: 43
Liked others: 0
Was liked: 3
Leela is using only one gpu but I have two gpus.
Exactly the same gpus.

{
"leelaz": {
"max-analyze-time-minutes": 60,
"analyze-update-interval-centisec": 10,
"network-file": "network.gz",
"max-game-thinking-time-seconds": 2,
"engine-start-location": ".",
"engine-command": "./leela-zero/leelaz --gtp --lagbuffer 0 --weights %network-file",
"print-comms": false
},
"ui": {
"comment-font-size": 0,
"board-color": [
217,
152,
77
],
"shadow-size": 100,
"show-winrate": true,
"autosave-interval-seconds": -1,
"append-winrate-to-comment": true,
"fancy-board": true,
"show-captured": true,
"weighted-blunder-bar-height": false,
"win-rate-always-black": false,
"show-move-number": true,
"winrate-stroke-width": 3,
"show-next-moves": true,
"show-comment": true,
"show-leelaz-variation": true,
"theme": "default",
"min-playout-ratio-for-stats": 0,
"fancy-stones": true,
"resume-previous-game": false,
"window-size": [
3840,
2160
],
"new-move-number-in-branch": true,
"shadows-enabled": true,
"show-variation-graph": true,
"show-dynamic-komi": true,
"minimum-blunder-bar-width": 3,
"large-winrate": false,
"show-blunder-bar": true,
"only-last-move-number": 1,
"confirm-exit": false,
"show-status": true,
"handicap-instead-of-winrate": false,
"large-subboard": false,
"dynamic-winrate-graph-width": true,
"show-subboard": true,
"window-maximized": true,
"show-best-moves": true,
"board-size": 19
}
}

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #2 Posted: Tue May 14, 2019 2:56 pm 
Lives in gote

Posts: 423
Liked others: 181
Was liked: 187
And if you add --gpu 0 --gpu 1 to that line, does it change anything?

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #3 Posted: Tue May 14, 2019 3:49 pm 
Dies in gote

Posts: 43
Liked others: 0
Was liked: 3
Calvin Clark wrote:
And if you add --gpu 0 --gpu 1 to that line, does it change anything?


Where exactly I need to paste this?

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #4 Posted: Tue May 14, 2019 4:38 pm 
Lives in gote

Posts: 423
Liked others: 181
Was liked: 187
goame wrote:

Where exactly I need to paste this?


Short answer:

These are command line parameters to the Leela Zero binary leelaz.exe itself, which is launched by Lizzie, so I suggest you try replacing the engine-command line with this:

Code:
"engine-command": "./leela-zero/leelaz --gtp --lagbuffer 0 --weights %network-file --gpu 0 --gpu 1",


Long answer in another post to follow...

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #5 Posted: Wed May 15, 2019 3:05 pm 
Dies in gote

Posts: 43
Liked others: 0
Was liked: 3
Calvin Clark wrote:
goame wrote:

Where exactly I need to paste this?


Short answer:

These are command line parameters to the Leela Zero binary leelaz.exe itself, which is launched by Lizzie, so I suggest you try replacing the engine-command line with this:

Code:
"engine-command": "./leela-zero/leelaz --gtp --lagbuffer 0 --weights %network-file --gpu 0 --gpu 1",


Long answer in another post to follow...


Does not work correctly.
Lizzie is open but
Leela Zero is loading... and loading and loading and loading
But I can use the x button and I can do moves on the board.

It looks like this now:
{
"leelaz": {
"max-analyze-time-minutes": 60,
"analyze-update-interval-centisec": 10,
"network-file": "network.gz",
"max-game-thinking-time-seconds": 2,
"engine-start-location": ".",
"engine-command": "./leela-zero/leelaz --gtp --lagbuffer 0 --weights %network-file --gpu 0 --gpu 1",
"print-comms": false
},
"ui": {
"comment-font-size": 0,
"board-color": [
217,
152,
77
],
"shadow-size": 100,
"show-winrate": true,
"autosave-interval-seconds": -1,
"append-winrate-to-comment": true,
"fancy-board": true,
"show-captured": true,
"weighted-blunder-bar-height": false,
"--gpu 0 --gpu 1 --gpu 2 --gpu 3": true,
"win-rate-always-black": false,
"show-move-number": true,
"winrate-stroke-width": 3,
"show-next-moves": true,
"show-comment": true,
"show-leelaz-variation": true,
"theme": "default",
"min-playout-ratio-for-stats": 0,
"fancy-stones": true,
"resume-previous-game": false,
"window-size": [
3840,
2160
],
"new-move-number-in-branch": true,
"shadows-enabled": true,
"show-variation-graph": true,
"show-dynamic-komi": true,
"minimum-blunder-bar-width": 3,
"large-winrate": false,
"show-blunder-bar": true,
"only-last-move-number": 1,
"confirm-exit": false,
"show-status": true,
"handicap-instead-of-winrate": false,
"large-subboard": false,
"dynamic-winrate-graph-width": true,
"show-subboard": true,
"window-maximized": true,
"show-best-moves": true,
"board-size": 19
}
}

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #6 Posted: Thu May 16, 2019 1:46 am 
Lives in gote

Posts: 423
Liked others: 181
Was liked: 187
Remove the entry that says "--gpu 0 --gpu 1 --gpu 2 --gpu 3": true, as that is not a valid parameter to Lizzie and try again.

If that does not work, then we need to see the output of running leelaz by itself. To show that, do this from a a command prompt the same directory Lizzie.jar resides:

If you are running on Windows:

Code:
.\leela-zero\leelaz.exe -w network.gz


If you are running on Mac or Linux, use forward slashes:

Code:
./leela-zero/leelaz.exe -w network.gz


On a two-GPU system, you should see output like this:

Code:
E:\Lizzie>.\leela-zero\leelaz.exe -w network.gz
Using 2 thread(s).
RNG seed: 2961845906471187626
Leela Zero 0.16  Copyright (C) 2017-2018  Gian-Carlo Pascutto and contributors
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see the COPYING file for details.

BLAS Core: Haswell
Detecting residual layers...v1...256 channels...40 blocks.
Initializing OpenCL (autodetecting precision).
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.2.102
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Device ID:     1
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: Tesla K80
with OpenCL 1.2 capability.
Half precision compute support: No.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.2.102
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Device ID:     1
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: Tesla K80
with OpenCL 1.2 capability.
Half precision compute support: No.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Using OpenCL single precision (less than 5% slower than half).
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.2.102
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Device ID:     1
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: Tesla K80
with OpenCL 1.2 capability.
Half precision compute support: No.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Setting max tree size to 3736 MiB and cache size to 415 MiB.

Passes: 0            Black (X) Prisoners: 0
Black (X) to move    White (O) Prisoners: 0

   a b c d e f g h j k l m n o p q r s t
19 . . . . . . . . . . . . . . . . . . . 19
18 . . . . . . . . . . . . . . . . . . . 18
17 . . . . . . . . . . . . . . . . . . . 17
16 . . . + . . . . . + . . . . . + . . . 16
15 . . . . . . . . . . . . . . . . . . . 15
14 . . . . . . . . . . . . . . . . . . . 14
13 . . . . . . . . . . . . . . . . . . . 13
12 . . . . . . . . . . . . . . . . . . . 12
11 . . . . . . . . . . . . . . . . . . . 11
10 . . . + . . . . . + . . . . . + . . . 10
9 . . . . . . . . . . . . . . . . . . .  9
8 . . . . . . . . . . . . . . . . . . .  8
7 . . . . . . . . . . . . . . . . . . .  7
6 . . . . . . . . . . . . . . . . . . .  6
5 . . . . . . . . . . . . . . . . . . .  5
4 . . . + . . . . . + . . . . . + . . .  4
3 . . . . . . . . . . . . . . . . . . .  3
2 . . . . . . . . . . . . . . . . . . .  2
1 . . . . . . . . . . . . . . . . . . .  1
   a b c d e f g h j k l m n o p q r s t

Hash: 9A930BE1616C538E Ko-Hash: A14C933E7669946D

Black time: 01:00:00
White time: 01:00:00

Leela:



The key thing to note here, other than I haven't gotten around to updating to LZ 0.17, is that two Device IDs are detected: 0 and 1. Show us what you are seeing. Also, have you recently added the 2nd GPU after running Lizzie before? If so, it's not a bad idea to remove the tuning file 'leelaz_opencl_tuning' before doing the above steps. This will cause Leela Zero to regenerate the tuning file by running some benchmarks. That can take a couple of minutes the first time, so be patient. The tuning process should also see your 2nd GPU. If it does not, then we need to look more deeply at OS specifics.

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #7 Posted: Thu May 16, 2019 11:11 am 
Dies in gote

Posts: 43
Liked others: 0
Was liked: 3
Calvin Clark wrote:
Remove the entry that says "--gpu 0 --gpu 1 --gpu 2 --gpu 3": true, as that is not a valid parameter to Lizzie and try again.

If that does not work, then we need to see the output of running leelaz by itself. To show that, do this from a a command prompt the same directory Lizzie.jar resides:

If you are running on Windows:

Code:
.\leela-zero\leelaz.exe -w network.gz


If you are running on Mac or Linux, use forward slashes:

Code:
./leela-zero/leelaz.exe -w network.gz


On a two-GPU system, you should see output like this:

Code:
E:\Lizzie>.\leela-zero\leelaz.exe -w network.gz
Using 2 thread(s).
RNG seed: 2961845906471187626
Leela Zero 0.16  Copyright (C) 2017-2018  Gian-Carlo Pascutto and contributors
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see the COPYING file for details.

BLAS Core: Haswell
Detecting residual layers...v1...256 channels...40 blocks.
Initializing OpenCL (autodetecting precision).
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.2.102
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Device ID:     1
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: Tesla K80
with OpenCL 1.2 capability.
Half precision compute support: No.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.2.102
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Device ID:     1
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: Tesla K80
with OpenCL 1.2 capability.
Half precision compute support: No.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Using OpenCL single precision (less than 5% slower than half).
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.2.102
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Device ID:     1
Device name:   Tesla K80
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 397.44
Device speed:  823 MHz
Device cores:  13 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: Tesla K80
with OpenCL 1.2 capability.
Half precision compute support: No.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Setting max tree size to 3736 MiB and cache size to 415 MiB.

Passes: 0            Black (X) Prisoners: 0
Black (X) to move    White (O) Prisoners: 0

   a b c d e f g h j k l m n o p q r s t
19 . . . . . . . . . . . . . . . . . . . 19
18 . . . . . . . . . . . . . . . . . . . 18
17 . . . . . . . . . . . . . . . . . . . 17
16 . . . + . . . . . + . . . . . + . . . 16
15 . . . . . . . . . . . . . . . . . . . 15
14 . . . . . . . . . . . . . . . . . . . 14
13 . . . . . . . . . . . . . . . . . . . 13
12 . . . . . . . . . . . . . . . . . . . 12
11 . . . . . . . . . . . . . . . . . . . 11
10 . . . + . . . . . + . . . . . + . . . 10
9 . . . . . . . . . . . . . . . . . . .  9
8 . . . . . . . . . . . . . . . . . . .  8
7 . . . . . . . . . . . . . . . . . . .  7
6 . . . . . . . . . . . . . . . . . . .  6
5 . . . . . . . . . . . . . . . . . . .  5
4 . . . + . . . . . + . . . . . + . . .  4
3 . . . . . . . . . . . . . . . . . . .  3
2 . . . . . . . . . . . . . . . . . . .  2
1 . . . . . . . . . . . . . . . . . . .  1
   a b c d e f g h j k l m n o p q r s t

Hash: 9A930BE1616C538E Ko-Hash: A14C933E7669946D

Black time: 01:00:00
White time: 01:00:00

Leela:



The key thing to note here, other than I haven't gotten around to updating to LZ 0.17, is that two Device IDs are detected: 0 and 1. Show us what you are seeing. Also, have you recently added the 2nd GPU after running Lizzie before? If so, it's not a bad idea to remove the tuning file 'leelaz_opencl_tuning' before doing the above steps. This will cause Leela Zero to regenerate the tuning file by running some benchmarks. That can take a couple of minutes the first time, so be patient. The tuning process should also see your 2nd GPU. If it does not, then we need to look more deeply at OS specifics.


Removing gpu 0 to 3 does not help.

Using OpenCL batch size of 5
Using 10 thread(s).
RNG seed: 5707416252940862580
Leela Zero 0.17 Copyright (C) 2017-2019 Gian-Carlo Pascutto and contributors
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see the COPYING file for details.

BLAS Core: Sandybridge
Detecting residual layers...v1...256 channels...40 blocks.
Initializing OpenCL (autodetecting precision).
Detected 2 OpenCL platforms.
Platform version: OpenCL 2.0 AMD-APP (2079.4)
Platform profile: FULL_PROFILE
Platform name: AMD Accelerated Parallel Processing
Platform vendor: Advanced Micro Devices, Inc.
Device ID: 0
Device name: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz
Device type: CPU
Device vendor: GenuineIntel
Device driver: 2079.4 (sse2,avx)
Device speed: 3200 MHz
Device cores: 6 CU
Device score: 520
Platform version: OpenCL 1.2 CUDA 10.0.150
Platform profile: FULL_PROFILE
Platform name: NVIDIA CUDA
Platform vendor: NVIDIA Corporation
Device ID: 1
Device name: GeForce RTX 2080 Ti
Device type: GPU
Device vendor: NVIDIA Corporation
Device driver: 411.70
Device speed: 1545 MHz
Device cores: 68 CU
Device score: 1112
Device ID: 2
Device name: GeForce RTX 2080 Ti
Device type: GPU
Device vendor: NVIDIA Corporation
Device driver: 411.70
Device speed: 1545 MHz
Device cores: 68 CU
Device score: 1112
Selected platform: NVIDIA CUDA
Selected device: GeForce RTX 2080 Ti
with OpenCL 1.2 capability.
Half precision compute support: No.
Tensor Core support: Yes.
OpenCL: using fp16/half or tensor core compute support.

Started OpenCL SGEMM tuner.
Will try 380 valid configurations.
(1/380) KWG=16 KWI=2 MDIMA=8 MDIMC=8 MWG=64 NDIMB=8 NDIMC=8 NWG=64 SA=1 SB=1 STR
M=0 STRN=0 TCE=0 VWM=4 VWN=2 0.1145 ms (5150.0 GFLOPS)
(2/380) KWG=16 KWI=8 MDIMA=8 MDIMC=8 MWG=64 NDIMB=8 NDIMC=8 NWG=32 SA=1 SB=1 STR
M=0 STRN=0 TCE=0 VWM=4 VWN=4 0.0943 ms (6257.2 GFLOPS)
(5/380) KWG=32 KWI=2 MDIMA=16 MDIMC=16 MWG=128 NDIMB=16 NDIMC=32 NWG=32 SA=0 SB=
0 STRM=0 STRN=0 TCE=1 VWM=2 VWN=2 0.0921 ms (6404.4 GFLOPS)
(10/380) KWG=16 KWI=2 MDIMA=32 MDIMC=32 MWG=64 NDIMB=8 NDIMC=16 NWG=32 SA=0 SB=0
STRM=0 STRN=0 TCE=1 VWM=2 VWN=2 0.0915 ms (6449.3 GFLOPS)
(11/380) KWG=32 KWI=2 MDIMA=8 MDIMC=16 MWG=64 NDIMB=32 NDIMC=32 NWG=32 SA=0 SB=0
STRM=0 STRN=0 TCE=1 VWM=2 VWN=2 0.0823 ms (7165.0 GFLOPS)
(14/380) KWG=16 KWI=2 MDIMA=16 MDIMC=32 MWG=128 NDIMB=16 NDIMC=16 NWG=32 SA=0 SB
=0 STRM=0 STRN=0 TCE=1 VWM=2 VWN=2 0.0697 ms (8461.8 GFLOPS)
(25/380) KWG=32 KWI=2 MDIMA=8 MDIMC=8 MWG=128 NDIMB=32 NDIMC=32 NWG=32 SA=0 SB=0
STRM=0 STRN=0 TCE=1 VWM=2 VWN=2 0.0613 ms (9616.3 GFLOPS)
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
Setting max tree size to 3736 MiB and cache size to 415 MiB.

Passes: 0 Black (X) Prisoners: 0
Black (X) to move White (O) Prisoners: 0

a b c d e f g h j k l m n o p q r s t
19 . . . . . . . . . . . . . . . . . . . 19
18 . . . . . . . . . . . . . . . . . . . 18
17 . . . . . . . . . . . . . . . . . . . 17
16 . . . + . . . . . + . . . . . + . . . 16
15 . . . . . . . . . . . . . . . . . . . 15
14 . . . . . . . . . . . . . . . . . . . 14
13 . . . . . . . . . . . . . . . . . . . 13
12 . . . . . . . . . . . . . . . . . . . 12
11 . . . . . . . . . . . . . . . . . . . 11
10 . . . + . . . . . + . . . . . + . . . 10
9 . . . . . . . . . . . . . . . . . . . 9
8 . . . . . . . . . . . . . . . . . . . 8
7 . . . . . . . . . . . . . . . . . . . 7
6 . . . . . . . . . . . . . . . . . . . 6
5 . . . . . . . . . . . . . . . . . . . 5
4 . . . + . . . . . + . . . . . + . . . 4
3 . . . . . . . . . . . . . . . . . . . 3
2 . . . . . . . . . . . . . . . . . . . 2
1 . . . . . . . . . . . . . . . . . . . 1
a b c d e f g h j k l m n o p q r s t

Hash: 9A930BE1616C538E Ko-Hash: A14C933E7669946D

Black time: 01:00:00
White time: 01:00:00

Leela:

Top
 Profile  
 
Offline
 Post subject: Re: How can I let Leela Zero use two rtx gpus ???
Post #8 Posted: Thu May 16, 2019 3:27 pm 
Lives in gote

Posts: 423
Liked others: 181
Was liked: 187
Ah, so your RTX device ids are 1 and 2, not 0 and 1. Try this instead, then:

"engine-command": "./leela-zero/leelaz --gtp --lagbuffer 0 --weights %network-file --gpu 1 --gpu 2",

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group