It is currently Mon Mar 01, 2021 9:59 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 13 posts ] 
Author Message
Offline
 Post subject: Contribute to Katago training using google colab
Post #1 Posted: Fri Feb 26, 2021 3:24 pm 
Dies in gote

Posts: 23
Liked others: 0
Was liked: 10
Rank: 18 kyu
I've made google colab notebook image which can contribute to katago training.
You can join the contribution without GPU now.
Just check the link below.
https://colab.research.google.com/drive ... sp=sharing


Attachments:
20210224_colab.png
20210224_colab.png [ 274.27 KiB | Viewed 467 times ]

This post by seventeen was liked by 3 people: ez4u, go4thewin, wineandgolover
Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #2 Posted: Fri Feb 26, 2021 5:36 pm 
Lives in sente
User avatar

Posts: 850
Liked others: 309
Was liked: 343
This looks cool. I’d love to know more. To start...

What gpu's does this use?

Is it free? For how long?

Is there some sort of limitation that might affect other google services?

Does it run in the background?

Thanks.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #3 Posted: Fri Feb 26, 2021 6:21 pm 
Lives with ko

Posts: 138
Liked others: 180
Was liked: 28
Rank: 25 kyu
edit: Is it possible to make a script like this to train the 15b net on the new s663 40b data? thanks!


Last edited by go4thewin on Mon Mar 01, 2021 7:26 am, edited 2 times in total.
Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #4 Posted: Fri Feb 26, 2021 8:40 pm 
Lives in sente
User avatar

Posts: 850
Liked others: 309
Was liked: 343
go4thewin wrote:
Really simple and fun to use. it uses a T4, it is free with usage limits. it turns off after 12 hours or less if you paste the following in the google chrome console (f12)
Code:
function ClickConnect(){
  console.log("Connnect Clicked - Start");
  document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
  console.log("Connnect Clicked - End");
};
setInterval(ClickConnect, 60000)


in 12 hours, you get more than 250 training games, 13000 rows, and a few ratings games, which is really nice. it will turn off in 90 minutes or less without the code above. if you close the browser, it will turn off. If you use it 12 hours everyday, you might get kicked off for a couple months, not sure. Every other day might be ok. It will not effect other google services. Thanks seventeen!


If one is non-technical, and has never messed with the chrome console, and doesn't want to screw things up, where exactly in the chrome console should one paste this? Top? Bottom, embedded somewhere, doesn't matter?

Also, I assume you mean. "it turns off after 12 hours or less UNLESS you paste..."?

Finally, I agree that it is really simple. Even I got it running easily. If you want to help make the strongest open-sourced go engine even better, please run this in the background when you use your computer. Highly recommended!!!!!

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #5 Posted: Sat Feb 27, 2021 3:03 am 
Lives with ko

Posts: 138
Liked others: 180
Was liked: 28
Rank: 25 kyu
Yes, sorry about that. In the picture below, paste the following code where the bottom most > sign is. Ill delete the previous redundant post.

Code:
function ClickConnect(){
    console.log("Clicked on connect button");
    document.querySelector("colab-connect-button").click()
}
setInterval(ClickConnect,60000)


This post by go4thewin was liked by: wineandgolover
Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #6 Posted: Sat Feb 27, 2021 3:42 am 
Oza
User avatar

Posts: 2325
Location: Tokyo, Japan
Liked others: 2229
Was liked: 1310
Rank: Jp 6 dan
KGS: ez4u
I am trying this out also. It is indeed simple to do. :tmbup:
I am running it in Firefox and for whatever reason, it does not shut down by itself after 90 minutes. Just now I came back after about 3 hours and it's still running. Great job! Thanks

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #7 Posted: Sat Feb 27, 2021 11:57 am 
Lives in sente
User avatar

Posts: 850
Liked others: 309
Was liked: 343
wineandgolover wrote:
This looks cool. I’d love to know more. To start...

1. What gpu's does this use?

2. Is it free? For how long?

3. Is there some sort of limitation that might affect other google services?

4. Does it run in the background?

Thanks.


To answer my own questions.

1. I’ve connected to a Tesla T4 each time. Getting around 390nn evals/ second

2. It’s completely free. It uses Google Colab, a free machine learning tool. It runs in the browser, so it’s platform independent.

3. It does not affect other google services. There is a limitation within Colab in that it will stop working for heavy users. Looking on discord, it seems running it for twelve hours every other day avoids any problems.

4. Yes it runs in the background. Google provides the CPU's and GPU's. All you need is a google drive account and a browser.

If you’d like to run it more stably, for longer, and probably get assigned a better GPU, you can consider Colab Pro, which costs $10 per month with a US or Canadian address. That seems pretty reasonable versus buying and powering your own Tesla V100. I might try Pro soon to check out it’s performance. I’d love to hear if anybody else has already done so.

Again, I encourage anyone who wishes to help make katago stronger to consider running this completely free utility in the background.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #8 Posted: Sun Feb 28, 2021 3:48 pm 
Dies in gote

Posts: 24
Liked others: 0
Was liked: 4
I followed your instruction and got an error as following...

Starting KataGo training...
2021-02-28 22:39:59+0000: Distributed Self Play Engine starting...
2021-02-28 22:39:59+0000: Attempting to connect to server
2021-02-28 22:39:59+0000: isSSL: true
2021-02-28 22:39:59+0000: host: katagotraining.org
2021-02-28 22:39:59+0000: port: 443
2021-02-28 22:39:59+0000: baseResourcePath: /
2021-02-28 22:39:59+0000: KataGo v1.8.0
2021-02-28 22:39:59+0000: Git revision: 8ffda1fe05c69c67342365013b11225d443445e8
2021-02-28 22:39:59+0000: Running tiny net to sanity-check that GPU is working
2021-02-28 22:39:59+0000: nnRandSeed0 = 10486611865130445872
2021-02-28 22:39:59+0000: After dedups: nnModelFile0 = katago_contribute/kata1/tmpTinyModel.bin.gz useFP16 auto useNHWC auto
terminate called after throwing an instance of 'StringError'
what(): OpenCL error at /home/dwugcloud/data/kata/cpp/neuralnet/openclhelpers.cpp, func err, line 263, error CL_PLATFORM_NOT_FOUND_KHR

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #9 Posted: Sun Feb 28, 2021 4:42 pm 
Oza
User avatar

Posts: 2325
Location: Tokyo, Japan
Liked others: 2229
Was liked: 1310
Rank: Jp 6 dan
KGS: ez4u
When the process is running on colab, I am seeing these security warnings constantly in the console
Code:
Content Security Policy: Ignoring “'report-sample'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “http:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/js/bg/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/recaptcha/” within script-src: ‘strict-dynamic’ specified

Is this something that should be fixed or can we just ignore it?

Meanwhile I am currently increasing the "maxSimultaneousGames" at the bottom of the script. Going from 8 (default) to 12 jumped "nn evals" from around 380/second to around 470/second.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #10 Posted: Sun Feb 28, 2021 4:53 pm 
Oza
User avatar

Posts: 2325
Location: Tokyo, Japan
Liked others: 2229
Was liked: 1310
Rank: Jp 6 dan
KGS: ez4u
deungsan wrote:
I followed your instruction and got an error as following...

Starting KataGo training...
2021-02-28 22:39:59+0000: Distributed Self Play Engine starting...
2021-02-28 22:39:59+0000: Attempting to connect to server
2021-02-28 22:39:59+0000: isSSL: true
2021-02-28 22:39:59+0000: host: katagotraining.org
2021-02-28 22:39:59+0000: port: 443
2021-02-28 22:39:59+0000: baseResourcePath: /
2021-02-28 22:39:59+0000: KataGo v1.8.0
2021-02-28 22:39:59+0000: Git revision: 8ffda1fe05c69c67342365013b11225d443445e8
2021-02-28 22:39:59+0000: Running tiny net to sanity-check that GPU is working
2021-02-28 22:39:59+0000: nnRandSeed0 = 10486611865130445872
2021-02-28 22:39:59+0000: After dedups: nnModelFile0 = katago_contribute/kata1/tmpTinyModel.bin.gz useFP16 auto useNHWC auto
terminate called after throwing an instance of 'StringError'
what(): OpenCL error at /home/dwugcloud/data/kata/cpp/neuralnet/openclhelpers.cpp, func err, line 263, error CL_PLATFORM_NOT_FOUND_KHR


My startup looks like this...
Code:
Starting KataGo training...
2021-02-28 21:51:52+0000: Distributed Self Play Engine starting...
2021-02-28 21:51:52+0000: Attempting to connect to server
2021-02-28 21:51:52+0000: isSSL: true
2021-02-28 21:51:52+0000: host: katagotraining.org
2021-02-28 21:51:52+0000: port: 443
2021-02-28 21:51:52+0000: baseResourcePath: /
2021-02-28 21:51:52+0000: KataGo v1.8.0
2021-02-28 21:51:52+0000: Git revision: 8ffda1fe05c69c67342365013b11225d443445e8
2021-02-28 21:51:52+0000: Running tiny net to sanity-check that GPU is working
2021-02-28 21:51:52+0000: nnRandSeed0 = 1331183443207076973
2021-02-28 21:51:52+0000: After dedups: nnModelFile0 = katago_contribute/kata1/tmpTinyModel.bin.gz useFP16 auto useNHWC auto
2021-02-28 21:51:52+0000: Cuda backend thread 0: Found GPU Tesla T4 memory 15843721216 compute capability major 7 minor 5
2021-02-28 21:51:52+0000: Cuda backend thread 0: Model version 9 useFP16 = true useNHWC = true
2021-02-28 21:51:52+0000: Cuda backend thread 0: Model name: rect15-b2c16-s13679744-d94886722
2021-02-28 21:51:54+0000: Tiny net sanity check complete

As far as I understand what we are doing here (questionable right there! :blackeye: ), should not be using OpenCL for anything. You should be using CUDA instead.
At the very beginning of the output from your run do you see...
Code:
Using Katago Backend :  CUDA
GPU :  TeslaT4
/content
Cloning into 'katago-colab'...

This is what I get every time.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #11 Posted: Sun Feb 28, 2021 6:16 pm 
Dies in gote

Posts: 24
Liked others: 0
Was liked: 4
My errors are fixed by changing notebook setting. Setting hardware accelerator to "GPU' lets colab use TeslaT4.

Now it works fine.


Last edited by deungsan on Sun Feb 28, 2021 6:33 pm, edited 2 times in total.
Top
 Profile  
 
Offline
 Post subject: Re: Contribute to Katago training using google colab
Post #12 Posted: Sun Feb 28, 2021 6:23 pm 
Oza
User avatar

Posts: 2325
Location: Tokyo, Japan
Liked others: 2229
Was liked: 1310
Rank: Jp 6 dan
KGS: ez4u
ez4u wrote:
...

Meanwhile I am currently increasing the "maxSimultaneousGames" at the bottom of the script. Going from 8 (default) to 12 jumped "nn evals" from around 380/second to around 470/second.

The story so far...
Code:
"maxSimultaneousGames" 08  "nn evals"  ~380/sec
"maxSimultaneousGames" 12  "nn evals"  ~470/sec
"maxSimultaneousGames" 16  "nn evals"  ~525/sec
"maxSimultaneousGames" 24  "nn evals"  ~545/sec
"maxSimultaneousGames" 32  "nn evals"  ~555/sec

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Online
 Post subject: Re: Contribute to Katago training using google colab
Post #13 Posted: Mon Mar 01, 2021 5:45 am 
Lives in gote

Posts: 644
Liked others: 105
Was liked: 768
Rank: maybe 2d
Yeah, GPUs really like it when you have large batches to run in parallel, and more games helps with that.

The one thing I would caution is - please don't make the number of simultaneous games too large compared to the number of games you are playing in a given run before you shut it down or it shuts itself down - ideally make sure the total number of games you're getting per session would be at least 10x or 20x the number of simultaneous games.

The reason is if the total is too small, such that the games are coming in relatively few "waves" before it gets killed, it will create a bias towards short games in the data - because in the last wave, disproportionately short games will be the ones that finish and get uploaded and not the longer ones. Games that were on small boards, or that had fewer fights and were more peaceful, or that initialized starting from later positions, etc. will be favored over the configured and desired distribution.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: lightvector and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group