KaTrain Questions

[RobertJasiek]

start with your proper work instead of getting lost in irrelevant myteria of KataGo.

Currently, I do not know lots of basics as to what is (ir)relevant or (not) needed.

People find it important to buy faster hardware but, if Katago tuning parameters affect speed by a factor 2 or 3, this is even much more relevant than hardware. OpenCL versus CUDA seems to be relevant but Tensor is said to accelerate by 1.5x so is also more relevant than the exact hardware. Now, I know what is the currently best net but in the future I want to able to recognise it, too, and just by file name and date this seems to be impossible.

Currently, I only know that OpenCL on the dGPU must be working. I do not know yet:

1) Have I already installed CUDA drivers as part of the Nvidia Studio driver?

2) Have I already installed cuDNN drivers as part of the Nvidia Studio driver?

3) Have I already installed tensor core drivers as part of the Nvidia Studio driver?

4) Which drivers (CUDA. cuDNN, Tensor) enable tensor cores?

5) Have I already installed a Katago instance that can run on tensor cores?

6) Else which current Katago instance do I need to download and install?

7) Can Katago OpenCL use OpenCL or tensor cores? If yes, due to which confíguration files and parameters does Katago use tensor cores?

8) Can Katago CUDA use CUDA or tensor cores? If yes, due to which confíguration files and parameters does Katago use tensor cores? Else see (6).

9) (How) can I see in process or Nvidia tools whether a running Katago process uses OpenCL, CUDA or tensor cores?

10) How do I assess whether some Katago tuning parameters are better than others on my PC?

11) How do I see numbers of playouts?

[Cassandra]

Currently, I only know that OpenCL on the dGPU must be working. I do not know yet:

1) ... 9)

In my experience, CUDA does not cause any problems (for the use cases game and analysis), but everything necessary should be installed.
You might want utilising the CUDA version first, as lightvector wrote that TensorRT will most likely run once CUDA did.

KataGo's TensorRT version:
As lightvector already wrote, you might have to install the original NVIDIA TensorRT drivers to your system.
As Karl once told me, he was ultimately successful with the TensorRT installation, although it was probably a bit tricky.
You will have to decide for yourself to what extent it is worth the additional effort for your purposes.

10) How do I assess whether some Katago tuning parameters are better than others on my PC?

During its installation process via Baduk Megapack, KataGo should have created a config-file suitable for your system.

If that is not enough for you, a few rounds of "trial and error" will be necessary.
And as long as it is not clear for which purposes you want to use KataGo, even lightvector will not be able to help you set the appropriate parameters.

11) How do I see numbers of playouts?

I have no experience with KaTrain (but with Sabaki only), but I think that the realisation of this in the various surfaces should be similar:

In Sabaki during ANALYSIS, both the winrate and the number of visits are displayed at the respective candidate move's point.
During "PLAY GAME" or "GENERATE MOVE", this data is displayed in the console window. You will also get several additional parameters for KataGo's top-10 candidate moves.

[RobertJasiek]

for which purposes you want to use KataGo

Different purposes, of course:) Among them:

- play games against strong AI
- play partial games with undos against strong AI
- continue play from a position against strong AI
- fast analysis of a game
- long analysis of a game possibly up to filling 64GB RAM
- fast analysis of a position
- long analysis of a position possibly up to filling 64GB RAM
- let strong AI continue play from a position against itself
- maybe for the fun of it let AI play against itself

It is possible that different purposes better profit from different parameters...

[Cassandra]

for which purposes you want to use KataGo

Different purposes, of course:) Among them:

- play games against strong AI
- play partial games with undos against strong AI
- continue play from a position against strong AI

The parameters will be absolutely irrelevant, in my opinion.

- fast analysis of a game
- long analysis of a game possibly up to filling 64GB RAM
- fast analysis of a position
- long analysis of a position possibly up to filling 64GB RAM
- let strong AI continue play from a position against itself
- maybe for the fun of it let AI play against itself

First walk a few steps with KataGo. And come back again if you stumble. Then there will certainly be enough people here or in the other KataGo forums who will be able to help you.

As a mathematician, you certainly know the Pareto principle.
Don't waste your time trying to get from 95% to 97% when 80% already more than satisfies your needs.

[RobertJasiek]

I prefer the best AI play immediately. This is the attitude of high dans.

(As a mathematician, only 100% correctness is valid. 97% is not mathematics but statistics;) )

[Cassandra]

I prefer the best AI play immediately. This is the attitude of high dans.
(As a mathematician, only 100% correctness is valid. 97% is not mathematics but statistics;) )

Dear Robert,

You do not really understand!
The percentages referred to the realisation of your presumed requirements / needs, not to the quality of KataGo's statements (which also rely on statistics, by the way ).

Even the "best" AI play will NEVER ALWAYS deliver "100% correctness", whatever you may understand by "correctness" (the best move in relation to the prospect of winning, the best move in relation to the size of the game's outcome, ...).
In the majority of cases, KataGo's objectives will NOT be yours.

And please always remember that KataGo (no matter how strong the network used is) will most likely NOT play "correctly" in positions that the network used has not worked through sufficiently during training (at least you cannot be sure that KataGo does). And I firmly believe that you have many positions of this type to be investigated up your sleeve.

IH120 is the most striking example.
For example, my IH120-60b (currently still) hallucinates changes in the order of moves where in fact there are none (i.e. no valid ones). And often it requires the gracious assistance of Karl's IH120-40b to determine that.
Do you know a valid solution for all the positions you want to investigate? For IH120 we already know some, which is incredibly helpful in the current phase of my IH120-60b training.

First of all, it is eminently important for you to gain experience. And that will be sufficiently possible with ANY strong KataGo network, even with the parameters' DEFAULT values!

[RobertJasiek]

Even the "best" AI play will NEVER ALWAYS deliver "100% correctness"

The joke has been beyond your perception...

First of all, it is eminently important for you to gain experience.

Your approach is not mine. For me, I prefer good settings before gaining experience. Besides, I would prefer not having to reinvent the wheel by rediscovering everything other Katago users might already have discovered as to executables, drivers and settings.

AI usage should not be alchemy but should be readily applicable.

[Cassandra]

First of all, it is eminently important for you to gain experience.

Your approach is not mine. For me, I prefer good settings before gaining experience. Besides, I would prefer not having to reinvent the wheel by rediscovering everything other Katago users might already have discovered as to executables, ...

You could already read that the benefit of a TensorRT installation may not justify the additional effort required for it.
You should start with CUDA anyway, so why not stick with it (for now)?

... drivers ...

If you are missing any, KataGo will not run. Usually there are also corresponding messages on the screen.

... and settings.

Do you know anyone else in this world who shares your needs / demands?
So try it first and then let us know what didn't go as planned.

[RobertJasiek]

You could already read that the benefit of a TensorRT installation may not justify the additional effort required for it.

No, but I could read two types of opinions, the other being: TensorRT is about 1.5 as fast on my GPU, and may or may not be stronger depending on the position and net.

You should start with CUDA anyway

Will do.

so why not stick with it (for now)?

Because TensorRT can be about 1.5 as fast. If it were 1.005, I might not care but 1.5 is extraordinarily better.

... drivers ...

If you are missing any, KataGo will not run. Usually there are also corresponding messages on the screen.

Ok, this is a useful hint.

Do you know anyone else in this world who shares your needs / demands?

Personalised meta-discussion leads nowhere.

As a broader meta-discussion, every user needs to decide between OpenCL, CUDA, Tensor and Eigen, would profit from 1.5x speed, and would profit from choosing good settings and nets with ease and confidence of sufficient understanding.

[xela]

Robert, you're recapitulating a debate that's been going on in academic circles since the 1980s. Unfortunately, "statistics and alchemy" is actually a pretty good description of machine learning. It's an empirical practice, not an exact science. It certainly isn't a branch of pure mathematics. See https://projecteuclid.org/journals/stat ... 13726.full if you're interested in the philosophy.

I'd strongly recommend that you get KataGo using your GPU in the simplest way possible, spend a few weeks using it to explore go positions, and then make a decision on whether it's worth optimising the performance.

1) Have I already installed CUDA drivers as part of the Nvidia Studio driver?

"Drivers" isn't quite the right term. CUDA is a software library for GPUs. No, most likely you will need to install it. Type "install CUDA windows" or similar into your favourite search engine, and find a set of instructions that make sense to you. Ditto cuDNN and TensorRT, if you want/need to go ahead and install them. But I'd leave this for later.

9) (How) can I see in process or Nvidia tools whether a running Katago process uses OpenCL, CUDA or tensor cores?

I'm on Linux, and I believe you're on Windows, so I can't give you the exact answer. But you probably can't, because OpenCL/CUDA/TensorRT refer to software, while the Nvidia tools will only tell you what the hardware is doing. I expect the graphics card drivers would include some sort of status monitor to show which processes are using the GPU, how much GPU memory is allocated to each one, etc. It won't be in the Windows task manager though, but somewhere else. Browse through your system tray and control panels. Or google for advice :-)

If you run katago from a command line, it will display a lot of status information, including which backend it's using. This may or may not be useful!

10) How do I assess whether some Katago tuning parameters are better than others on my PC?

Using the same network and the same board position, different tuning parameters will give you different numbers of playouts per second. Higher numbers are better.

11) How do I see numbers of playouts?

Yes, good question. I just opened up katrain for the first time in a while, and remembered why I prefer to use Lizzie. The Lizzie status bar shows both total playouts and playouts per second. I haven't found how to do this in katrain. Hopefully someone else is reading this and can answer.

[Cassandra]

CUDA is a software library for GPUs. No, most likely you will need to install it. Type "install CUDA windows" or similar into your favourite search engine, and find a set of instructions that make sense to you. Ditto cuDNN and TensorRT, if you want/need to go ahead and install them.

According to my experience, you will have to make sure that ALL directories that contain the DLL's from these software libraries are known to your system via the PATH environment variable.
You will have to add some of these manually (Google will help you with this), if this has not already been done by the respective installation programme.

But I'd leave this for later.

I'd strongly recommend that you get KataGo using your GPU in the simplest way possible, spend a few weeks using it to explore go positions, and then make a decision on whether it's worth optimising the performance.

I second this.

+ + + + + + + + + +

Most likely, you will (also) want to use KataGo to confirm your theories or to gain new insights into them.
In the process, as I have already mentioned, you will run into challenges that have NOTHING to do with performance. The more "unusual" the position you want to investigate, the higher the probability that KataGo will have to rely on its "global" Go knowledge only. KataGo will still play super-strong, but NOT as well and reliably as if the network used had sufficiently researched this (type of) position in its training phase.

In the games of my IH120-60b against Karl's IH120-40b, it may well happen that 60b has (sometimes noticeably) more playouts than 40b (per adjusted time unit, because 60b is by principle only half as fast as 40b).
In my estimation, this is an indication that 60b is more familiar (from the training) with the corresponding position than 40b.
In individual cases, this may well result in 40b being "fooled".
And this happens even though 60b is little trained compared to 40b and usually has no chance against 40b. Mainly because 60b currently still has the assessment that White will clearly win, and consequently tries avoidance strategies that naturally end in disaster.

In the time of Karl's 40b project, we have noticed that KataGo seems to have a strange preference for Triple-Ko, which is of course not so optimal in cases where one should correctly settle for a Double-Ko.
(Disclaimer: we do not know to what extent this is specific to IH120 only.)

[RobertJasiek]

CUDA is a software library for GPUs. [...] OpenCL/CUDA/TensorRT refer to software

For three years, I have watched Youtube videos and read tech webpages on graphics cards. Everybody told that they have CUDA cores, tensor cores and other cores. Nobody there has ever mentioned that whatever cores could not be simply used by some softwares. Therefore, apparently naively I expected that the Nvidia either Gaming or Studio drivers would simply enable all software to use whichever cores it prefers to use.

On my ordinary PC, the iGPU uses several drivers, among which is C:\Windows\System32\OpenCL.dll. Therefore OpenCL can be both a driver and a software.

Now, from your explanations, I start to understand that there are CUDA cores, and CUDA or CuDNN software, for which Nvidia also refers to as containing more drivers. Similarly, there seem to be tensor cores and TensorRT software.

Hence I guess that KataGo might rely on drivers and dynamically linked software of Nvidia's software packages for CUDA cores and / or tensor cores.

I see, so the softwares mentioned in Youtube videos and on ordinary webpages might not be sufficiently advanced to need Nvidia's software packages while machine learning software, such as KataGo, is more advanced in its software design and needs them.

I hope I understand this roughly right.

[RobertJasiek]

Most likely, you will (also) want to use KataGo to confirm your theories or to gain new insights into them.

You must distinguish my theories by their natures.

1) Mathematical theory established as proved theorems: I do not need any AI ever to confirm such theory because it is already established truth due to the proofs. We can only study whether AI is able to reach the same level of 100% correctness for those situations to which the theorems apply or whether AI does (much) worse.

2) Other go theory formulated as principles, methods or values for which there is a high correlation to professional / strong players' play: We can study whether AI agrees or how / when it disagrees.

3) Other go theory formulated as principles, methods or values that is useful to some extent but does not have a high correlation to professional / strong players' play: Our study of human or AI play might enable improved theory.

Apart from theory, there are also example positions for which I want to see whether AI "analyses" better than I have done. In particular, I am curious about my triple ladder analysis for a pro game's position. Will AI even be able to construct the complete ladders?

[RobertJasiek]

After a break, I have touched KaTrain again and changed the executable from <path>\lizzie\katago.exe to <path>\lizzie\katago_cuda.exe. This has been enough to run the latter. By name, I guess this means it is the CUDA version of KataGo. Of course, I do not trust names and have studied running files, processes and drivers in ProcessExplorer and Explorer as below.

However, first let me observe the different behaviours of the graphics card as to GPU and VRAM loads as follows. The CUDA version uses a bit more VRAM but loads the GPU more efficiently.

Code: Select all

Item       katago   katago_CUDA

GPU load   94%      81 ~ 90%

VRAM load  1.15GB    1.8 GB

Note what I have, or have not installed as follows. Instead of knowing in advance whether the Nvidia Studio drivers and Baduk AI Megapack have, or have not, already installed CUDA drivers and due to missing statements by experienced users, I have had to find out by trial and error that - apparently, I cannot be sure yet - CUDA drivers and CUDA libraries have already been installed. By file names, it appears - but again I cannot be sure yet - that CUDNN libraries have already been installed.

I do not know if some of the libraries must be in the same directory as the used katago_cuda.exe. It just happens to be so for the one I have been using so far. However, I cannot know whether this is a necessity. Is it?

From the Katago webpage and statements by some experienced users, there has been the strong recommendation to install cuda_12.1.1_531.14_windows.exe and cudnn-windows-x86_64-8.9.2.26_cuda12-archive.zip for CUDA and CUDNN libraries. However, I am not about to program my own neural net, have experienced that unnecessary driver / library installations can corrupt a system (in fact, for my very new PC, I have already experienced this for the AMD iGPU drivers) and it seems that katago_cuda.exe runs without the extra 4GB of installers. Therefore, at least before proceeding to tensor cores, it seems that their installation has been a bad recommendation. And this is what I call alchemy: forcing each go AI user to find out the correct installation procedure by trial and error.

Code: Select all

NOT INSTALLED YET

cuda_12.1.1_531.14_windows.exe        Nvidia CUDA installer               3,3GB
cudnn-windows-x86_64-8.9.2.26_cuda12-archive.zip
                                      Nvidia CUDNN installer-ZIP          0,7GB

INSTALLED

Baduk_AI_Megapack_v4.18.0_x64.exe     Baduk AI Megapack

KaTrain Command

<path>\lizzie\katago_cuda.exe analysis -model <path>\lizzie\KataGo40b.gz
 -config <path>\KaTrain\analysis_config.cfg -analysis-threads 12
 -override-config homeDataDir=C:\Users\<username>/.katrain

Instead, I want to contribute information with which future AI newbies can make more informed decisions than mine. I have observed the following processes, libraries etc. on my new PC, among which many indicate (OpenCL and) CUDA and CUDNN. So if you do not know yet whether you have already installed, or still need to install, such, check for the following files or processes when running katago_cuda.exe:

Code: Select all

Go AI 64b Processses

<path>\KaTrain\KaTrain.exe            KaTrain                              1.7.2.0
<path>\lizzie\katago_cuda.exe         katago_cuda.exe
C:\Windows\System32\conhost.exe       Host für Konsolenfenster             10.0.22621.1194

Lizzie Katago

<path>\lizzie\katago_cuda.exe         katago_cuda

Lizzie NVIDIA DLLs

<path>\lizzie\cublas64_11.dll         NVIDIA CUDA BLAS Library             11.7.3.1
<path>\lizzie\cublasLt64_11.dll       NVIDIA CUDA BLAS Light Library       11.7.3.1
<path>\lizzie\cudnn_cnn_infer64_8.dll NVIDIA CUDA CUDNN_CNN_INFER Library  11.4.128
<path>\lizzie\cudnn_ops_infer64_8.dll NVIDIA CUDA CUDNN_OPS_INFER Library  11.4.128
<path>\lizzie\cudnn64_8.dll           NVIDIA CUDA CUDNN Library            6.5.0
<path>\lizzie\                        <contains more CUDA files>

Lizzie OpenSSL DLLs

<path>\lizzie\libcrypto-1_1-x64.dll   OpenSSL library The OpenSSL Project
<path>\lizzie\libssl-1_1-x64.dll      OpenSSL library The OpenSSL Project

Lizzie Misc

<path>\lizzie\libz.dll                zlib data compression library
<path>\lizzie\libzip.dll              libzip for Windows
<path>\lizzie\                        <contains more library files>

Nvidia Studio Driver DLLs

C:\Windows\System32\nvapi64.dll       NVIDIA NVAPI Library                 531.61
C:\Windows\System32\nvcuda.dll        NVIDIA CUDA Driver                   531.61
C:\Windows\System32\                  <contains more Nvidia (CUDA) files>
C:\Windows\System32\DriverStore\FileRepository\
nv_dispsig.inf_amd64_89cdd9f6f9724565\nvcuda64.dll
                                      NVIDIA CUDA Driver                   531.61
C:\Windows\System32\DriverStore\FileRepository\
nv_dispsig.inf_amd64_89cdd9f6f9724565\
                                      <contains more Nvidia (CUDA) files>

Nvidia System Services

C:\Windows\System32\DriverStore\FileRepository\
nv_dispsig.inf_amd64_89cdd9f6f9724565\nvcubins.bin

NVIDIA C:\Windows\System32\DriverStore\FileRepository\
nv_dispsig.inf_amd64_89cdd9f6f9724565\Display.NvContainer\NVDisplay.Container.exe
                                      NVIDIA Container                     1.37.3103.4323
"C:\Windows\System32\DriverStore\FileRepository\
nv_dispsig.inf_amd64_89cdd9f6f9724565\Display.NvContainer\NVDisplay.Container.exe"
 -f %ProgramData%\NVIDIA\DisplaySessionContainer%d.log
 -d C:\Windows\System32\DriverStore\FileRepository\
nv_dispsig.inf_amd64_89cdd9f6f9724565\Display.NvContainer\plugins\Session
 -r -l 3 -p 30000 -cfg NVDisplay.ContainerLocalSystem\Session -c


RTX 4070 drivers, extract

C:\Windows\System32\drivers\NVIDIA Corporation\Drs\dbInstaller.exe
C:\Windows\System32\drivers\NVIDIA Corporation\Drs\nvdrsdb.bin
C:\Windows\System32\drivers\NVIDIA Corporation\license.txt
C:\Windows\System32\lxss\lib\libcuda.so
C:\Windows\System32\lxss\lib\libcuda.so.1
C:\Windows\System32\lxss\lib\libcuda.so.1.1
C:\Windows\System32\lxss\lib\libnvcuvid.so
C:\Windows\System32\lxss\lib\libnvcuvid.so.1
C:\Windows\System32\lxss\lib\libnvidia-ml.so.1
C:\Windows\System32\lxss\lib\<various>
C:\Windows\System32\MCU.exe
C:\Windows\System32\nvapi64.dll
C:\Windows\System32\nvcpl.dll
C:\Windows\System32\nvcuda.dll
C:\Windows\System32\nvcuvid.dll
C:\Windows\System32\OpenCL.dll
C:\Windows\System32\<various>
C:\Windows\SysWow64\<various>
C:\Windows\System32\DriverStore\FileRepository\nv_dispsig.inf_amd64_89cdd9f6f9724565\<various>

Needless to say, I still have the questions on tensor cores, to start with: have I already been using them?

[RobertJasiek]

For possible later use, I have downloaded Nvidia's installers / ZIPs for CUDA, CuDNN and TensorRT. The files are not always labelled Windows 11 but sometimes Windows 10. The installation instructions are often outdated referring to older file versions in various combinations. Since I cannot know yet which combination of installers / ZIPs will work, I have downloaded several of their file versions. Once in a lifetime, Nvidia is not at full fault but only at partial fault for its outdated installation instructions and cryptic installer versions. With only 14 minutes download time left, the download of 5 of some dozen files stopped. It has turned out that this is my fault: my download partition got stuck at 0KB remaining space - it was full! At least, Windows kept working flawlessly (maybe because it is not my system partition). So I had to clean up the partition, login to Nvidia's webpage again and start the 5 remaining downloads afresh for another some half an hour.

Meanwhile, I have noticed that KataGo 1_13_0 of Baduk AI Megapack is reported to have a bug for tensor cores. So I have also downloaded the install ZIP of KataGo 1_13_1. Plus every installation instructions I could get hold of, of which some address the PATH jobs. Furthermore, I will install a tool listing files of directories so I can protocol any damage further installation might do to some files when older installers might replace newer with older files, because such is exactly what can happen (and has happened in the past). Therefore, I guess I might have every installer / ZIP I possibly need.

I think the order of installation is:
1. Nvidia GPU driver (done)
2. Nvidia CUDA version x
3. Nvidia CuDNN for CUDA version x
4. Nvidia TensorRT for CUDA version x
5. If necessary, set Windows system settings | environmental PATHs.

Independently, install KataGo 1_13_1.

Am I on the right track?

Even so, I still do not understand what in KataGo 1_13_1 enables use of tensor cores instead of CUDA cores. In earlier days, there were KataGo installers / ZIPs specific for Tensor[RT] but currently there are not. Is the KataGo 1_13_1 for CUDA also for Tensor[RT]? Is it sufficient to have told Baduk AI Megapack to create the right configuration files for using tensor cores or what, besides the command (line) calling KataGo in KaTrain or another GUI program, do I also need to do to let KataGo actually use tensor cores instead of CUDA cores?