Although some current AI programs are said to be 2 stones stronger than top pro level. However, they are much weaker than AlphaGo Zero. E.g., Leela Zero (via AI Sensei) is described as having occasional problems with ladders and sekis, although not every weak or lazy user notices them. I have witnessed another occasional weakness: wrong LD status assessment. Therefore, never simply trust AI. When will current AI reach AlphaGo Zero level on typical computers?

To be fair we don't know how strong AG0 is on typical computers, just on high end hardware. The AlphaGo teaching tool (better Master not Zero) is on 10 million playouts, maybe it too made embarrassing ladder mistakes on 3000 playouts like LZ does and DeepMind wanted to hide that bad publicity. Does anyone know how many playouts Master was getting in the online series? I seem to recall that was on a single machine with a TPU or two in sub 30 seconds.

Well perhaps we should distinguish three, not jus two levels of computers.

a) Typical computers
b) high end gaming machines (probably a couple K to buy and maybe 300w to run)
c) high end hardware, equivalent of a mini supercomputer. Maybe even water cooled.

There were some claims on LZ discord that most of today's bots, including LZ and Minigo already reached AG0 levels (raw nets I guess, hw oc makes difference).

This question has some practical consequences since LZ for example is struggling to gain further strength at the moment, and it is unclear if this is already the limitation of 40b networks (which AG0 used as well), or a consequence of some other problem.

hw = hardware?



There is no guarantee that the fitness landscape for go is unimodal. Perhaps LZ has climbed the wrong hill. OC, in infinite time it will find its way to another hill.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

LeelaZero has 16 million self-play games, while AlphaZero has 29 million, so Leelazero may be a bit less strong than AlphaZero, but is probably not very far.

How many playouts are necessary to avoid problems with ladders, life and death, semeais, etc.?

There may be path dependency.



Well, what do you mean by a problem? Certainly if human amateurs can do better with such things in some positions, that may be considered a problem. John Tromp has shown that ladders can pose very difficult problems, and even some "simple" ladder problems are well over 100 go moves deep. Similarly, Berlekamp and Wolfe composed endgame problems that are easy for humans who know the secret, but are almost 100 moves deep, and the candidate plays at each turn are very similar, so that it is unclear how much MCTS resembles brute force for such positions.

For the Elf commentaries, candidate plays, except for the actual game play, with fewer than 1500 playouts were not reported. If Elf's top choice had few competitors, it typically got more than 100k playouts. OC, that was not enough to avoid occasional reading errors where the human choice was demonstrably better.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Let me ask a more precise question. Are there examples of human games (pro or amateur) in which a recent version of LeelaZero with 100000 playouts chooses a wrong move because of a misread ladder, or life-and-death problem, or semeai?

If the answer is yes, then by which number should I replace 100000 so that the answer becomes "no"?

I trust AI, even if it makes some mistakes from time to time. I even trust my fellow human beeings, even if they make some mistakes from time to time. The concept to only accept flawless results is inherently dangerous, allthough many people do not realize this, instead of the obvious examples in history and personal experiences.

It is better to be aware that mistakes will happen and react to them in proper ways.

(I acknowledge that the quote says never SIMPLY trust AI.)

Yes.


In my experience random differences in end results are usually small with different training runs using identical parameters. Seemingly minor differences having unexpectedly large effect on the final net is more common. (In that case the above mentioned question: what did LZ do wrong?)


It is not easy to compare LZ nets to AG0 nets since the published games not only use high playouts, but are also hand-selected (which may explain some brilliancies). But I doubt LZ would be near AG0 atm. Comparing selfplay numbers also unreliable since there ARE/were significant differences, so a factor of 2 in selfplay amount can easily slip left or right.

The question about ladders is even more difficult. There were some recent examples where LZ (on normal low-ish playouts) misplayed ladders with only a few disturbing stones in vicinity. Low and even mid playouts must rely on net guess about ladder to an extent, since actual playouts may not go along the ladder line for the player the net thinks the ladder is bad for (thus wont see its real outcome until very high playouts).

Yes, I lost faith in the DeepMind team when they only published selected games, which made them useless for scientific purposes. Good PR, I suppose. You got the same kind of thing in chess. They knew better.

At least the Master vs. human games were not selected, and the Elf commentaries are comprehensive. (Different team. )

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Except that they only chose to reveal that Master was associated with AlphaGo *after* it showed an impressive result

_________________
be immersed

That's true.

But the games were still representative. No selection bias.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

So using something like a Huawei Atlas 900 cluster and the latest LZ, that means it can beat the AlphaGoZero?

Would Huawei and Google we willing to do a match?

also for what it is worth, had they played Lee with the Fan version of AG, good chance Lee would have won the match, had they played Ke Jie with the Lee version of AlphaGo, good chance Ke Jie might have won some games etc... I think they were conservative in that they wanted to assure wins at each step up the AI strength ladder, while in retrospect it seems clear the Fan version of AlphaGo was very weak and not "superhuman", certaintly current LZ is already past even the Lee version of AlphaGo

Of course. There are huge problems with complicated fights.

The number depends on how FAR the complicated fight is. If it's right now, maybe 1 million can resolve it. If the fight is in ONE MOVE it might take ten million to see and move elsewhere. If it's in three moves, then one hundred million may be necessary.

In other words, there's a horizon effect where it takes much more power to see the surprising way you can lose the game.

If you or someone has a concrete example of such a complicated fight, I would be interested (not an artificial example, but an example from a real game).

I don't know about LeelaZero, but Elf has misread a semeai in its commentaries, which use more than 100k playouts. See https://lifein19x19.com/viewtopic.php?f=15&t=16641

Edit: zermelo reported that a recent version of Leela Zero did not find the correct play in 1.5 million playouts. https://lifein19x19.com/viewtopic.php?p=244857#p244857

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I recall that the teaching tool took 10 minutes for 10M playouts, so with 30 seconds on the same hardware it would reach 500k playouts.