It is currently Thu Mar 28, 2024 4:36 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 13 posts ] 
Author Message
Offline
 Post subject: AI matching rates
Post #1 Posted: Fri May 19, 2023 3:15 am 
Oza

Posts: 3647
Liked others: 20
Was liked: 4626
I have seen some data from Japan, China and Korea that purports to measure the rate at which the moves of past masters match the moves recommended by AI (usually Katago).

On the whole, the past masters seem to score very well (according to pro commentators, not me), and of course there is also some interest in comparing players relative to each other, even if they score badly against Katago.

But can the tests be trusted?

The latest list I have seen gives matching rates from 54% down to 46% for players in the following order:

Huang Longshi
Fan Xiping
Honinbo Shusaku
Honinbo Jowa
Shi Xiangxia
Zhou Donghou
Guo Bailing
Honinbo Dosaku

This list doesn't surprise me much, on the basis of the many opinions I have read about each player, but I would have liked to see other players included. In particular, I have seen Shuwa ranked above Shusaku elsewhere. I would have liked to see Xu Xingyou, Zhou Xiaosong and Chen Zixian included, and would expect them to be ranked with those above.

But I am talking there of my own impressions. When it comes to tagging figures on players, the first and obvious question has to be: what is the methodology? This never seems to be given in any detail. I would expect many factors to make a significant difference: age, stage in career, level of opponents, handicaps, etc, etc. Then for the old players there is the important question of how many games remain, and sub-questions such as: are the games that survive only the selected best ones?

I am also puzzled by the lack of references to a comparison point among modern players (Go Seigen would be a fascinating one). I have heard that Sin Chin-seo scores, at least sometimes, over 90% matching rate, but I infer, from the way I've read about that, that they may count a match of his move with any the top N (3?) of Katago's moves as a match. I have no idea what the method is for any other list or player I've seen.

So, some questions:

1. Are these comparisons worth making?
2. If so, what is the best and/or typical methodology for the matching rate?
3. If results are in any way valid, are the results skewed a lot in favour of very modern players by the fact that they have had a chance to see the AI style of play?
4. Why have I got the impression that players like Go Seigen are excluded from this research?
5. Is there anywhere all this data can be found online.? All my information have come from written sources.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #2 Posted: Fri May 19, 2023 3:30 am 
Judan

Posts: 6087
Liked others: 0
Was liked: 786
An important aspect is playing style.

- Some playing styles may be closer to AI play than others.

- Styles with long and comparatively well played endgames may be closer to AI play on average over whole games.

Unless all factors can be correlated to percentages, such data say rather little. It is fun to read such lists but more likely than not we cannot trust them.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #3 Posted: Fri May 19, 2023 7:47 am 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
John Fairbairn wrote:
2. If so, what is the best and/or typical methodology for the matching rate?
 
Depends on what meaning would you like to read into the results?

If this is about the strength of those old players (or comparing them), a raw matchrate % seems dubious. A player can have a low matchrate by playing non-bot like moves - that are still not necessarily bad pointwise.

I think a more interesting method is running a deep search before and after each move, comparing the bot's score evaluations, to estimate the points dropped by the game move. This gives an error distribution that can be studied, compared between players etc. In a few years this might even become a common automatic function in modern go servers.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #4 Posted: Fri May 19, 2023 8:42 am 
Gosei

Posts: 1625
Liked others: 542
Was liked: 450
Rank: senior player
GD Posts: 1000
My first reaction is that this "matching" mostly provides entertainment rather than a valid method for comparing human players. For example, I wouldn't be surprised to see human player A rated higher than human B according to this method but in practice B tends to beat A. This gives me a feeling of being related to the problems of tie-breaking in tournaments. The reason for this phenomenon is that human players are affected by circumstances outside the game itself. I'm thinking about such things as personal feelings about the opponent or politics. Another issue could be that player A matches AI 30% of the time and player B matches 24% of the time but their agreement with AI occurs in rather different areas of the game. We see this in human-human go where some player is quite strong in the opening while another is very strong in the endgame.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #5 Posted: Fri May 19, 2023 3:15 pm 
Dies with sente

Posts: 89
Liked others: 8
Was liked: 27
John Fairbairn wrote:
Are these comparisons worth making?

Not really, it is just for fun. Such comparisons seem to be based on several mistaken ideas.
First, that the AI comes with the best move. But this "best move" sometimes turns out to be a bad mistake if the number of visits is increased or another AI is used.
Next, that it would be desirable to play the best moves found by the AI. But such play may lead to positions that can be defended only by reading many variations to great depth. A human would be wise to create somewhat more robust positions.
Third, that play is independent of one's opponent. But often one knows what can be expected from a given opponent.
Fourth, that the human player and the AI are optimizing the same function. But especially when one feels one is behind one may wish to play risky moves that have the potential to turn the tables.


This post by vier was liked by: Waylon
Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #6 Posted: Sat May 20, 2023 9:46 am 
Lives with ko

Posts: 163
Liked others: 19
Was liked: 32
It appears to work for chess, I see no reason why it shouldn't for Go.

There is a research paper that uses the interpretation of a game as a Markovian model and a comparison with Stockfish to analyse all 26'000 games that the world champions played. The prediction of the winning chances for the games that they played among themselves is appareantly better than what their ELO rating would predict.

Wikipedia: https://en.wikipedia.org/wiki/Compariso ... ut_history
(Section Moves played compared with computer choices)

Referenced article: https://content.iospress.com/articles/i ... al/icg0012

_________________
If something sank it might be a treasure. And 2kyu advice is not necessarily Dan repertoire..

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #7 Posted: Mon May 22, 2023 11:14 am 
Lives in sente

Posts: 724
Liked others: 1023
Was liked: 30
Rank: BGA 3 kyu
KGS: Elom, Windnwater
OGS: Elom, Elom0
Online playing schedule: The OGS data looks pretty so I'll pause for now before I change it.
The correct way seems to be simply to find the mean percentage drop per move and minus that from 100 to get the accuracy. When you're already at less than 10% AI faith/confidence in a game, playing sub-ootimal moves doesn't lower the percentage much.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #8 Posted: Mon May 22, 2023 2:39 pm 
Lives in gote

Posts: 392
Liked others: 29
Was liked: 176
GD Posts: 1072
Elom0 wrote:
The correct way seems to be simply to find the mean percentage drop per move and minus that from 100 to get the accuracy. When you're already at less than 10% AI faith/confidence in a game, playing sub-ootimal moves doesn't lower the percentage much.


That seems like one option, although it might also be possible to use the absolute point loss rather than winning percentage.

In either case, it seems like this would only be usable when the game is "close". Based on Michael Redmond's comment about him being unable to easily detect a 60% advantage, we might guess something like a winning estimate of <70%. Beyond that we might argue a player was playing conservatively to guarantee a win.

Another thought I had recently was along the line of John Fairbairn's comment that professionals are strong--that is to say, the game changed a lot when the AI came along, but really not that much in the grand scheme of things. Professionals played largely good moves, but had misjudged certain positions and josekis. Looking at play at the top level, my impression is that modern pros do quite well in sequences of moves, punctuated by largely mistakes in direction of play.

I am wondering if we could perhaps examine the time between big mistakes or something similar as a measure of goodness?

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #9 Posted: Tue May 23, 2023 12:44 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
I think point loss would be the better metric to use, because % loss will depend on the strength of the AI used, growing more extreme as AI grow stronger in the future. A perfect player may give an objective (integer) point loss for each move, but it would not be able to give % losses other than 0%, 50% and 100%.

To approximate the score loss evaluation that a perfect player would give, one could tally moves that cause AI score evaluation to cross integer values (relative to the komi). Assuming that ranks are spaced by a score difference of twice komi, I expect a rank gap of 1 will result in ~0.1 extra average score loss per move in a 1st approximation (~12 points over ~120 moves per player in a full game).
I suspect there also needs to be some weight factor applied to those score losses, and a lot of experimentation would need to be done to empirically determine a weight factor function that results in a decent match with actual ranks.

Matching rates are an indication of level as well, but IME it's not a very accurate metric.
Using KataGo some 1000 visits/move, my matching rate (blue moves) varies between ~30-50% in correspondence games, while you may find ~50-80% matching rate in pro games. So a 50% matching rate won't be enough to distinguish between 3d level and 9d level play. I don't think from a single game with 50% match rate you'd be able to give a more accurate rank estimation than 5d EGF ±4 ranks, which may not be as accurate as we might hope, and I expect the error bars to grow even bigger with lower match rates.
Also, blue moves match% can be quite sensitive to differences in playing style between the specific player and the specific reviewing AI, especially in the earlier stages of the game.

So I suspect a metric based on score loss would be the more robust metric, as I expect it to be more future-proof than win% and not as dependent on playing style as blue move match%.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #10 Posted: Tue May 23, 2023 2:45 am 
Lives in sente

Posts: 724
Liked others: 1023
Was liked: 30
Rank: BGA 3 kyu
KGS: Elom, Windnwater
OGS: Elom, Elom0
Online playing schedule: The OGS data looks pretty so I'll pause for now before I change it.
pwaldron wrote:
Elom0 wrote:
The correct way seems to be simply to find the mean percentage drop per move and minus that from 100 to get the accuracy. When you're already at less than 10% AI faith/confidence in a game, playing sub-ootimal moves doesn't lower the percentage much.


That seems like one option, although it might also be possible to use the absolute point loss rather than winning percentage.

In either case, it seems like this would only be usable when the game is "close". Based on Michael Redmond's comment about him being unable to easily detect a 60% advantage, we might guess something like a winning estimate of <70%. Beyond that we might argue a player was playing conservatively to guarantee a win.


As I said, and this is self-evident, if a player is at 30% winrate or lower moves that aren't objectively the do not lower the winrate much because there isn't much winrate left to lose. In fact this is why it's better than objective point loss. If you play a game and early on you were at 50%, but play a move that loses 2 territory points and takes you down to 45%, and then in the endgame one player is at 3% winrate, then if either player plays a move that loses 50 territory points the maximum percentage loss will be 3%. You're making the common fallacy of assuming a certain act is uniquely human and wouldn't be understood or naturally done by AI, but you have to remember that AI naturally always plays AlphaGo is winning and would lose points to win by half a point or completely tilt when it's losing. AI not only understands the concept of playing conservatively to win or playing recklessly to come back from a losing position, it does it far more than humans and we have to actually manually add to the neural net the desire to care about points as much as humans when it's playing. It's quite self-evident that accuracy based on 100 minus the mean percentage drop per move naturally ignores moves played when one player is already ahead because the correlation between point difference and percentage gets weaker. Absolute point loss is the worst because you precisely can only use it when the game is close . . .

Forget AI for a moment, if a human judged a collection of a pros games by how likely each move makes it possible for the player to win, they would know when a move losses points but increases winning chance by making the game more complicated or simple. AI does this to the extreme. On the other hand, in both cases if we focus on point loss, then the issue your talking about arises

In fact the real problem is that comparing post-2016 games to pre-2016 games might not be fair because humans learned from AI moves.

We should run this experiment for all players from the most ancient Chinese players all the way to the top players in 2015. Then from 2016 onwards, we do two versions, one the same as before but another that adjusts the weighting of each move in the calculation of mean percentage drop according to distance to the middle move in the game record, for example in a game with 270 moves, moves 135 and 136 would have a weighting of 99.631%, which is how much a percentage drop from that move would be multiplied by in the mean calculation, but the weighting for moves 1 and 270 would be 0.738%, meaning the percentage drops of those moves would mean very little.

We should do it for the 100 most important games played by each player, to make it fair. It would be really interesting to see how Fujisawa Hideyuki and Wu Qingyuan ranks . . .

Quote:
Another thought I had recently was along the line of John Fairbairn's comment that professionals are strong--that is to say, the game changed a lot when the AI came along, but really not that much in the grand scheme of things. Professionals played largely good moves, but had misjudged certain positions and josekis. Looking at play at the top level, my impression is that modern pros do quite well in sequences of moves, punctuated by largely mistakes in direction of play.

I am wondering if we could perhaps examine the time between big mistakes or something similar as a measure of goodness?

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #11 Posted: Tue May 23, 2023 4:53 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
Perhaps a good measure of go skill would be the expected point loss L(T) at temperature T, i.e. associate a curve to each player. Then we could compare L(5), L(10) or L(20) for different players. Or perhaps the sum over T of p(T)L(T) where p(T) is the probability that a position in a pro game has temperature T.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #12 Posted: Tue May 23, 2023 4:55 am 
Oza
User avatar

Posts: 2408
Location: Ghent, Belgium
Liked others: 359
Was liked: 1019
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
I assume a matching rate of x% means that in x% of the moves, the master's move was close enough to the AI move.
Closeness is measured by difference in winning percentage (threshold y percentage points), or by difference in predicted end score (threshold z points).

I would further assume the thresholds y and z are kept stable throughout the game.

And as John says, we don't know if certain games get a higher weight in the overall computation of x. Youth games, handicap games, casual games, faster games ...

For amateurs, but even for professionals I reckon, the matching rate ignores how bad the deviation is. From my own game reviews with AI, I have long concluded the precision matters a lot more than the accuracy. That is, my chance of winning will be more drastically improved by reducing the margin of error on each move, than by increasing the number of good moves.

Top
 Profile  
 
Offline
 Post subject: Re: AI matching rates
Post #13 Posted: Tue May 23, 2023 6:21 am 
Lives in sente

Posts: 724
Liked others: 1023
Was liked: 30
Rank: BGA 3 kyu
KGS: Elom, Windnwater
OGS: Elom, Elom0
Online playing schedule: The OGS data looks pretty so I'll pause for now before I change it.
jlt wrote:
Perhaps a good measure of go skill would be the expected point loss L(T) at temperature T, i.e. associate a curve to each player. Then we could compare L(5), L(10) or L(20) for different players. Or perhaps the sum over T of p(T)L(T) where p(T) is the probability that a position in a pro game has temperature T.


As I said point loss will always be a very bad way to measure go skill compared to winrate percentage loss, since point loss doesn't take into account when players play risky or safe moves when they're behind or ahead, but winrate percentage always does since playing safe when you're ahead is just as likely to increase winrate, it only won't if a move the player thinks is risky AI doesn't see as risky. And no matter what if you're at a low percentage it doesn't matter if you play like a 30 kyu you can't lose that many winrate points no matter what.

Knotwilg wrote:
I assume a matching rate of x% means that in x% of the moves, the master's move was close enough to the AI move.
Closeness is measured by difference in winning percentage (threshold y percentage points), or by difference in predicted end score (threshold z points).

I would further assume the thresholds y and z are kept stable throughout the game.


I really, really dislike the method of considering a move that is close enough or one of the candidates a 100% match. I have no idea how this strange idea came into peoples heads, I would never think of it yet alone assume it, yet it's somehow the normal assumption. Then again, this pattern is pretty much my life . . . Haha!

Quote:
And as John says, we don't know if certain games get a higher weight in the overall computation of x. Youth games, handicap games, casual games, faster games ...


Yes, for each player we should use 100 games with a maximum one stone equivalent handicap from serious competitions with long time limits.

Quote:
For amateurs, but even for professionals I reckon, the matching rate ignores how bad the deviation is. From my own game reviews with AI, I have long concluded the precision matters a lot more than the accuracy. That is, my chance of winning will be more drastically improved by reducing the margin of error on each move, than by increasing the number of good moves.


Yes! Another reason why it's silly to even consider AI matching rate anything other than the continuos average linear percentage drop per move. Forget silly discrete methods, for the sake of basic common sense! I thought computer type people were supposed to be smart but in this instance they used a sill methodology a 10-year old could demonstrate as inferior to linear percentage drop! I'm disappointed.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group