Personal library of mistakes

[Jæja]

Hi everyone,

I would love to use a software application that allows me to store and study mistakes I made in my (online) games, for example using spaced repetition. Ideally it would work like this:

Analyzer module
- I select a folder holding my SGF files and the application will monitor it to check for new files.
- The application will invoke an AI to analyze the games and find mistakes. I can set what a mistake is using e.g. an X percent drop in winning rate.
- The application will turn each mistake into a kind of flash card. It knows who I was in the game because I entered a list of my online handles. Optionally I could turn my opponent's mistakes into flashcards as well.

Study module
- The main idea of spaced repetition is that "...newly introduced and more difficult flashcards are shown more frequently while older and less difficult flashcards are shown less frequently and to increase rate of learning" (Wikipedia).
- The application keeps track of these statistics: card last answered correctly/incorrectly Y cards ago and use it to determine which card to show next.
- Perhaps it could allow for tagging flash cards as well, using a list of common mistakes like this one (e.g. overplay, slow, etc.) I could then study a certain mistake I've made in the past.

Does something like this exist? If not, what do you think of this idea?

[hyperpape]

It's a cool idea, but I've not heard of anything like that.

One question is whether you want a manual review. Do you think you'll gain from studying mistakes where you know the machine's verdict, but don't know the reason? Or is pruning them better?

[jlt]

The point of flashcards is to forget your bad habits and replace them by good moves, so it would be better to categorize the correct moves (connect, make good shape, find a big point, find an invasion point...) instead of categorizing the wrong move.

[MikeKyle]

I tried to do something very similar to this a few years ago with Leela 11.

I am absolutely not a programmer, I just enjoy hacking around with personal projects. I use the flashcard app anki on my phone and on the web-app for learning other stuff so my project just involved making nice go flashcards for that.

I found a nice python package someone wrote for making nice images from sgf files (can't find it now but I could fish around if it would really be useful.) I wrote one basic script that took a file and a move number as input and output two images. The "front" image for the flashcard was the position to play with the words "[black/white] to play", the back image was the same position marked with all the moves that Leela considered within 1% or so of its top choice (you could also go with just one move, but I preferred to learn the choice of strong moves.) the script also edited a text file for importing the cards into anki.
I then made a second script that scanned through a game for falls in win rate over a threshold.

I didn't last very long with these flashcards personally but playing with the code was fun for me.
My problem with the flashcards for me was that most of the mistakes were in quite specific fighting situations. Once I'd read the fight out properly with hindsight it looked obvious to me and spaced repetition didn't really feel like it was necessary to learn those specific lessons. The challenge in go is often around abstraction of learning - taking these specific learning points and linking them up with novel positions. I don't think there was a learning advantage over just playing more games and reviewing the new novel positions.

I do think it's a very cool idea though. Maybe I just needed a gentle tweak to the system to help me get more from it?

[hyperpape]

The point of flashcards is to forget your bad habits and replace them by good moves, so it would be better to categorize the correct moves (connect, make good shape, find a big point, find an invasion point...) instead of categorizing the wrong move.

I assumed it would be "X to play" in a circumstance where you'd previously made a mistake. The answer would be the right move.

[Jæja]

Thanks for all your reactions. I'm getting the feeling this might be worth working on!

The point of flashcards is to forget your bad habits and replace them by good moves, so it would be better to categorize the correct moves (connect, make good shape, find a big point, find an invasion point...) instead of categorizing the wrong move.

I assumed it would be "X to play" in a circumstance where you'd previously made a mistake. The answer would be the right move.

Correct! I think my remark about tagging might have confused things. The main idea is to learn better moves and the categories could be fun to study specific topics.

One question is whether you want a manual review. Do you think you'll gain from studying mistakes where you know the machine's verdict, but don't know the reason? Or is pruning them better?

This definitely a good point! The AI review is just to make it generic and usable for anyone, even if you don't have somebody to review your mistake. You could perhaps use the AI to find possible mistakes and mark them as such if you like.

I found a nice python package someone wrote for making nice images from sgf files (can't find it now but I could fish around if it would really be useful.) I wrote one basic script that took a file and a move number as input and output two images. ... I then made a second script that scanned through a game for falls in win rate over a threshold.

This sounds very interesting to me. I like how this could be a pragmatic, simple solution to test ideas. I might want to play with this around myself. Do you have any code you could share? If not, I'm already happy with the idea you shared, so thank you

I didn't last very long with these flashcards personally but playing with the code was fun for me. My problem with the flashcards for me was that most of the mistakes were in quite specific fighting situations. Once I'd read the fight out properly with hindsight it looked obvious to me and spaced repetition didn't really feel like it was necessary to learn those specific lessons. ... I do think it's a very cool idea though. Maybe I just needed a gentle tweak to the system to help me get more from it?

I can understand you feelings about the outcome. However, I also agree that it might be more effective if it's tweaked a little bit more.

The software is not open-sourced yet because it is undocumented; the tags, markup and workflow really need some documentation.

A large part of the open source community actually helps with documentation. Developers often don't find it interesting to do this, so other people take this out of their hands. Can we help you with this? I like what you did with this software, but the level of your problems is too high for me. However, if we can use your software as a basis, then we could make it easier for other to build problem collections. What do you think?

[Bill Spight]

However, I also agree that it might be more effective if it's tweaked a little bit more.

For flashcards, to prevent erroneous recognition, I always recommend adding a card with a small change, such as moving the position of one stone, that changes the answer. That will help you to pay attention to significant things on the card, and the exercise of producing those extra cards may help you to understand the positions better.

[Jæja]

For flashcards, to prevent erroneous recognition, I always recommend adding a card with a small change, such as moving the position of one stone, that changes the answer. That will help you to pay attention to significant things on the card, and the exercise of producing those extra cards may help you to understand the positions better.

Nice! Great idea

Already there have been a number of great ideas by several of you. Please let me know if you have more suggestions and if you'd like to collaborate on making this project a reality. I don't have enough time by myself, but I'm sure it's doable if a few of us work on it.

[thirdfogie]

A few years ago, my grade in BGA tournaments fell from 3kyu to 4kyu. Wishing
to avoid a further fall, I too decided to make a record of my most ridiculous
mistakes. I put them into a PDF document which can be loaded on to a tablet
computer for review before playing.

Jokey titles were used to ease recall. The first page of the contents list is
hidden below. It was made by converting the relevant PDF page into a .png file.
It is hoped that the result does not overburden the Lifein19x19 server.

00-contents.png (252.46 KiB) Viewed 19566 times

In the contents list, "Call that a kikashi" is an allusion to "Call that a
knife" in Crocodile Dundee, and "Don't panic!" is an allusion to Corporal
Jones. (See https://www.youtube.com/watch?v=nR0lOtdvqyg for
a surfeit of examples.)

Here is a sample page preserving three stupid mistakes.

00-calamities.png (247.66 KiB) Viewed 19566 times

There would have been more samples at this point, but only 3 attachments are
allowed per post. A follow-up post will contain some more examples, if I'm
not banned for abuse first.

Did the catalogue work as intended? It did not prevent the feared drop to
5kyu, but it probably slowed the rate of decline.

Successes

I have stopped making some of the more stupid mistakes such as reading out that
a move is bad and playing it anyway. I have also learnt to "rein in the
killing frenzy and survey the board", though I am still terrible at identifying
the most important weakness to protect in such cases. Premature nakade appears
much less often and I now know my way around the tripod group.

Another benefit of the catalogue was that its construction left an organised
collection of challenging Go positions, and it was illuminating to work through
them with Leela Zero.

Failures

Learning the status of the notchers and corner groups recorded in the catalogue
never seems to help during play. More equanimity is needed. (Maybe flash
cards would indeed be better for training than my attempts at literature.)

There has been no progress with "Don't panic" and "Beware of a Glass Dagger",
The latter refers to loosely captured enemy stones, which catch me out again and
again.

Another problem is that new hideous mistakes have begun to appear. These are
largely caused by poor concentration. I will be 71 this week, so that is no
surprise. Here are two examples which have yet to make their way into the
catalogue.

1. Deciding to play on a certain point, then finding that the stone landed on
a nearby point - the real-board equivalent of a mis-click. That has
happened twice.

2. Embarking on a squeeze which should win the game, and then forgetting
what I was trying to do and switching to saving the stones which should have
been sacrificed.

Details About the Writing

I now can imagine myself as a Go author. That means finding answers to lots of
detailed puzzles, which are hidden below.

I like to put foreign words and phrases in italics, but some Japanese terms are
so familiar to an English Go player that they no longer feel foreign. Examples
are "ko", "atari", "joseki", "hane" and the name of the game itself. To avoid
agonising over the boundary between foreign and naturalised words, the decision
was to italicise every Japanese term except "Go". But then "self-atari" looks
odd with one half italicised and the other half not.

Should it be "aji keshi", ajikeshi" or "aji-keshi"? The last one looks right
to me. Whatever choice is made, it needs to be implemented throughout.

I write "black" for the stones or groups, but "Black" for the player with the
black stones. But there are borderline cases such as "the last Black play"
where it is not immediately clear what the referent is. All the games were
face to face, so the gender of the players is known, and there is no problem
with "he" and "she".

The catalogue uses a fairly large font size to help my aged eyes. Even so,
when a numbered stone is embedded in the text, the stone is rendered too large
for the vertical spacing, which looks ugly. I decided to live with that.

Text appearing to the right of small diagrams should be vertically aligned with
the centre of that diagram. The right-hand edge of that text should be aligned
within and between diagrams. It took some time to work out how to do this so
that the alignments persist when the text grows or shrinks. I still need to
rework the whole thing according to the best methods.

Finally, here is a grumble about the lack of a universal coordinate system for
points on a Go board. This appears in the catalogue as part of a section
reminding me how to amend and process the catalogue itself.

00-coordinates.png (377.52 KiB) Viewed 19566 times

Note: edited immediately after posting to really hide the first attachment.

[thirdfogie]

Here are two more sample pages from my catalogue of calamities.
I made them a bit smaller this time.

The first page shows a case where a glass dagger in the heart proved fatal
later in the game. It also shows a truly stupid response to a cross-cut.

00-glass.png (362.67 KiB) Viewed 19583 times

The second page shows a failure to push White down to the third line, which I would
have done immediately if I had recognised the situation properly.

00-push.png (342.27 KiB) Viewed 19583 times

Whole-board situations like these might not be well served by flash cards, but I have
not tried that approach.

[Jæja]

That looks truly amazing thirdfogie I really like the clear writing, jokey titles and consistent, clear layout. It really like this idea!

I'd like to start working on something similar for my own mistakes. Could you tell us something about your method of working? I'm interested in how much effort it is to go from an actual game to an example in your book. Do you record an SGF on something like a tablet and then convert it to a printable diagram? It looks like you might be using LaTeX with the igo package. Is that correct? I've exported a diagram to EPS using GOWrite2, which was easy and produces nicely looking diagrams.

[John Tilley]

ThirdFogie - Celebrating your 71st birthday is no excuse - Iwamoto Kaoru 9p defeated Takemiya 9p by resignation in the Oza preliminaries in Feb 1977 - two days short of his 75th birthday. Takemiya was 26 years old. I find that quite amazing. Please note that I celebrated my 72nd birthday last week.....

Jaeja - I think there is a limit to what you can do with AI analysis - maybe it is related to your ranking.

Once you get into the amateur dan ranks AI becomes more useful - I had a game of mine analysed some 50 years ago, when I was 1d - the feedback from Haruyama 9p was that I was not aggressive enough - there were several times when I was failed to play kikashi. Once you know that, you can use AI to spot such things. What I don't think you can do (yet) is to use AI to build such generic advice.

ThirdFogie has put togther useful material for a 3-4 kyu player, I don't think you can use AI to build this generic material for a SDK or DDK Player, but you could write some code to run on top of AI to look for certain basic mistakes - eg playing atari unnecessarily, not playing kikashi, playing too close to strength, leaving weak groups about, making vacant triangles etc etc

You could then run an AI analysis of a game, look at the big % swings and then see if any of these are due to these basic mistakes.

I have to say that after a gap of some 30 years in playing at go clubs, I found that SDK players were making the same mistakes that were being made in the 1970s..... The hardest thing is taking these mistakes seriously enough. If I had been paid £5 every time someone made a vacant triangle or played an unnecessary atari at the Winchester Club, I would have been quite well off!

Best Wishes - John Tilley

[Jæja]

ThirdFogie has put togther useful material for a 3-4 kyu player, I don't think you can use AI to build this generic material for a SDK or DDK Player, but you could write some code to run on top of AI to look for certain basic mistakes - eg playing atari unnecessarily, not playing kikashi, playing too close to strength, leaving weak groups about, making vacant triangles etc etc

You could then run an AI analysis of a game, look at the big % swings and then see if any of these are due to these basic mistakes.

Thanks for your reply, John.

If we can automatically categorize a mistake, the entire process can in theory be automated.

1. Identify the swings
2. Label the mistake as empty triangle, missed kikashi, etc.

We can use a bot for the first point, but no system exists for the second. Humans could manually annotate the mistakes, but it's probably hard to find volunteers. However, if many examples would be gathered, we could even automate this step with machine learning! I'm pretty sure we'll have something like this in the near future

Technical (you can skip this ): Perhaps the representations produced by the value and policy networks can be concatenated to produce a feature vector on which we can build a model to classify the mistakes?

[xela]

A few years ago, my grade in BGA tournaments fell from 3kyu to 4kyu. Wishing
to avoid a further fall, I too decided to make a record of my most ridiculous
mistakes. I put them into a PDF document which can be loaded on to a tablet
computer for review before playing.

I'm really enjoying the titles and the overall style of the presentation! Where can I buy a copy of your book?

[Jæja]

Where can I buy a copy of your book?

I'm in too!