Hi everyone,

I would love to use a software application that allows me to store and study mistakes I made in my (online) games, for example using spaced repetition. Ideally it would work like this:

Analyzer module
- I select a folder holding my SGF files and the application will monitor it to check for new files.
- The application will invoke an AI to analyze the games and find mistakes. I can set what a mistake is using e.g. an X percent drop in winning rate.
- The application will turn each mistake into a kind of flash card. It knows who I was in the game because I entered a list of my online handles. Optionally I could turn my opponent's mistakes into flashcards as well.

Study module
- The main idea of spaced repetition is that "...newly introduced and more difficult flashcards are shown more frequently while older and less difficult flashcards are shown less frequently and to increase rate of learning" (Wikipedia).
- The application keeps track of these statistics: card last answered correctly/incorrectly Y cards ago and use it to determine which card to show next.
- Perhaps it could allow for tagging flash cards as well, using a list of common mistakes like this one (e.g. overplay, slow, etc.) I could then study a certain mistake I've made in the past.

Does something like this exist? If not, what do you think of this idea?

It's a cool idea, but I've not heard of anything like that.

One question is whether you want a manual review. Do you think you'll gain from studying mistakes where you know the machine's verdict, but don't know the reason? Or is pruning them better?

_________________
Occupy Babel!

The point of flashcards is to forget your bad habits and replace them by good moves, so it would be better to categorize the correct moves (connect, make good shape, find a big point, find an invasion point...) instead of categorizing the wrong move.

I tried to do something very similar to this a few years ago with Leela 11.

I am absolutely not a programmer, I just enjoy hacking around with personal projects. I use the flashcard app anki on my phone and on the web-app for learning other stuff so my project just involved making nice go flashcards for that.

I found a nice python package someone wrote for making nice images from sgf files (can't find it now but I could fish around if it would really be useful.) I wrote one basic script that took a file and a move number as input and output two images. The "front" image for the flashcard was the position to play with the words "[black/white] to play", the back image was the same position marked with all the moves that Leela considered within 1% or so of its top choice (you could also go with just one move, but I preferred to learn the choice of strong moves.) the script also edited a text file for importing the cards into anki.
I then made a second script that scanned through a game for falls in win rate over a threshold.

I didn't last very long with these flashcards personally but playing with the code was fun for me.
My problem with the flashcards for me was that most of the mistakes were in quite specific fighting situations. Once I'd read the fight out properly with hindsight it looked obvious to me and spaced repetition didn't really feel like it was necessary to learn those specific lessons. The challenge in go is often around abstraction of learning - taking these specific learning points and linking them up with novel positions. I don't think there was a learning advantage over just playing more games and reviewing the new novel positions.

I do think it's a very cool idea though. Maybe I just needed a gentle tweak to the system to help me get more from it?

I wrote software to generate problems out of specially annotated SGF trees; these could be local situations or whole games.

Annotating the trees is not an automated process; for my games I use AI and reviews from stronger players to add variations that I want to remember. Then the problems are generated and can be used with WGo.js; I extended the Tsumego class that comes with WGo.js.

There are some principles such as "the student should never play a bad move" and that if at some point the opponent has several plausible responses, multiple problems will be generated.

You can see the result on my site; for example, see http://gogamespace.com/training-module- ... ilter/game

Moves or situations can be annotated with tags as well; for these tags I generate problem collections that feature problems with that tag. See the menu at http://gogamespace.com/training-module/

It is not a spaced repetition system. If there was a way to use WGo.js on Anki flashcards, it would be good.

The software is not open-sourced yet because it is undocumented; the tags, markup and workflow really need some documentation.

So there is something that I believe goes in the right direction but unfortunately you cannot yet use it. I don't know when I'll get around to documenting and releasing it.

By the way, Hwang In-seong 8d (from the Yunguseng Dojang) gave me permission to use his lectures for this system; I have annotated and adapted a few of the lessons. You can see them at http://gogamespace.com/yunguseng-dojang-practice/

I assumed it would be "X to play" in a circumstance where you'd previously made a mistake. The answer would be the right move.

_________________
Occupy Babel!

Thanks for all your reactions. I'm getting the feeling this might be worth working on!

Correct! I think my remark about tagging might have confused things. The main idea is to learn better moves and the categories could be fun to study specific topics.


This definitely a good point! The AI review is just to make it generic and usable for anyone, even if you don't have somebody to review your mistake. You could perhaps use the AI to find possible mistakes and mark them as such if you like.

This sounds very interesting to me. I like how this could be a pragmatic, simple solution to test ideas. I might want to play with this around myself. Do you have any code you could share? If not, I'm already happy with the idea you shared, so thank you

I can understand you feelings about the outcome. However, I also agree that it might be more effective if it's tweaked a little bit more.

A large part of the open source community actually helps with documentation. Developers often don't find it interesting to do this, so other people take this out of their hands. Can we help you with this? I like what you did with this software, but the level of your problems is too high for me. However, if we can use your software as a basis, then we could make it easier for other to build problem collections. What do you think?

For flashcards, to prevent erroneous recognition, I always recommend adding a card with a small change, such as moving the position of one stone, that changes the answer. That will help you to pay attention to significant things on the card, and the exercise of producing those extra cards may help you to understand the positions better.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Nice! Great idea

Already there have been a number of great ideas by several of you. Please let me know if you have more suggestions and if you'd like to collaborate on making this project a reality. I don't have enough time by myself, but I'm sure it's doable if a few of us work on it.

A few years ago, my grade in BGA tournaments fell from 3kyu to 4kyu. Wishing
to avoid a further fall, I too decided to make a record of my most ridiculous
mistakes. I put them into a PDF document which can be loaded on to a tablet
computer for review before playing.

Jokey titles were used to ease recall. The first page of the contents list is
hidden below. It was made by converting the relevant PDF page into a .png file.
It is hoped that the result does not overburden the Lifein19x19 server.



In the contents list, "Call that a kikashi" is an allusion to "Call that a
knife" in Crocodile Dundee, and "Don't panic!" is an allusion to Corporal
Jones. (See https://www.youtube.com/watch?v=nR0lOtdvqyg for
a surfeit of examples.)

Here is a sample page preserving three stupid mistakes.



There would have been more samples at this point, but only 3 attachments are
allowed per post. A follow-up post will contain some more examples, if I'm
not banned for abuse first.

Did the catalogue work as intended? It did not prevent the feared drop to
5kyu, but it probably slowed the rate of decline.

Successes

I have stopped making some of the more stupid mistakes such as reading out that
a move is bad and playing it anyway. I have also learnt to "rein in the
killing frenzy and survey the board", though I am still terrible at identifying
the most important weakness to protect in such cases. Premature nakade appears
much less often and I now know my way around the tripod group.

Another benefit of the catalogue was that its construction left an organised
collection of challenging Go positions, and it was illuminating to work through
them with Leela Zero.

Failures

Learning the status of the notchers and corner groups recorded in the catalogue
never seems to help during play. More equanimity is needed. (Maybe flash
cards would indeed be better for training than my attempts at literature.)

There has been no progress with "Don't panic" and "Beware of a Glass Dagger",
The latter refers to loosely captured enemy stones, which catch me out again and
again.

Another problem is that new hideous mistakes have begun to appear. These are
largely caused by poor concentration. I will be 71 this week, so that is no
surprise. Here are two examples which have yet to make their way into the
catalogue.

1. Deciding to play on a certain point, then finding that the stone landed on
a nearby point - the real-board equivalent of a mis-click. That has
happened twice.

2. Embarking on a squeeze which should win the game, and then forgetting
what I was trying to do and switching to saving the stones which should have
been sacrificed.

Details About the Writing

I now can imagine myself as a Go author. That means finding answers to lots of
detailed puzzles, which are hidden below.


Finally, here is a grumble about the lack of a universal coordinate system for
points on a Go board. This appears in the catalogue as part of a section
reminding me how to amend and process the catalogue itself.



Note: edited immediately after posting to really hide the first attachment.

Here are two more sample pages from my catalogue of calamities.
I made them a bit smaller this time.

The first page shows a case where a glass dagger in the heart proved fatal
later in the game. It also shows a truly stupid response to a cross-cut.



The second page shows a failure to push White down to the third line, which I would
have done immediately if I had recognised the situation properly.



Whole-board situations like these might not be well served by flash cards, but I have
not tried that approach.

That looks truly amazing thirdfogie I really like the clear writing, jokey titles and consistent, clear layout. It really like this idea!

I'd like to start working on something similar for my own mistakes. Could you tell us something about your method of working? I'm interested in how much effort it is to go from an actual game to an example in your book. Do you record an SGF on something like a tablet and then convert it to a printable diagram? It looks like you might be using LaTeX with the igo package. Is that correct? I've exported a diagram to EPS using GOWrite2, which was easy and produces nicely looking diagrams.

ThirdFogie - Celebrating your 71st birthday is no excuse - Iwamoto Kaoru 9p defeated Takemiya 9p by resignation in the Oza preliminaries in Feb 1977 - two days short of his 75th birthday. Takemiya was 26 years old. I find that quite amazing. Please note that I celebrated my 72nd birthday last week.....

Jaeja - I think there is a limit to what you can do with AI analysis - maybe it is related to your ranking.

Once you get into the amateur dan ranks AI becomes more useful - I had a game of mine analysed some 50 years ago, when I was 1d - the feedback from Haruyama 9p was that I was not aggressive enough - there were several times when I was failed to play kikashi. Once you know that, you can use AI to spot such things. What I don't think you can do (yet) is to use AI to build such generic advice.

ThirdFogie has put togther useful material for a 3-4 kyu player, I don't think you can use AI to build this generic material for a SDK or DDK Player, but you could write some code to run on top of AI to look for certain basic mistakes - eg playing atari unnecessarily, not playing kikashi, playing too close to strength, leaving weak groups about, making vacant triangles etc etc

You could then run an AI analysis of a game, look at the big % swings and then see if any of these are due to these basic mistakes.

I have to say that after a gap of some 30 years in playing at go clubs, I found that SDK players were making the same mistakes that were being made in the 1970s..... The hardest thing is taking these mistakes seriously enough. If I had been paid £5 every time someone made a vacant triangle or played an unnecessary atari at the Winchester Club, I would have been quite well off!

Best Wishes - John Tilley

Thanks for your reply, John.

If we can automatically categorize a mistake, the entire process can in theory be automated.

1. Identify the swings
2. Label the mistake as empty triangle, missed kikashi, etc.

We can use a bot for the first point, but no system exists for the second. Humans could manually annotate the mistakes, but it's probably hard to find volunteers. However, if many examples would be gathered, we could even automate this step with machine learning! I'm pretty sure we'll have something like this in the near future

Technical (you can skip this ): Perhaps the representations produced by the value and policy networks can be concatenated to produce a feature vector on which we can build a model to classify the mistakes?

I'm really enjoying the titles and the overall style of the presentation! Where can I buy a copy of your book?

I'm in too!

I've already had a go at classifying mistakes in a very crude way. I think this can be greatly improved if someone has time to invest in it. Certainly the policy network values are very useful. Other ingredients would be to compare policy values from weaker and stronger nets (go to https://neuralnetgoproblems.com and click "about"...) and a shape recognition engine similar to Sabaki's auto-generated move names.

Then we just need a large and non-copyrighted collection of labelled examples to train the model on.

Jaeja: thanks for your kind words about the Key to All Mythologies^W^W^W^W Catalogue of Calamities. Apologies for the late reply: a birthday and a kitchen fire took priority.

Diagram production for the catalogue is largely automated, but you probably don't want to copy my methods. Here are the reasons.

1. I use Debian Linux: you probably do not.
2. If I was starting over, I would do it differently.
3. It all took much longer than expected: I started the catalogue in June 2017,
and it is still incomplete. Of course, I haven't been working on it full time.

The "Typesetting guidelines" section of the following article on the BGA website might be a better guide for you.

http://britgo.org/bgj/guidelines

Returning to my work-flow, instructions for working on the catalogue form two pages of that document.
They are hidden below.

First page.


Second page.


Section 7.1 of page 2 omitted to mention a C program I wrote to remove captured stones from starting diagrams, and to note stones played at a previously occupied point. The notes provide hints for annotating ko and other recaptures below the relevant diagram. I couldn't find a way to make sgf2dg do those jobs, but I may not have tried hard enough.

Finally, the next hidden section shows a simple BASH script used to prepare the .png files for these posts. I will reply to John Tilley separately.

I use Ubuntu Linux and LaTeX. You are not alone (if you admit Ubuntu as an acceptable substitute).


No, you tried the right amount. If you try hard enough, you can make LaTeX macros do anything, but a higher level language is probably a good idea in this case.

A few more thoughts about my Catalogue of Calamities.

Xela wrote:


Surely you jest. Anyway, it will never be published, except for occasional
excerpts on this forum.

John Tilley wrote.


I want to disagree with this slightly. To adapt Tolstoy, every weak player is weak
in his or her own way. At least to some extent.

Here is the justification. Since November 2019, I have been lucky enough to
find a regular opponent of similar strength: perhaps half to three-quarters
of a stone weaker. The 40 games played so far tend to fall into
a regular pattern: I get ahead in the opening, followed by chaotic fighting
where my opponent usually does better. If I haven't already resigned, I usually gain
many points during the last part of the game.

In several recent games, Leela Zero thinks my winning percentage has been up to 93% after
only 14 moves. The reason is obvious: the opponent does not know any joseki or many
principles of fuseki.

My relative weakness in fighting is probably down to the fact that he is much fitter than
I am, though of the same age.

The only obvious difference in the later parts of these games is that I have better
internalised the idea that it is often more efficient to puncture
the other player's territory than to defend one's own.

None of these relative weaknesses seem easily amenable to change.

The two of us have learnt some things from one another.

At first, I would often catch some of his stones in a net: he no longer lets that happen.

He no longer lets me get all the (supposed) double-sente points.

He has a tactic for breaking into my potential area by attaching to the side of one of
my stones. When I answer with a hane, he crosscuts and then forces a way
in with a series of threats to capture. I have slowly and painfully developed
a new personal proverb to cover these cases: To preserve an area, don't hane
but back off to thwart the cross-cut. All I need to do now is to apply it sensibly.

Here is an anecdote about John Tilley. The only time we have been in the same
room was at the old London Go Centre, so more than 40 years ago. Four or five
people were watching a game in its late stages. The bottom of the board looked
something like this. I don't recall how it came about.

Click Here To Show Diagram Code

[go]$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . X X . . . . . . . . . |
$$ | . . . O . . X X X O X X X . . , . . . |
$$ | . . . . . X X . . . . . . X . X . . . |
$$ | . . . . . . . X X O X X X . . . . . . |
$$ | . . . . . . . . . X . . . . . . . . . |
$$ ---------------------------------------[/go]

JT breezed into the room, took one glance at the board and immediately said "Ikken tobi
is never a bad move". It was clear that a superior intellect was at work.

Lastly, here are some plans for improving the Compendium of Catastrophes.
It should be more positive. In particular, the sections of pure despair
should somehow be expanded into helpful psychological advice. Knotwilg's
frustration equation will have a place of honour. I also want to plunder
Richard Hunter's excellent books. I only understand about the first 10% of
each book, so the plan is to distill the bits that I do understand into pithy
statements that will prompt my memory. Incidentally, this is one reason for
not publishing the whole Catalogue: it shamelessly incorporates other peoples' work.
This is OK for private study, but not for publication. (The extracts given
in this forum are mostly my own work, leavened with an occasional quotation.)

I have also been compiling material concerning the common situation shown
below.

The analysis which follows in the Catalogue is still a mess. For what it's worth, here is a shortened version.
1. Assume any forcing moves by the opponent have already been played before counting liberties.
2. If the white stones have fewer than 5 liberties, jumping ahead will result in at least
one white stone being captured if Black pushes in.
3. If the white stones have exactly 5 liberties, Black will be able to push through and
squeeze.
4. Above 5 liberties, much depends on how much space there is at the other end of the
white stones, whether that space has any value to White, and whether either side can grab it.

Any suggestions?

Edited on 10 March 2020 to remove reference to quarantine, caused by a misremembered post.