Spaced Repetition Software for Tsumego

[Boidhre]

One obvious problem is that you could declare a problem overlearned when you still get it wrong more often than not. What if you have a success or two and then fail? In that case I added 2 to the number of successes needed. One thought is that if you already have a success, count a failure as double. So if you fail three times and then succeed twice, so that F = 4 and S = 2, and then you fail again, add 2 to F so that F = 6 and S = 2, and you need two straight successes for overlearning.

In a simple SRS system what you'd do in this kind of situation is reset the frequency back to as if you'd gotten it wrong first time because in essence you've forgotten it. If you then afterwards start getting right a lot again because of previous memorisation, it'll very rapidly go back to a long frequency between repeats so long as you're not getting it wrong and finding it easy to solve. You get overlearning but equally you get long spacing between seeing a problem so long as you keep getting it right. The key though is the grading of difficulty after you get something right. It makes a difference when overlearning whether something is easy or hard to solve, I think.

[Bill Spight]

One obvious problem is that you could declare a problem overlearned when you still get it wrong more often than not. What if you have a success or two and then fail? In that case I added 2 to the number of successes needed. One thought is that if you already have a success, count a failure as double. So if you fail three times and then succeed twice, so that F = 4 and S = 2, and then you fail again, add 2 to F so that F = 6 and S = 2, and you need two straight successes for overlearning.

In a simple SRS system what you'd do in this kind of situation is reset the frequency back to as if you'd gotten it wrong first time because in essence you've forgotten it.

But then getting it right the next time would satisfy the overlearning criterion and it would be dropped.

If you then afterwards start getting right a lot again because of previous memorisation, it'll very rapidly go back to a long frequency between repeats so long as you're not getting it wrong and finding it easy to solve. You get overlearning but equally you get long spacing between seeing a problem so long as you keep getting it right. The key though is the grading of difficulty after you get something right. It makes a difference when overlearning whether something is easy or hard to solve, I think.

One thought is to have the odds of a problem being presented proportional to (S+1)/(S+F+1). Harder problems (those with more failures) would have a lower probability of being presented. New problems would have higher probability of being presented. (OC, you might want a minimum delay.) You want problems to be easy, but not too easy.

[Boidhre]

One thought is to have the odds of a problem being presented proportional to (S+1)/(S+F+1). Harder problems (those with more failures) would have a lower probability of being presented. New problems would have higher probability of being presented. (OC, you might want a minimum delay.) You want problems to be easy, but not too easy.

That's interesting. How I'm doing it at the moment is manually breaking problems up into batches of easy problems and hard problems and allocating much more time (when timeboxing) and new cards introduced to easy problems than hard problems and varying my "workload" this way (so my current system is 15 minutes a session, with as many or few sessions as I feel like per day, and X number of new cards to be introduced each day to the system). I like the idea of a proportional system though, especially if you could choose settings for how much bias there would be towards easier or harder problems depending on your learning methodology.

Edit: I do realise this is completely overthinking and overcomplicating the issue as a beginner! I'm just fascinated by learning methodologies and systems.

[Mike Novack]

I am questioning some of the premises in this discussion. I think we need to step back just a bit and discuss those.

What is an "easy" problem? What is a "hard" problem? How do we determine the difference?

There seems to be dissatisfaction with the rather simple approach of going by results. If you keep answering a particular problem correctly then it has become easy for you (and it should be presetnnted to you less frequently) If you continue to make mistakes answering it, then it has remained hard for you (and you should be presented with it more frequently). That is independent of any other a priori considerations of what should make a particular problem hard or easy for you.

Why should "it ought to be easy" or "it ought to be hard" be considered more valid/correct than the experienced results? Were I designing a "flash card" learning system with automated adjustment of the frequency of repetition (and of inserting new material) I would use the actual results and consider any preconceived notions of difficulty irrelevant

[Bill Spight]

I am questioning some of the premises in this discussion. I think we need to step back just a bit and discuss those.

What is an "easy" problem? What is a "hard" problem? How do we determine the difference?

There seems to be dissatisfaction with the rather simple approach of going by results. If you keep answering a particular problem correctly then it has become easy for you (and it should be presetnnted to you less frequently) If you continue to make mistakes answering it, then it has remained hard for you (and you should be presented with it more frequently). That is independent of any other a priori considerations of what should make a particular problem hard or easy for you.

Why should "it ought to be easy" or "it ought to be hard" be considered more valid/correct than the experienced results? Were I designing a "flash card" learning system with automated adjustment of the frequency of repetition (and of inserting new material) I would use the actual results and consider any preconceived notions of difficulty irrelevant

You will note, Mike, that my suggestion about presentation probabilities presents hard problems (those with a high proportion of failures) less often, easier problems more frequently, and the easiest problems (those that have been overlearned) least frequently (not at all ).

[Bill Spight]

Well, I did a quick online search for spaced repetition. It appears that it is used for things that are memorized, like foreign vocabulary. But there are several memory systems in the brain, and more than one is relevant to go problems. Skill learning is not the same as memorization.

When I was starting out I did not have much in the way of problem material. It was out there (I was living in Japan), but I did not buy any Japanese go books until I was SDK, and I bought them in department stores, which did not provide much of a selection. I might have only a couple of dozen tsumego problems that were not too easy (I could solve at a glance) or not too hard (I could spend several minutes without solving). The trouble was that if I reviewed a problem after a week, I had memorized the answer. So I waited at least a month before review, so that I had to work on many of the problems instead of just remembering them. I suppose that that is a kind of spaced repetition, although it was not in the service of memorization.

I do think that memorization is more important for go skill than I gave it credit for back then. Especially for elementary material. Why not memorize nakade shapes, for instance? Why not memorize the empty triangle? Why not memorize the snapback? Why not memorize the basic ladder? the basic net? the basic connection shapes?

When it comes to tsumego problems, however, I think that things get past the elementary fairly quickly.

$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . . . . . . .
$$ --------------

Click Here To Show Diagram Code

[go]$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . . . . . . .
$$ --------------[/go]

Experienced players will recognize that Black to play can make ko. I had no qualms about showing it to Boidhre in a review, even though I doubt if he could solve it as a problem. If he sees it as a problem, he will probably recognize it. My bad.

Is it useful in a game to know this position? You bet. Not that it comes up so much, but you may play to avoid it. It is lurking in many games. Would I make it as a flash card for beginners? No. For SDKs? I don't know.

$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . . . X . . .
$$ --------------

Click Here To Show Diagram Code

[go]$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . . . X . . .
$$ --------------[/go]

How about this position? White to play can live. This is a better flash card for beginners, I think. It helps them see the vital point.

$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . . O X . . .
$$ --------------

Click Here To Show Diagram Code

[go]$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . . O X . . .
$$ --------------[/go]

I think that this would be a good flash card for beginners. Live or die.

$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . O . X . . .
$$ --------------

Click Here To Show Diagram Code

[go]$$ Flash card?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | . X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | . . O O X . .
$$ | . O . X . . .
$$ --------------[/go]

How about this one? Not a beginners' flash card, I think. And maybe not at all, because it is an uncommon position. Still, it is not a bad tsumego, combining the eye stealing tesuji with shortage of liberties. It arises when White makes a bad play in the previous position two diagrams ago. (And that arose from a mistake by Black in the original position. ) It is something that I have never seen, because I saw the correct play for White right off the bat, years ago. But it is something that I could have learned by trying to solve the problem. Why it is a bad move is harder to understand than why the correct move is right. You can learn something from problems, even when you do not solve them.

Let's go back to learning the snapback. The prototypical snapback would make a good flash card, IMO. But learning that specific shape does not mean that much, even though it comes up or threatens to come up fairly often. The snapback is more abstract. It is the sacrifice of a single stone to reduce the dame of an opposing string to one, and then capture. (There is a multi-stage snapback, as well. ) The abstract snapback cannot be put on a flash card of a go position, but it is easy to learn from examples. You get spaced repetition of the flashback simply by playing go. It turns up, at least as a threat, in game after game. It also turns up as an element in problem after problem. Do we need to have a spaced repetition schedule for it?

[Boidhre]

I am questioning some of the premises in this discussion. I think we need to step back just a bit and discuss those.

What is an "easy" problem? What is a "hard" problem? How do we determine the difference?

There seems to be dissatisfaction with the rather simple approach of going by results. If you keep answering a particular problem correctly then it has become easy for you (and it should be presetnnted to you less frequently) If you continue to make mistakes answering it, then it has remained hard for you (and you should be presented with it more frequently). That is independent of any other a priori considerations of what should make a particular problem hard or easy for you.

Why should "it ought to be easy" or "it ought to be hard" be considered more valid/correct than the experienced results? Were I designing a "flash card" learning system with automated adjustment of the frequency of repetition (and of inserting new material) I would use the actual results and consider any preconceived notions of difficulty irrelevant

Well there's no reason one can't use a system like Anki that way if one chooses to, or to focus on a particular level of difficulty for you. The idea is, say solving a problem is trivial for you, it gets a long gap before being shown next, it is still feels trivial next time it gets a longer gap again and so on so long as it keeps feeling trivial for you. If it starts feeling hard it starts getting shown more often and if you get it wrong even once it resets back to the beginning frequency again since it clearly isn't easy anymore. I agree that there are philosophical issues with whether someone can mark something as easy or hard to solve in general but there is a feeling of a problem feeling trivial or not. If you've overlearned it beyond a certain point so that you can nearly instantly solve it and (in my opinion) it's time to see that problem less often relative to other problems in the set and other problems more often so long as the problem remains this easy for you (assuming of course one is not aiming to do as many very easy problems as possible).

As in, at the start, my approach is simply to mark problems wrong or correct. It's only on my third or more time getting a problem right first time that I might start mark it as easier than that based on the subjective feeling of difficulty. I do the same with language learning.


Again though, as a beginner, I'm loath to argue strongly here as I may be missing some fundamental point (which I'd be glad to have pointed out to me) and my opinions are preliminary at best.

[Boidhre]

Let's go back to learning the snapback. The prototypical snapback would make a good flash card, IMO. But learning that specific shape does not mean that much, even though it comes up or threatens to come up fairly often. The snapback is more abstract. It is the sacrifice of a single stone to reduce the dame of an opposing string by one. (There is a multi-stage snapback, as well. ) The abstract snapback cannot be put on a flash card of a go position, but it is easy to learn from examples. You get spaced repetition of the flashback simply by playing go. It turns up, at least as a threat, in game after game. It also turns up as an element in problem after problem. Do we need to have a spaced repetition schedule for it?

Well, the idea of spaced repetition is that it works with the brain's ability to learn something through repetition but is more efficient for memorisation than repeating every element every day or at the same frequency. It allows more material to be memorised more quickly than a simpler system of fixed gap repetition for languages. I don't know whether it'd be as useful in go as I'm very new, thus the thread asking if people had any luck with the method. Oh for a sample study with groups of people doing the same problem set using different methods (of course this wouldn't address the question of whether we do tsumego in order to solve them correctly first time or for some other reason...).

At the moment, I am nearly always doing simple Life and Death problems. I think SRS is very amenable to this since what it comes down to is memorisation of certain shapes. With complicated examples, I've no idea. Sorry, I can't develop my ideas more, minding a very active toddler at the moment.

[Bill Spight]

Well, the idea of spaced repetition is that it works with the brain's ability to learn something through repetition but is more efficient for memorisation than repeating every element every day or at the same frequency. It allows more material to be memorised more quickly than a simpler system of fixed gap repetition for languages. I don't know whether it'd be as useful in go as I'm very new, thus the thread asking if people had any luck with the method. Oh for a sample study with groups of people doing the same problem set using different methods (of course this wouldn't address the question of whether we do tsumego in order to solve them correctly first time or for some other reason...).

At the moment, I am nearly always doing simple Life and Death problems. I think SRS is very amenable to this since what it comes down to is memorisation of certain shapes. With complicated examples, I've no idea. Sorry, I can't develop my ideas more, minding a very active toddler at the moment.

As I indicated, I used spaced repetition when I started out, not so much to aid memorization as to defeat it. But I do think that memorization, particularly at the elementary level, is useful.

The results of some search:

This site provides a good review: http://www.gwern.net/Spaced%20repetition

"Skills like gymnastics and music performance raise an important point about the testing effect and spaced repetition: they are for the maintenance of memories or skills, they do not increase it beyond what was already learned."

A very important point.

This site by Dr. Wozniak, a spaced repetition expert, gives "20 rules of formulating knowledge in learning." http://www.supermemo.com/articles/20rules.htm

Rule 1. "Do not learn if you do not understand."

Rule 2. "Learn before you memorize."

IMO, simply making flash cards (or the computerized equivalent) out of go problems is not such a good idea. It does not follow Wozniak's rules. I suspect that making flash cards based upon go problems is a good idea, and that the main value comes from making them. For instance,

$$ Dead or alive?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | X X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | X O O O X . .
$$ | . O . X . . .
$$ --------------

Click Here To Show Diagram Code

[go]$$ Dead or alive?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | X X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | X O O O X . .
$$ | . O . X . . .
$$ --------------[/go]

I think that this would make a good flash card based upon this problem, but devising the card would be an important part of the learning.

Edit: In fact, I would suggest making status flash cards from tsumego problems. I. e., zero move tsumego.

[Mike Novack]

I am getting confused by this discussion.

I thought it began with a question. A question about the existence of non-existence of software that presented a set of problems with the following properties (my interpretation)

a) The problems reapear in different orientatons and colr with a frequency dependent on the history of getting them correct or not.
b) As the number of problems being presented frequently becomes low (many currently in the set have been learned) new problems are introduced.

And the discussion then diverged into whether "learned" should be based upon some subjective opinion or as I suggest be just a matter of the statistics of getting it right or wrong.

I said I thought such software currently exists. Yes of course you can use old fashioned flashcards but a virtual flashcard deck managed by the computer is what I thought was meant.

[Boidhre]

I am getting confused by this discussion.

I thought it began with a question. A question about the existence of non-existence of software that presented a set of problems with the following properties (my interpretation)

a) The problems reapear in different orientatons and colr with a frequency dependent on the history of getting them correct or not.
b) As the number of problems being presented frequently becomes low (many currently in the set have been learned) new problems are introduced.

And the discussion then diverged into whether "learned" should be based upon some subjective opinion or as I suggest be just a matter of the statistics of getting it right or wrong.

I said I thought such software currently exists. Yes of course you can use old fashioned flashcards but a virtual flashcard deck managed by the computer is what I thought was meant.

a) is correct, b) is not, new problems are introduced at a fixed rate per day. At least with the implementations that I've seen anyway. I'm sorry if I was unclear about this. Such a system could be used with either traditional flashcards or virtual ones. I prefer virtual for convenience but it's relevant to people who would prefer to do it with books with numbered problems or traditional flashcards.

Yes, the conversation has diverged into a philosophical one/practical one on the best way to learn and/or what the use of tsumego are. I apologise for dragging it off-topic, I find this interesting and relevant but I can see how it might not be to someone.

[Bill Spight]

I said I thought such software currently exists. Yes of course you can use old fashioned flashcards but a virtual flashcard deck managed by the computer is what I thought was meant.

Just speaking for myself, I am using the term, flashcard, in a generalized sense, just as I say, capture that stone, instead of saying, click on the display so that the software will refresh the image of the board so that that point will appear as an empty intersection.

[Ortho]

I used spaced repetition for language learning and it worked very well, and have dabbled with it in Go, but I am not convinced it's useful here in the way we're talking about.

In language there is one word for "chair", there are set forms for putting things together, etc. Mostly you just need to remember what they are, but not necessarily how to reason from them.

I agree that Go has some things like this, but mostly you have a set of principles or experiences or pattern recognitions and the challenge is to figure out which ones apply in the current context.

When I SRS Go problems there seem to just simply be too many cues based on the problem itself. I can't actually remember ever saying to myself "oh, this is like that problem I did" in a game and making a better move than I would've done.

I think maybe SRS would be useful in blindly memorizing joseki sequences, for example, but I am not sure that's a good idea anyway.

But I do think there is somewhere that memory plays a huge part, and that is in remembering the principles themselves. I don't usually lose games currently because I didn't know what to do---it's usually that I know the principle but didn't apply it. I can see an advantage in using SRS to remember things that you just have to remember, the mental checklist itself. But I'm not actually sure that the checklist is actually long enough that you need spaced repetition to remember it, especially if you're doing good game reviews after--the fact that you blew it again by not defending that cutting point just naturally turns up right when you have forgotten it, and even Anki couldn't remind you at a better time then when you're still smarting from messing it up again.

Anyway, I just make a list of the problems I fail or have trouble passing and look at them again a week later when I've hopefully forgotten them. I cross problems off the list when I think the problem is boring or when I immediately remember what the answer is.

[Boidhre]

As I indicated, I used spaced repetition when I started out, not so much to aid memorization as to defeat it. But I do think that memorization, particularly at the elementary level, is useful.

The results of some search:

This site provides a good review: http://www.gwern.net/Spaced%20repetition

"Skills like gymnastics and music performance raise an important point about the testing effect and spaced repetition: they are for the maintenance of memories or skills, they do not increase it beyond what was already learned."

A very important point.

Indeed and one I agree with. One should read out the answers initially and only solve instantly if one remembers the reading. Is this any different to building memories of patterns through continuous playing of games?

Do not learn what you don't understand. Doesn't that go against what the received wisdom for beginners is? i.e. don't worry about theory, just play games. I agree that blindly memorising the solution to one move problems and never reading them out would be a very bad idea, but doesn't reading them out before answering get around this issue?

This site by Dr. Wozniak, a spaced repetition expert, gives "20 rules of formulating knowledge in learning." http://www.supermemo.com/articles/20rules.htm

Rule 1. "Do not learn if you do not understand."

Rule 2. "Learn before you memorize."

IMO, simply making flash cards (or the computerized equivalent) out of go problems is not such a good idea. It does not follow Wozniak's rules. I suspect that making flash cards based upon go problems is a good idea, and that the main value comes from making them. For instance,

$$ Dead or alive?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | X X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | X O O O X . .
$$ | . O . X . . .
$$ --------------

Click Here To Show Diagram Code

[go]$$ Dead or alive?
$$ | . . . . . . .
$$ | . . . . . . .
$$ | X X X . X . .
$$ | . O X , . . .
$$ | O . O X X . .
$$ | X O O O X . .
$$ | . O . X . . .
$$ --------------[/go]

I think that this would make a good flash card based upon this problem, but devising the card would be an important part of the learning.

Edit: In fact, I would suggest making status flash cards from tsumego problems. I. e., zero move tsumego.

Hmm, dead I think since even if white wins the ko they only have one eye and white cannot do better than a ko here I think. I could be wrong though.

Do not learn what you do not understand. Does this not go against the received wisdom for beginners? i.e. do not worry about theory and just play lots of games?

[Boidhre]

I used spaced repetition for language learning and it worked very well, and have dabbled with it in Go, but I am not convinced it's useful here in the way we're talking about.

In language there is one word for "chair", there are set forms for putting things together, etc. Mostly you just need to remember what they are, but not necessarily how to reason from them.

I agree that Go has some things like this, but mostly you have a set of principles or experiences or pattern recognitions and the challenge is to figure out which ones apply in the current context.

When I SRS Go problems there seem to just simply be too many cues based on the problem itself. I can't actually remember ever saying to myself "oh, this is like that problem I did" in a game and making a better move than I would've done.

I think maybe SRS would be useful in blindly memorizing joseki sequences, for example, but I am not sure that's a good idea anyway.

But I do think there is somewhere that memory plays a huge part, and that is in remembering the principles themselves. I don't usually lose games currently because I didn't know what to do---it's usually that I know the principle but didn't apply it. I can see an advantage in using SRS to remember things that you just have to remember, the mental checklist itself. But I'm not actually sure that the checklist is actually long enough that you need spaced repetition to remember it, especially if you're doing good game reviews after--the fact that you blew it again by not defending that cutting point just naturally turns up right when you have forgotten it, and even Anki couldn't remind you at a better time then when you're still smarting from messing it up again.

Anyway, I just make a list of the problems I fail or have trouble passing and look at them again a week later when I've hopefully forgotten them. I cross problems off the list when I think the problem is boring or when I immediately remember what the answer is.

Hmm, see I it as useful for something like one move life and death problems where you just have to find the vital point. You're not trying to memorise it per se (thus you *want* long gaps between problems you solve instantly) but trying to get a lot of practice reading these kinds of things out. I've honestly found this kind of repetition (not SRS) very useful in games for quickly killing or living within certain eye shapes.