If your training set has 3% error and your test set has 43%, you're overfitting way too much : it doesn't work.

So you either need more data or an other network configuration

I doubt if I am underestimating the amount of training data required. I was quite surprised that training on only 40 games did so well.

Anyway, the question is required for what. To help is all that I require.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

The training is still ongoing (performed 98 iterations), but it showing signs of over-fitting already: 27.8% vs 39.7%

Still, it is better than before.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

There is always some difference between the training set and the test set, depending on amount of training (if you couldn't heavily overfit the training with infinite iterations that would mean a problem with the net or the data, esp. with small training sets - oc in practice you stop much earlier).

I'd be more worried about the test results - what does it learn, and how reliably it generalizes in the long run and against different (unseen) opponents? My guess would be simple statistics like average distance from opponents move, tenuki percentage, second and third line move frequency, number of ko captures etc. Which could appear to work in one case, but could also fail in another. Did you also have move number as input?

A possible idea :

Step 1 : Start by training a level x policy network on games of players with level x. This network is reusable.

Step 2 : Add another couple layers at the end and train the last couple layers to discriminate between player A and not player A (you can further train the whole network when your last layers are well trained)


By building on a network that already "know" something, it may take much less training data than by starting with a fresh network

Obviously, this is just an idea and testing is needed (and I'm not able to try this experiment before a couple weeks)

Well, this is out of my league in fact

I gave the previous run, it was stuck at the same error levels for the whole day. I had a few more try with different network size, but it's all the same, the control group hardly get past 39% error.

I guess that's it for me, I will upload the script I used to prepare the training data for other to try.

Just for fun: I noticed the SGF games I downloaded include the ELO rating of the bots, so I'm thinking trying to build again a training set with bots that are more or less at the same level (in a given ELO interval), and have try again to see if it makes a difference (I will probably download more archives from CGOS for that).

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Distinguishing between player A and player B should in general be easier than distinguishing between player A and not player A.

Training policy networks at different levels is an important first step in rating the level of a player's play in a specific game, something which is now done in chess.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

That's actually an important result.

Many thanks for your efforts.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.