is, I suppose, the newest move so far, but it was not an AlphaGo invention. What if its policy network had not been trained on human play, but from scratch? What would its first seven plays look like?This sounds good in theory, but you may be underestimating the number of possible positions on 19x19, and the extent of "drying up" when you try to fill a wholeboard NN with data.Uberdude wrote:Isn't the point of reinforcement learning that as you train the networks, you are essentially transferring skill that was initially derived from tree search into the weights of the neural network: e.g. AlphaGo started by playing the slide to 2-4 after approaching a 4-4 like humans did in the initial training data, but after millions of games it found that tended to lead to poor results so now the policy network doesn't much like that move.
).
looks like the right side to block on, but you never know. Maybe we can think of the 3-3 invasion as a probe. Anyway, - 
follow the traditional joseki.
- 
White has sente, but I do not like White's chances. AlphaGo apparently agrees, because it does not play . It plays else where. Even the great Go Seigen did not see that! I suppose that AlphaGo has pretty well killed as joseki. It still may be a situational move, OC. looks like just the right spot. It has room to make a base with "a" or "b", and if White plays at "b", there is room for another extension to "c", if need be. The wedge is not too close to Black's thickness. It feels just right. ). Black would not have much of an attack that way. Many strong players of yore, perhaps even into the 20th century, would have had few qualms about approaching from the top, despite a bit of overconcentration, anticipating something like 
- , securing the corner. After 
a White hane at "a" would be bothersome, so they would probably descend to "a" first, with sente, before approaching the wedge.(This is also in reply to moha)Bill Spight wrote: I may be wrong, but my impression is that neural networks generalize from what they are trained on, and so they can produce some new things from time to time.
Here's my understanding of how AlphaGo works, described in layman's terms - at least the Fan Hui version, of which the Nature paper was based:Bill Spight wrote: I may be wrong, but my impression is that neural networks generalize from what they are trained on, and so they can produce some new things from time to time.
The NN can only give answers based on the data it received. The point is, IMO this answer will not be "globally" correct (except for the opening), there is simply not enough data for this. It will contain similar generalizations, localizations, simplifications to human intuition. For local shapes, this can be quite accurate because a distilled, generalized view can emerge for them (though will still blunder from time to time without search), but for global strategy, effect of one part of the board on another, sente, attacking maneuvers etc. you need search. Even for local fights it can only give options to search on, and not correct answers (pro level). But you CAN search, if you have NN for pruning! This is the real innovation here.Baywa wrote:Now, in order to train such a network you don't have to feed it all possible 10^100 or so board positions. This is the whole point of NNs! Somehow (by magic, or better by variants of the gradient descent method) it learns from a far smaller number of training examples.
My understanding is anything that's not explicitly spelled out in their paper(s) -- ie. anything that's implemention dependent, or "user" adjustable -- is part of DM's "secret sauce". And that, plus the massive resource requirements (custom TPU's, other custom hardware, massive power supply, etc.) are why AG has a significant (?) lead from its nearest machines (DeepZen, etc.).How much weight to give to MTCS vs. the value network is not clear to me.
I recall seeing a weight of 0.5 mentioned somewhere.Kirby wrote:How much weight to give to MTCS vs. the value network is not clear to me.
The question whether a method is globally correct (that is will always give the best answer) is pretty theoretical and practically irrelevant. You can only prove that a move is not optimal by finding a better move. How do you find such a move if the machine beats you every time? Especially for opening moves this is very difficult to determine. From a practical standpoint the selflearning AlphaGo plays very good moves. But it is more than likely that AlphaGo still does not play the optimal move.moha wrote:The NN can only give answers based on the data it received. The point is, IMO this answer will not be "globally" correct (except for the opening), there is simply not enough data for this.Baywa wrote:Now, in order to train such a network you don't have to feed it all possible 10^100 or so board positions. This is the whole point of NNs! Somehow (by magic, or better by variants of the gradient descent method) it learns from a far smaller number of training examples.
Of course! But that's the point of the AlphaGo-architecture. However, by playing itself many many times and by learning from it it gets new intuition.It will contain similar generalizations, localizations, simplifications to human intuition.
Maybe just the opposite! Those local situations with crosscuts, shortages of liberty require heavy reading.For local shapes, this can be quite accurate
Sorry, I don't understand that. ...because a distilled, generalized view can emerge for them
Well of course you do. But what would you do without intuition? Searching for a needle in a haysteck...but for global strategy, effect of one part of the board on another, sente, attacking maneuvers etc. you need search.
Forget your notion of correctness! (see above) Even local losses in many cases turn out to be globally good. AlphaGo and skilled human players show that in many games. Of course that requires good reading but also good intuition to even consider.Even for local fights it can only give options to search on, and not correct answers (pro level).
Now, you're with me. Did we go in circles? Sorry, I have to stop here. That part with Lee Sedol's game 3 I have to read up.But you CAN search, if you have NN for pruning! This is the real innovation here.
I have no quarrel with the AlphaGo team. However, I have a certain disappointment with computer science papers in general. I have written a couple of papers on the mathematics of go, which got published in computers and games collections. When I made use of a computer program, I included it in an appendix. My programs were in Prolog, and I wrote them to be human readable.EdLee wrote:My understanding is anything that's not explicitly spelled out in their paper(s) -- ie. anything that's implemention dependent, or "user" adjustable -- is part of DM's "secret sauce".
Quite often computer science papers include pseudocode, which is also human readable. However, I often found that the pseudocode was not enough for me to verify the claims that papers made. I shrugged it off because I am not really a computer scientist. Recently, though, I have heard of some research where computer scientists other than the authors were also not able to verify the results in some papers. In at least some cases the authors stated that they had not, in fact, included the tweaks necessary to produce the results in their papers, and were not going to make them public. This practice seems to be quite common. Sorry, but irreproducible results do not science make, IMO. 
Yes, I have seen that, too.moha wrote:I recall seeing a weight of 0.5 mentioned somewhere.Kirby wrote:How much weight to give to MTCS vs. the value network is not clear to me.
As moha points out, I believe that they weight each equally. Since they both produce probabilities, I suppose that they use the geometric mean, but I don't know.Kirby wrote:Step 3.) Train a value network, not with sample data like in Step 1, but directly by playing games like in Step 2: At a random board position, predict who will win the game. Use the Policy network from Step 2 to play out the rest of the game, and see who won. Just like before, if the prediction was correct, add positive value to the weights; if prediction was wrong, add negative value.
Step 4.) Combine policy network, value network, and Monte Carlo Tree Search: Tree is constructed, starting at root board state. At each node in the tree, the policy network gives a prior probability that any of the given moves will be good (e.g. 62% chance I should play move X). Then you can traverse the tree to search for best outcome. Outcome is defined by a linear combination of the value as defined by the value network PLUS the outcome that would occur by doing monte carlo simulation from that point in the tree. How much weight to give to MTCS vs. the value network is not clear to me.
) The value network may produce play that is too "honest", particularly if you are a bit behind. Then you need to play for errors, and using an intermediate value may help to do that. Another possible advantage is what moha points out, that the MC probabilities are based upon the actual position at hand, not some aggregate of more or less similar positions. An interesting point. Where to draw the line between "fundamental pure research" and commerce. Toward the end of 2015, before AG was made public, the top non-AG engines were about 4-5 stones from pro (they lost to pros at 5 stones; I forget the name of the computer tourney; afterwards, exhibition matches between the top engine and pros). That was when many who didn't know about AG were still saying "not for at least another decade" to beat top pros. After DM published their paper(s), the other engines jumped to near pro level (MLily: DeepZen v. human pros, 1-1). AG's results were consistent and reproducible (AG-Master's 60-0, and AG-2017's 3-0 v. Mr. Lee Sedol), and like Coca-Cola's secret formula, they were reproducible by the proprietor, just not (yet) by others.In at least some cases the authors stated that they had not, in fact, included the tweaks necessary to produce the results in their papers, and were not going to make them public. This practice seems to be quite common. Sorry, but irreproducible results do not science make, IMO.
I think this is the essence of Alphago (& co).Bill Spight wrote:As I have stated elsewhere, the MC probabilities are those of semi-random player vs. semi-random player. which is not the situation at hand. Why, then, use the MC probabilities at all, since they are inherently inaccurate?