q30 wrote:
The results (with pondering - without pondering):
MoGo 3 - 1;
Pachi 3 - 1;
Ray 3 - 1;
Leela 3 - 1;
in all 12 - 4 (details).
I don't know, what about quantitatively results (in ELO), but definitely there is qualitative effect, and in sparrings of equivalent strength Go engines the same with pondering may pass in rating engine without pondering.
One more thing about this test: If you play with one engine against itself the ponder hits are close to 100%, That means the pondering side will benefit a lot more than against other engines.
I did also some testing:
In my League A there are two engines that do not ponder: Hiratuka and DarkGo. DarkGo plays instandly, so it doesn't matter if the opponent is pondering because he don't get any time to ponder. So the only Engine that may be affected is Hiratuka, so I testet Hiratuka against Pachi in 30min games with ponder on and ponder off.
First Round:
Ponder Off: Pachi vs. Hiratuka 5:11
Ponder On: Pachi vs. Hiratuka 8:8
This could indeed be a hint that Pachi becomes significantly stronger with pondering. Because I don't believe this I did the same match again with the same conditions. Now I got:
Ponder Off: Pachi vs. Hiratuka 6:10
Ponder On: Pachi vs. Hiratuka 2:14
That is exactly what I expected: The statistical fluctuation when playing matches between engines with similar strength is very high, just like rolling a dice. The difference between pondering and not pondering is simply not meassurable with such a small amount of games.