I recently bought both Zen and Crazy Stone and played 10 games(5 black/5 white, Japanese rules) each against them on my laptop. Difficulty levels were set to highest and time limits were disabled on both (running on a 2013 MacBook Pro/Win 7 Pro with bootcamp). They are both very strong for me and I read from somewhere that Zen beat Crazy Stone in the final of a computer go competition. But somehow I won 5 games(4black/1 white) against Zen but only 1(1 black) against Crazy Stone. And almost none of the games I lost against Crazy Stone were even close (while always close games against Zen). Is the commercial versions of Zen just easier than Crazy Stone? Does anyone know how strong they are (commercial version with highest settings)?

Zen would like to win by a little, at least on KGS.

There are program version and hardware issues involved.

When you read Zen beat Crazy Stone did you think that in either case the hardware used was as puny in power as in your laptop? In the case of Zen, in these competitions being used is a program version designed to be able to run on a small closely connected network of computers, each of which likely far more powerful than your laptop*. The commercial version you bought likely designed to run well on a single machine, perhaps able to make good use of multiple cores/threads, perhaps not.

MCTS evaluators are computationally intensive. Look up the "crunch power" of the CPU in your laptop where such benchmarks are published. Then look up some of the cpus on which these programs are running on in the competitions.



* Some of them definitely more powerful than even the most powerful laptops, but I'm not sure about all of them as there are some pretty powerful "portable workstations" out there. Understand, being compact and ntended to be able to operate for at least a certain amount of time on its battery, there are practical limitations to laptop CPU power (heat and power consumption).

I think you over-estimated the supercomputers and under-estimated laptops.
I don't think my laptop CPU is "puny". It's a i7 quad-core 2.8 GHz and my laptop is always plugged-in so battery-saving feature is also off. It is a "portable workstation". My day job is programming and doing computations on supercomputers and clusters(with at least 512 CPUs). And I'm very certain the clusters("a small closely connected network of computers" as you say) are only faster than my laptop because of their scale, efficiency-wise each CPU is only 50%-80% comparing to one core in my laptop (which has 4 cores, of course) because I usually estimate needed computational time on clusters by running a small-scale job on my laptop. There's also time limits in competition while I set no limits on time. During my games Crazy Stone used ~30 min for a move couple of times (avg.10-12 hours per game) which is above the limits of most tournaments anyway.
I'm just wondering how strong the commercial versions are with good hardware.

Why not try playing them against each other? That would be an interesting experiment to see if it's consistent that way too.

Have you use game analysis feature of the software? I'm interested in how well that works.

I haven't played them against each other, because I don't have two laptops with same hardware. Playing them both on one computer could be unfair to one of them (if one of them is a "better" resource hog).
Which "game analysis" feature do you mean? In Zen I used the one counting territories when reviewing games; it is pretty good with territories, estimating influences from a group towards outside is always tricky (even for pros, all those thick-or-thin debates), but at least you can get an idea from it. I never used the "find next move" feature in Zen. I haven't played around Crazy Stone enough to look for these features.

Look, I was in the cypher mines too. But as soon as you say "an i7 quad core" thinking it doesn't matter which ....... well maybe not point in me talking (for those who don't know, the most powerful i7's are several times more powerful than many of the i7's). That's why I was referring to "look up a standard crunch benchmark".

"Zen, running on a mini cluster of a dual Xeon E5-2690 v2@3 GHz 32 GB RAM, a dual Xeon X5680@3.5 GHz 8 GB RAM and a dual Xeon X5680@3.8 GHz 12 GB RAM computers connected via a GbE LAN. 44 cores total"

OK, there are i7's comparable to a single x5680 (but I'm not sure if any of the laptop i7's are, a lot of heat to dissipate). But no i7 comes close to a single E5-2690.

It's very interesting they are using an unevenly distributed cluster. You are right about modern i7s are comparable to x5680. My laptop uses i7-4980HQ which is about 10% faster than X5680 and about 50%-60% slower than a E5-2690 depending on E5-2690's version. But the most powerful i7 (extreme editions, not for laptops of course) do come very close (90% in most bench marks) to the fastest E5-2690. They are CPUs of same tech/time, with different focuses (i7 focuses on GPU integration and power saving while Xeons don't have multimedia needs but focuses on stability during continuous high load computing). In a short computation, i7s are as good as Xeons. We even built i7 clusters here for short computations(~1 week long) because i7s are much less expensive than Xeons.

CPU talks aside, I do think Zen 5 commercial version has a smaller number of simulations in the coding (comparing to Crazy Stone). The longest move it takes is just couple of minutes (without time limits), while Crazy Stone 2013 can easily take 10 min for a move (and 30 min several times). And yesterday I also found out Crazy Stone has this shortcut to force it to make a move when you grow impatient. lol

Even if the processing power is similar, if the version you're using isn't parallelized over the cores, it might not matter, right?

_________________
Tactics yes, Tact no...

Modern CPUs all have multiple cores and the version should be using multiple threads to calculate (otherwise it would be ancient programing, DOS?). Parallelization actually makes individual threads slightly less efficient (needs to add communications with a master thread running on a different computer, not just a different core in the same CPU), it only wins by scaling(you have to have 10+ cores to be faster if you want to parallelize). You should compare CPU time instead.
I don't know what the average time does these programs take for a move in tournaments though.

Whether parallelization costs very much depends very much on the relative size of the task being handed off to the overhead of coordinating the tasks. I rather suspect this one of those cases where the task (play out a game) is gigantic compared to the overhead. But being able to get this right a major factor of success with the MCTS evaluator. Just before the first MCTS programs one of the leaders in go playing programs was go++7. But Mick Reiss dropped out of the race when he failed to get his MCTS go++8 to use multiple cores efficiently because he lacked experience designing programs for multithreading (I was beta testing).

Go back to the original MCTS papers to understand "time" issues. Given unlimited time, the pure algorithm ultimately finds the strongest move (relative to alternative strategies) but by the rules of the game, time is limited. The practical programs use various pruning strategies optimized for finding the strongest move they can within the time constraint. That means (on given hardware) the pruning strategies best for "average of 20 seconds/move" might not be the same as for "average of 20 minutes/move". Assuming that the programs were not adjusting for time except the number of playouts (same pruning used) we could very well have program A beats program B at 20 seconds/move but program B beats program A at 20 minutes per move.

Stopping at a certain point? Not as crazy as it might seem if you consider statistics. We are trying to determine if move A is (actually) better than move B by sampling outcomes. As the sample size increases from very small to very large we can evaluate the probability of error (our sampling says A better than B but actually B better than A). Initially the probability is huge (sample size very small) but it decreases as the sample size increases. However eventually we reach a point of diminishing returns where the decrease in error becomes miniscule even with huge increases in sample size. Remember, the pruning has introduced error that is not going to be reduced by more sampling.

I agree with this. In reality these programs can only do limited amount of simulations per move (or CPU times per move), the actual time per move can be scaled according to number of CPUs reversely.


I agree with this also. I guess commercial version of Zen is cutting down number of simulations to achieve a compromise between performance and time/hardware limits. I'm surprised commercial version of Crazy Stone is not making such compromise(not as much as Zen). It doesn't seem a good decision to make a player press "I'm bored now. Play you move now!" button. I wonder how much compromises these commercial programs are making.

I'm not sure about the version of Zen you are using, but one common error I see on Chinese forums is to select "5 dan" as the setting rather than selecting a time. Even the 10 seconds Zen is much much stronger than the "5 dan" Zen. You can only select either the time or the strength, so make sure you have time selected.

This Thanks for pointing this out. Do you know if it's the same with crazy stone?
And my colleague just told me she mistakenly sent a computation to my laptop 3 weeks ago. I immediately killed her process and revoked her remote access permission, no wonder crazy stone was crazy slow.
I just played 1 game against Zen with 10 seconds setting, it's easier than '5 dan'. An easy win before 200 moves. After that '120 seconds' Zen took a full 60 seconds on first move and 90 seconds on second move , I'll play against that tomorrow to see if it is very strong.

One thing you might want to consider is the possibility that one of these programs (the commercial versions) might have had parameters adjusted to attempt to optimize for the expected CPU power of their customer's machines while the other using the same parameters as for a powerful machine.

Your i7-4980 is much more powerful than a vendor would expect customers to have. None of the machines used in the competitions (or the top bots on the server) are much more than 2-3 times as powerful. If the same program versions as the bots playing on KGS are using you shouldn't find them more than a stone weaker on your machine (if at all).

But as you just discovered, maybe need to check "what other processes running?"

I have never heard of anyone saying 10s is weaker than the 5 dan setting (even on really terrible laptops). Would you mind checking to make sure there are no resource hogs running in the background and that ONLY the time setting is selected? I wouldn't bother with the 120s without making sure that the 10s is stronger than 5 dan first.

BTW, just today CS won the first round in the UEC cup.
(Links go to Rémi Coulom’s posts on Google+)

_________________
“The only difference between me and a madman is that I’m not mad.” — Salvador Dali ★ Play a slooooow correspondence game with me on OGS?

Crazy Stone won the tourney
And then my go idol(Cho Chikun) defeated Crazy Stone in a 3-stone handicap game.

I have checked settings again. I rebooted my laptop and stopped most unnecessary processes in the background. I didn't mean "10s Zen" was significantly easier than "5 dan", maybe it's just me getting use to Zen's style and more relaxed during the games. I tried "5 dan" again and now I also felt it easier than I first played it. I even defeated '120s' dan during the weekend. When I first played Zen, I was behind most of the time and often won only on yose (I noticed Zen is weak on yose), now I usually gain advantage during mid games.
However, I still have poor win ratio(<50%) against crazy stone.

Thank you for the upload.
Now i know why they call it crazy stone.
Playing crazy moves time to time... i think i can beat that program in 3 stone myself.
I guess there are still many areas of improvment in Computer go.

_________________
"The more we think we know about
The greater the unknown"
Words by neil peart, music by geddy lee and alex lifeson