Webcam + Go

quantumf · **#41**

GregA wrote:

[*] Install linux and develop in there. It sounds like that is especially difficult with windows 8 - probably no easier than the above.

That's what I ended up doing. Wrestled with Visual Studio and eventually gave up. Admittedly my Windows box needs a reinstall, too many versions of Visual Studio have been installed, and it's virtually unusable now, at least from the command line.

I used VirtualBox, using Windows as the host, which works fine. I'm not aware of any issues with Windows 8 as the host.

GregA wrote:

EDIT: Update... I hacked up a quick pcf.py that is a drop-in replacement for pcf.c (fortunately pfc.c was still very close to the original python code). With that, and installing a few things (I installed the win32 python 2.7, and also the python 2.7 versions of pygame, PIL, and numpy - all compiled for win32) I was able to run the test image through and it worked! That's all for tonight, but I'll see about digging in to the meat of the code another night.

Did you upload it to github? If not, can you share it some way?

Rémi · **#42**

GregA wrote:

I see 3 paths to getting it going and none seem particular easy or fun:

It may be too late now, but just in case: cygwin is a really good linux-like environment for Windows.

Rémi

GregA · **#43**

quantumf wrote:

Did you upload it to github? If not, can you share it some way?

Python is notoriously picky about spacing, so you might need to fix up all the indenting, but this might work:

Code:

"""Unoptimized pcf module - rewritten to avoid having to compile pcf.c
"""

import Image
import math

def combine(im_bg, im_fg):
   sum = 0
   area = 0
   for i in range(0,len(im_bg)-1):
      if im_fg[i] != chr(0):
         sum += ord(im_bg[i])
         area += 1
   return float(sum) / float(area)
   
def edge((x, y), image):
   n_image = (len(image)) * [chr(0)]
   for i in range(0,2 * x - 1):
      n_image[i] = chr(0)
      n_image[(y - 2) * x + i] = chr(0)
   for i in range(0,y - 1):
      n_image[x * i] = chr(0)
      n_image[x * i + 1] = chr(0)
      n_image[x * i + x - 2] = chr(0)
      n_image[x * i + x - 1] = chr(0)
   for i in range(2,(x-2)-1):
      for j in range(2,(y-2)-1):
         sum = (ord(image[x * j + i - 2]) + ord(image[x * j + i - 1]) + ord(image[x * j + i + 1]) + ord(image[x * j + i + 2]) + 
            ord(image[x * (j - 2) + i - 2]) + ord(image[x * (j - 2) + i - 1]) + ord(image[x * (j - 2) + i]) + 
            ord(image[x * (j - 2) + i + 1]) + ord(image[x * (j - 2) + i + 2]) + 
            ord(image[x * (j - 1) + i - 2]) + ord(image[x * (j - 1) + i - 1]) + ord(image[x * (j - 1) + i]) + 
            ord(image[x * (j - 1) + i + 1]) + ord(image[x * (j - 1) + i + 2]) +
            ord(image[x * (j + 2) + i - 2]) + ord(image[x * (j + 2) + i - 1]) + ord(image[x * (j + 2) + i]) + 
            ord(image[x * (j + 2) + i + 1]) + ord(image[x * (j + 2) + i + 2]) +
            ord(image[x * (j + 1) + i - 2]) + ord(image[x * (j + 1) + i - 1]) + ord(image[x * (j + 1) + i]) + 
            ord(image[x * (j + 1) + i + 1]) + ord(image[x * (j + 1) + i + 2]) 
            - (24 * ord(image[x * j + i])))
         if sum < 0:
            sum = 0
         if sum > 255:
            sum = 255
         n_image[x * j + i] = chr(sum)
   return "".join(n_image)
   
def hough((x,y), image, init_angle, dt):
    matrix = len(image) * [0]
    for i in range(0,x*y-1):
      matrix[i] = 0
    d = x/2
    for i in range(0,x-1):
        print i
        b = (i - x / 2)
        for j in range(0,y-1):
            c = (j - y / 2)
            if (image[j * x + i]!=chr(0)):
                for a in range(0,y-1):
                    th = a*dt + init_angle
                    distance = b * math.sin(th) - c * math.cos(th) + d
                    column = int(distance+0.5)
                    if ((0 <= column) and (column < x)):
                        matrix[a * x + column] += 1
   n_image = len(image) * [chr(0)]
   minimum = matrix[0]
   maximum = matrix[0]
    for i in range(1,x*y-1):
        if (matrix[i] < minimum):
            minimum = matrix[i]
        if (matrix[i] > maximum): 
            maximum = matrix[i]
    maximum = maximum - minimum + 1
    for i in range(0,x*y-1):
        n_image[i] = chr(int((float(matrix[i] - minimum) / float(maximum)) * 256.0))
    return "".join(n_image)

And by the way, I've been thinking about how machine learning might be used to improve grid finding. I tried out some random photos from the web and the line finding worked great, but the grid was often pretty far off - my thought is clever algorithms can only get your so far, but machine learning with a lot of data can go farther. The big question first question is always how do you get a lot of training data. Here's a plan I'm going to try out when I get a chance:

Get an SGF file of a game and print out the move order
have a helper place the stones one by one in order
for each stone, first take a very clear top-down photo with the board nearly filling the frame
then take 9 other photos from different perspectives, within 45 degrees of pitch and yaw
number the photos something like Board-XXX-YYY-Z (X is SGF #, Y is move #, Z is photo # for that move)
Then you can automate a process - run existing algorithm to find lines, then do a search to find corners consistent with the known stone positions, save these off
verify corner positions: run through all images and their saved-off corner positions, display zoomed in on the corners and iterate through all images - user hits Y to accept as good and N to reject as wrong

At the end of it, you'll have about 1000 sample images with known grid corners and known stone positions. Once you've done it a couple of times, you could probably gather 1000 images with 30 minutes of human effort. A few hours would give you 10,000 samples - and you'd have the lines found, the corners, and the stone positions, everything you need to train for both grid finding and stone recognizing.

Rémi · **#44**

GregA wrote:

And by the way, I've been thinking about how machine learning might be used to improve grid finding.

If you are interested, I posted my dataset to the computer-go mailing list:
https://groups.google.com/d/msg/compute ... Y7zQwVKtwJ
I agree that machine learning helps a lot for this kind of task. It helped me for kifu-snap.

Rémi

GregA · **#45**

GregA wrote:

At the end of it, you'll have about 1000 sample images with known grid corners and known stone positions. Once you've done it a couple of times, you could probably gather 1000 images with 30 minutes of human effort. A few hours would give you 10,000 samples - and you'd have the lines found, the corners, and the stone positions, everything you need to train for both grid finding and stone recognizing.

FYI, I took some time to do a something like the above. I ended up with 2400 images for a 240 move game from a book. Some suggestions for anyone who wants to give it a shot:

Set your camera's resolution low. I left mine at full-res, which probably added an extra second per image just for the camera to snap it. When processing with imago, the first thing it does is cut it down to 640x480, I think, so that is mostly wasted. I filled a 16G SD card, and just copying to my computer took a long time.
Keeping track of move number and number of images per move is tricky when going fast. I got confused a few times, so sometimes I got only 8 images per move and sometimes 15.
It's very helpful to start each 10 photo batch with a nice clear full-frame top-dowshot. Given the above, I found it very easy to open an explorer window with large icons, and I could easily select / move each batch of 10 photos to a separate move-NNN folder.
I misplayed some stones. A helper would have prevented that, I think. As is, it isn't really possible to get a single coherent sgf file for the whole game I captured, as there are a 2-3 times where I misplayed, and then a few turns later moved a stone or two. One fixup solution could be to just find those moves in my image set and redo them, but it would be nice to not have to worry about it.
using a tablet with an SGF viewer to click through the moves would probably have been a better choice than using a book.
It probably took me 3 hours of snapping photos. Taking into account some of the above, I could probably do it in 1 hour next time.

I'm about halfway through running imago through all the images. My first impressions are:

As expected, nearly empty boards get very nice hough transforms, nearly full boards get messy hough transforms. I think stone edge detection at an early stage will be necessary for highly accurate grid detection with nearly-full board.
Line detection could probably be improved - it seemed like some useful information in the hough transforms was lost. I bet an algorithm that knows a bit more about go board hough transforms and where the important information generally is would be able to tease out more useful information, even in messier end-game images.
that said, in nearly all cases, the outer detected lines give you a quadrilateral that fully encloses the grid - though it often includes the outer board edges and sometimes nearby table edges. I bet this could be useful as a rough area to do stone edge detection within, and as a sanity check for detected grids that are heavily skewed or drastically smaller.
There are a lot of errors in grid detection where it misjudges the grid spacing. It looks like perspective isn't being taken into account. If you look at the hough transforms, you can see one of the dotted lines starting with narrow gaps and the gaps growing as you move along. The end result is a half-size grid that is otherwise aligned to the board. This might be an opportunity for machine learning, using the hough images as input and grid spacing as output. Or maybe just some simple tweaks to the algorithm.
When a correct grid is found, it is highly accurate. I'll get some stats eventually, but aside from mistaking the board edges for grid boundaries, you rarely find a grid that would be hard to know is inaccurate since it's close, but is still wrong enough to cause problems down the pipeline.
Stone color detection needs work. Probably the two things that would help the most are using stone edge detection to improve classifying board vs stone, and shifting the grid vertically to account for perspective shift of stones in the back corners. Once you know the skewed grid, you can compute the perspective shift. You can then transform a vector that points up a few millimeters into pixel space to find a better offset for the stone centers. Also, this is a great candidate for machine learning. I have 2400 images, which is almost 1 million images of individual stones. Once my data is fully marked up, I would imagine doing the perspective warp and vertical shift for all the images, resulting in a whole bunch of small rectangular images with known white/black/board classification. As these images would be very small - roughly 25x25 pixels, this would be pretty straightforward to feed into a neural network. I figure the inputs would be 3x25x25 for the HSV components of the warped original image, plus 1x25x25 for the warped edge detect image. The output would be a three-choice classifier, white stone, black stone, board. Of all the components, this seems like it would be the most straightforward to apply machine learning to.

Anyway, once I get the data cleaned and marked up a bit, I'd be happy to share it, and even happier if someone were to take a stab at getting another 2000+ images. I don't have a good place to drop the ~1 Gig of data, though. If someone really plans on working on this, I'd be happy to burn a DVD and mail it. Also, if someone wants to try taking some photos, I'd be happy to give more detailed advice.

Fantasio · **#46**

pasky wrote:

Hi! A student of mine, Tomas Musil, has created open source Go board optical recognition software that seems pretty nicely usable. We have focused on completely automatic runs, so it automatically detects the board corners and then the stones on the board. You can find it at http://tomasm.cz/imago together with a lot of pictures, documentation and bachelor thesis describing the algorithms in detail.

Hi Pasky and all!
Musil's work is remarkable indeed. I and my fellow colleague, prof. Mario Corsolini, studied the thesis and found it of the greatest interest, both on the theoretical side and the practical one. BTW, Musil seems to believe that "we have not found any other work that we can meaningfully compare our results with"; but since november 2012 we have developed and distributed PhotoKifu, a program aimed to reconstruct whole Go games by means of a series of photograph. We're now releasing version 2.1 (a paper is also in progress) and are working on version 2.5, which will implement Open CV instead of the external (and slow) suite Image Magick. We're also working on VideoKifu, a program that will reconstruct a game from a live video feed; we could not develop VideoKifu before because of the slow Image Magick suite, but Open CV will allow that.
At the moment (version 2.1) we achieve on single pictures the same - very good - results Musil got, and take 1 to 4 seconds for picture depending on its size. But our program is optimized for whole games, for which we have 100% stones recognition, even in the presence of hands and whole arms between the camera and the goban (of course the stones we're looking for must still be visible), given the pictures are not too small (I estimate at least 1024 x 768). That requires about 1/10 of second for each picture plus image optimization either by Image Magick or Open CV.
Of course, we too are interested in EGC 2015. We hoped to attend the scientific conference at EGC 2013, but could not find reliable informations. Maybe Musil will now be able to contact EGC 2015's organizers and catch their interest: it would be a remarkable feat if we could both go and talk about (and, of course, demonstrate) two programs making use of different approaches. For the moment we're planning to do such a thing during the big international Pisa tournament at the beginning of March, but the EGC 2015 would be the ideal stage.
Let us know if you're interested, of course!

mr. Andrea Carta, BSc(IT)

Atreides · **#47**

Does anyone have any of the files linked in this thread? For me all the links are dead. I am starting a Go Club and I have all the equipment but I can't find an app that will work for me.

I want our players to have sgf of thier matches so they can study them and get better.

Thank you!

sybob · **#48**

Atreides wrote:

Does anyone have any of the files linked in this thread? For me all the links are dead. I am starting a Go Club and I have all the equipment but I can't find an app that will work for me.

I want our players to have sgf of thier matches so they can study them and get better.

Thank you!

Hello Atreides,

If it is for club purposes and probably beginners, maybe it is best if you do not use image recognition software, but instead use an app on smartphone or tablet to keep track of moves. There are several around, both Apple and Android.
At our club, if a game is tracked by one of the players, he/she gets more time (say 15 minutes extra).

For iPad, you can use Easygo (lite, free) or RealGoBoard (free) for example.
For Android, you can use apps like ElyGo (lite, free), BW-Go (free), Gobandroid (free).

Regards.

Fantasio · **#49**

Atreides wrote:

Does anyone have any of the files linked in this thread? For me all the links are dead. I am starting a Go Club and I have all the equipment but I can't find an app that will work for me.

If you want to try PhotoKifu, you can download it from http://www.oipaz.net/PhotoKifu.html.
If you try PhotoKifu and encounter problems, you can write me (or my friend OiPaz at http://www.oipaz.net/eMail.html).

Atreides · **#50**

Again if anyone else has any of the programs above which links are broken I would much appreciate it. Thank you.

Fantasio wrote:

Atreides wrote:

Does anyone have any of the files linked in this thread? For me all the links are dead. I am starting a Go Club and I have all the equipment but I can't find an app that will work for me.

If you want to try PhotoKifu, you can download it from http://www.oipaz.net/PhotoKifu.html.
If you try PhotoKifu and encounter problems, you can write me (or my friend OiPaz at http://www.oipaz.net/eMail.html).

Thank you. Yours is the only program I have been able to find that is complete and I can fashion into a sensible workflow.

I am having an error right now and will be PMing you the error log. Thank you

Fantasio · **#51**

Atreides wrote:

Thank you. Yours is the only program I have been able to find that is complete and I can fashion into a sensible workflow.

Thank you very much.

For everyone interested: we have put online a demonstration video of PhotoKifu's usage. It can be watched and downloaded from the PhotoKifu's homepage, at http://www.oipaz.net/PhotoKifu.html.

From the same page are also available seven complete demo datasets. In a few days we'll present a paper, with a full description of the algorithms employed, at the 2nd Science Conference of the EGC 2015, held in Liberec. The paper itself can be downloaded from https://github.com/pasky/iggsc2015proc/ ... solini.pdf.

Atreides · **#52**

Any chance you guys will move to video any time soon?

Rémi · **#53**

Atreides wrote:

Any chance you guys will move to video any time soon?

Fantasio · **#54**

Atreides wrote:

Any chance you guys will move to video any time soon?

In a few weeks we'll start working on VideoKifu. We released yesterday PhotoKifu 3.00, that heavily relies on OpenCV instead of the nice, but slow, Image Magick suite. We are now able to process each picture in about 1/10 of a second, pre-processing included.
Everything may be found at http://oipaz.net/PhotoKifu.html, including a new demo video, this time featuring the Crazy Manja-Guo Juan game, played this summer in Liberec during the EGC scientific conference. This was an important game from many points of view, as not only a professional player (Guo Juan) was involved, but a test was performed on the advantages of human-computer cooperation, with full success (the human-computer team performed beyond expectations for human alone).
This demo video is available at http://www.oipaz.net/REP/GuoJuan-CrazyManja.mp4. All the pictures from the game are at http://www.oipaz.net/REP/GuoJuan-CrazyManja.zip.

We'll probably release one more version of Photokifu (a 3.1); hopefully this summer we'll be able to demonstrate a beta version of VideoKifu, although it's not certain that a scientific conference will be held in St. Petersburg too.

quantumf · **#55**

Another very impressive webcam sgf application: https://github.com/CmdrDats/igoki

PANPAN · **#56**

Hello,

If you accept to share your sources, I would be interested to use them to continue developments.
Idea is different: it's to develop a hub management software where this video analysis can be used to check correct parts positioning before boarding. I can keep you updated on my future developments around this. This for my studies and certainly not to be commercially used.
Can you please answer me.

Thanks a lot in advance for your collaboration,
Raoul RICHARD

Fantasio · **#57**

Atreides wrote:

Any chance you guys will move to video any time soon?

We have now moved to video. Yesterday we released the first demo version of VideoKifu, that you can download from http://www.oipaz.net/VideoKifu.html. There are also a demo video and a paper we recently published in the "Journal of the Korean Society for Baduk Studies".
Everyone interested may either contact us here, or by means of the addresses provided on our websites.

Webcam + Go

Who is online