Moyo Go Studio DevBlog

I often watch documentaries when I am waiting for something to finish crunching. Here is one of the most interesting things I have watched in a few years, and I watch about a thousand documentaries a year. It is hard to follow for folks without some basis in electronics, but it is an excellent explanation of how electronics really work. Plus the best explanation of the core of Tesla’s work. Almost half an hour into the “lecture”, Tom mentions that the Tunguska event in fact was caused by Tesla. I believe him, as Tesla mentioned to some polar explorers to watch the sky that day, because he would give them a “sign”. The Tungushka event happened simultaneously with Tesla’s Wardenclyffe demonstration.

Later (around 36 minutes) he mentions Zero Point Energy, something the Dutch scientist Casimir (the Casimir effect) has proven to be real. What we learn in school about wave theory is only half of the picture. About an hour into the monologue, Bearden becomes too controversial even for me.

While I am making a name filter for the pattern results (example patterns, Fuseki and “Kombilo” patterns), I discovered that a lot of pro games have alternate spellings:

Pak Yeong-ch’an
Pak Cheong-sang

Is this one and the same person or not? If so, which spelling is preferrable? I am fixing this problem as we speak:

This is the socalled “McCune-Reischauer form”, a standard to spell Korean names. The best feature will be that deviating spellings will automatically be normalized, when entered, doing a SQL database search. So even if you don’t know the proper name, it will still work.

I found game records for Seo Nung-uk, Seo Neung-wook, Seo Nung-wook, So Nung-wook, Suh Neung-wook, Suh Nung-wook, Seo Neung-uk, Seo Neung Uk and Seo Nungwuk. In fact the preferred spelling for the name of this very same person is Seo Neung-uk.

A serious Go study tool can’t be without automatic normalization of player’s names, how would you otherwise be able to study (filter on) the style and statistics of a certain player?

Dariush has been shouting that it has a better pattern system than Moyo Go, so since I decided to start working on the program again, I had a look at Dariush. It is easy to make critical comments about the look & feel and functionality of that software, but that might be unfair, as I have no idea how long they have been working on it. The pattern system is what I was afraid of - and so I should be:

I took the first game in one of my SGF folders marked “Cyberkiwon 7d”. I have the folders a through z there, with numbered games in them. The first game was numbered 9, and the file is here. The strongest Cyberkiwon games are often played by pro’s, and statistical analysis shows hat most games in that category use pro Joseki.

Of that 126-move game, Dariush predicted 55 moves correctly, Moyo Go 54. You can try it yourself and let us know here if you found a mistake.

This (statistically insignificant) result does not daunt me. On the contrary. Darius obviously has a much less sophisticated pattern system than Moyo Go has (when I look at which info is obviously in its patterns and even the size of its pattern database). So there must be something very important the Dariush dudes have implemented, and Moyo Go didn’t implement that at all. So if I figure out, using some detective work, what that crucial difference is the programmers of Dariush “got”, and I didn’t get, then wham, Moyo Go will be the undisputed leader again.

I am excited because I already think I got it. I think I know what Dariush has, in its pattern system, that Moyo Go hasn’t. It is obvious, when replaying 9.sgf in both programs. (On a tangent: I think Moyo Go actually outperforms Dariush here, because where some moves are not at all considered by Dariush, Moyo Go actually classifies them as the second-best choice, then again I haven’t counted how often that happens with both programs). And Darius, contrary to Moyo Go, does not give statisticaly sound move estimations, but hey, as long as the moves are good :-)

Anyway. I think that Dariush (it seems obvious when you observe its pattern classifications) greatly emphasises locality. Computer Go researchers are divided over whether locality should be used in a move-suggestion system, but Erik van der Werff seems to use it to good effect. And now Dariush too, it seems. If this is really true, if locality plays a role in move prediction, then adding it to Moyo Go should give it a crushing ahead to Dariush.

I have once talked about locality in move prediction with Michael Reiss. We both thought it was “sinful”, bad, unscientific, etc. But first Erik, and now Dariush, hm..

You see, competition is good. One of the first things I’m gonna do is to add locality to Moyo Go’s pattern system, and test it.

I’ll be back.

Some customers are not very happy that I promised them free bugfix releases an am now reneging on that promise.
I’ll see what I can do. One of the main problems (the only really big problem, perhaps) is downwards compatibility with the databases. When I add fields to the databases, the bugfix release should somehow downgrade its available feature set. That is such a can of worms that I don’t want to go there. Not to mention the need to either let people export all their games and re-import them in order to rebuild the database to its new format, or to redistribute the Gigabyte of game databases for free.

The problem with the first alternative is that importing all 450 000 games would take over a week, on the average PC. The problem with the last alternative is how to get 2 GB of the latest databases to hundreds of people, without providing access to those tenthousand users of the free version. These huge databases really are a problem. In order for the program to have such an unique pattern capability (there are rumors of another program being “better” but I believe it when I see it, at least it has not half a million example games like Moyo Go has), the importing process is relatively slow at about a game a second. You can redistribute the SGF’s as 7zip files, but you can’t expect a 125-hour importing process to go well, especially not because the larger the database, the more RAM required for the importing process (up to 1 GB free RAM, and RAM only, not pages on disk).

Sometimes I hear complaints on how slow Moyo Go is. That always enrages me :-)

Because Moyo Go is not slow. On the contrary, it is incredibly fast. The tricks used to get it so fast are so complex, that at the moment I don’t remember how half of them work, and I have to reverse-engineer them from the (comments in the) source.

So whence those occasional claims of totally-unusable slowliness? They are few but they exist and I take them seriously.

First of all, by all means ensure that there is at least 350 MB free for the program. If you have a machine with 512 MB and you run nothing else but Moyo Go, it still doesn’t mean there is 350 MB free. What you should do is Start - Run and type msconfig. On the Startup tab are usually a large number of totally useless applications, starting up with Windows. They come with the soundcard installer, with Adobe Acrobat etc. get rid of that crap - they use RAM. If you don’t know what something does, Google it or see what happens when you disable it.

A way to see how much memory running processes use is CTRL + SHIFT + ESC, and go to the Processes tab. Then View - Select Columns and choose Memory (working set). Then you’ll notice (as I now do) that Firefox likes to claim 350 whopping megabytes at the moment, on my machine!

When you’ve eliminated memory-starvation problems, go on to another very important issue: Bad NVIDIA screen card drivers. They are the single most common reason for slowing down Moyo Go. Update that driver. I had half a dozen customers who had their slowdown problem disappear that way.

Now we come to the game-import process. Again, this should be fairly quick with a limited number of games. When it isn’t, it is most likely an antivirus scanner. Moyo Go creates 4096 temporary files before it imports games. A virus scan of so many files can add a minute or more, perhaps, to the import process. You can temporarily disable a virus scanner to see if it makes any difference.

I just realized that the game packs for sale for just a few dollars (Tygem, IGS, NNGS, Cyberkiwon, KGS) should also work on the now-discontinued free version!

If that means that those tenthousand people who downloaded the free version, will now each spend 5 bucks on some games, I’ll be rich (I’ll get about half).

Moyo Go currently does not have a way to export all its SGF games in one go. You can export them very easily one-by on though.

I decided not to built in limitations (SmartGo has to be re-started every ten games you export).

A few people asked me for a feature that exports all games, and I am reluctant to provide it, because exporting all games takes ages. The games are heavily compressed in a database, and exporting them would go with ten games a second on a fast PC. I can’t really see what the use of all those games is, if you already own Moyo Go Studio, with a queryable SQL database, a “Kombilo” pattern searcher, an SGF annotation program and all the rest.

Of course it is very interesting for me to know why one would want to use those games in another program, because I will react by adding the features of that program to Moyo Go!

Meanwhile, I am now exporting all games from a Moyo Go install to SGF, because I had thrown away all my SGF’s, because those 450 000 SGF’s occupied 1.75 GB. I could have kept them but as I mentioned before, in a bout of anger I got rid of everything. Using R-Studio to recover the SGF’s is not an option due to the sheer number of SGF’s, and the fact that they have non-descriptive names. I need to be able to know from which Go server they came from, and R-Studio will not give me their parent folders’ name, only files grouped by file extension. Using a modified Moyo Go build to export the games, this still takes a waking day on a very fast machine. I have not much to say about the exporting speed because I use a third-party SQL database source code licence for the game data for practical reasons: Writing a state-of-the-art SQL database system takes a few years.

The reason I need those raw SGF’s is that I want to add a feature to Moyo Go that neccessitates altering its database fields. I need to add a way to quickly (!) filter on player’s name. Not in the SQL database, but in the example (”learned”) pattern database. When moving over the board, you see patterns and the system spits out the games in which they occur with a speed of tens of thousands a second. What I need to add is a way to get only Go Seigen’s games, etc. But it should not make the retrieval speed slower. That’s really hard to do. I tried it twice already. Yesterday I concluded that the only way to do it is a little comp-sci trick. So I’ll have to build it and re-import the SGF’s.

But that trick changes a database format. In case of free updates, I would have to redistribute 1.5 GB worth of databases to customers. Not very practical. Having only one (major) update a year for not too high of a price solves this problem, at least for the pro games.

I finally found a webshop that let me upload 1.5 GB worth of files. BMT Micro has just about the best reputation, so I’m happy.

Moyo Go with just the pro games is significantly cheaper (especially in Euro’s) than before, but all game packs included, the software hits 94 USD, and if you want it sent to you by BMT Micro on a DVD, it’s 10 bucks extra. I’m done with free demo’s and asking prices that give a wrong impression of the software.

The webshop is here: http://www.moyogo.com/buy.htm

At the final checkout option (click the “checkout” button) there is an option for a physical CD/DVD by US mail:

I am interested in the download speed, if you buy something, let me know. The largest file is the main program, a whopping 308 MB. Still, on a typical 4 mbit ADSL line it should take no more than 15 minutes to download. You’ll have 31 days to download a file, so if it fails for some reason - no worries.

I think that you can only make really good software when you use it yourself. They call it “eating your own dogfood”.

Initially, I focused a lot on the technical aspects of the software, like superfast pattern recognition, and that those patterns were indeed relevant. I could do that with just programming skills, because it is possible to use harvested Go patterns to predict pro moves, thereby avoiding the need to become a Go player myself.

A long time ago (around 1988), my job was to design electrotechnical installations. We used a 2D CAD program by Rotring, and it sucked. So I borrowed ten thousand bucks and bought an Acorn Archimedes, a multisync monitor and an A3 plotter. Two years later I had a CAD system for the electrotechnical industry that was ripe for sale. Not because I was a CAD expert, but because after one year into its development, we dumped Rotring’s system and used mine instead. So I was forced to do my daily design work on my own CAD system. Nothing concentrates the mind better :-)

Seen the success of eating my own dogfood, I was thinking to learn to play Go online (I already planned to do that before). Some lessons and some online play. I am still a bit apprehensive to do that though, but I’ll have to jump in one day soon. I think that when I will use my own software to print Kifu’s and replay games, that I will find a billion annoyances that just have to be fixed ASAP. And that this will lead to a very much improved user experience. I think this is why SmartGo still is a scary competitor. All those programmers actually play Go themselves. I think Moyo Go Studio can not survive, long-term, without me spending some serious time on actually playing and studying Go myself. Actually using the software.

This is a dilemma, spending time playing Go versus working on the software. Because I can still improve the program without possessing the slightest Go skills. One of the things I want to make is filtering of pattern search results, so that you can get only the players you’re interested in. This has been requested a lot. But if I want to take this software seriously, I have to become a regular Go player myself, there is no way around the paradigm of Eating Your Own Dogfood. Otherwise there will be a communication gap between me and the customers.

The good thing is that I like to play mind games (I was an avid Chess player before) and that Go appeals to me. I played a dozen or so games and I improved rapidly and I liked it. With a much more relaxed release schedule (once or twice a year), it seems that I can divide my time more sensibly between programming for Go players and playing Go myself.

It pays off to be a data-salvage guy (I started a PC-repair shop at the beginning of this year).

In a bout of frustration, after not having coded on Moyo Go for almost a year, I had deleted its files because they took a lot of space. Not just the sources, especially all kinds of miscellaneous stuff like a million SGF’s and all kinds of other things. I had hundreds of Go-abstracts, board textures, pre-compiled databases, setup scripts and enormous pattern database files. But I mainly deleted them out of sheer spite. It’s not easy, to work thousands of hours in the best years of your life on something, only to discover that there is no way in hell anyone will ever put up a review on any well-visited Go site. AGA rejects all non-negative reviews for Moyo Go Studio, not to mention Sensei’s Library, where all mention of it is forbidden. And Gobase and the BGA pretend the software doesn’t exist.

I was very rigorous getting rid of it all: For example I had been working for months on a TsumeGo solver, I had coded a super-efficient move engine and I spent a few days designing a way to speed it up even more. I hand-drew a large sheet of complex binary matrices that would have to be hardcoded to be able to replace a lot of IF’s with a lookup table. All that work is gone now. I threw away some twenty kilo worth of computer-Go related material, and used the folders for day-to-day accounting of the PC-help shop.

Anyway - I have the code back (it appears to be the latest code but I am not sure yet) and all it took was a round of R-Studio. R-Studio is the data-recovery software I use for folks that have reformatted their disks and then they discover that they forgot to backup their wedding pictures. After R-Studio did its work, it still took me four days to get the sources to compile on my new laptop. At age 41 (I’ll be 42 in 5 days), it is my first laptop. I bought a cheap one: The HP 530. No fuel cell or Flash “disk” because things move slowly in computerland. I’m very glad with it, it hardly makes any noise.