Navigation

Mercurial for Adium

Wednesday, October 10th, 2007

So I took the time today to set up Mercurial. “Why set up Mercurial when I just said we should switch to Monotone?” you might ask. Good question! The things that had been bothering me about Mercurial that I had mentioned on the Adium mailing list turned out to be misconceptions on my part.

So I decided to go ahead with a test import. What a wild ride it’s been.

I decided to use Mercurial’s (a.k.a hg, as in the chemical symbol for Mercury) excellent hg convert built-in extension, combined with svnsync to speed up repository access, hopefully making for a quick import. Turns out I was mostly right.

svnsync

Installing svnsync from MacPorts was easy & painless. So far so good. Figuring out exactly how to get the destination repository to have revprops changeable was a bit tricky. Here’s how you do it:

 % cd dest-repo
 % cat > hooks/pre-revprop-change
 #!/bin/bash
 exit(0);
 ^D

After that, I just did an svnsync init and then a svnsync sync, which took roughly two hours. The repository it generated is roughly 1.6GiB.

hg convert

The next step was a little harder. I had to clone the convert-svn hg repository, because the svn option to hg convert hasn’t been in a release yet.

 % hg clone http://www.cecm.usp.br/~alexis/cgi-bin/hgwebdir.cgi/convert-svn 
 % cd convert-svn
 % make local

That got me an hg binary I could run in as ~/convert-svn/hg. Great. This is where it stopped being fun.

 % ~/convert-svn/hg convert adium-svn-copy adium-hg
 ...
 KeyError: u'svn:dfb16b33-163c-0410-afca-a8323011ef18/branches/adium-1.0@20779'

Um. What? That is not what I wanted to happen at all.

Hackers

Undiscouraged, I started pawing through the source to the svn converter (with help from the excellent folks in #mercurial on freenode). After a number of hours, I determined two things:

  1. It’s all Peter’s fault. ;)

    His habit of doing multi-branch merges screws with the internal data structure of the converter, which expects each commit to only be on one branch (not too unreasonable of an assumption, considering the normal use cases of svn).

  2. I don’t know how to fix it.

    The parser is bailing on the adium-1.0 branch, but the output from --debug leads me to believe that the adium-1.1 branch is horked similarly. Instead of fixing it outright, I decided to make a hack to only do a trunk import. You can pull that here. I kind of failed at using Mercurial there, ignore the odd history, please :)

Sharing Is Caring

Once I got hg convert running, it only took about an hour and a half to actually convert everything.

I decided to publish the repository on my personal website (which is hosted on Dreamhost), rather than on Adium website because this repo is a) not official and b) I wanted to see what it would be like for a contributor who wanted to host their own hg repository. The results are only kind of encouraging.

Setting up the tool to serve repositories, hgwebdir.cgi is not too bad. There’s plenty of information on the Mercurial wiki, and I’ll blog about it soon too.

Push It Along

At this point, I’m ready to push my changes. Sweet.

 % hg push http://cbarrett@evilfork.com/hg/adium/trunk-only
 ...
 error: broken pipe

Wait, what? That took almost an hour, and all I get is that there was a broken pipe?

After a bunch of googling around I found that others were having the same problem. Through tests it looked like I could push about 100 revisions at a time via http. This is totally unacceptable, as the repo has about 17,000 revisions, and they all need to go upstream.

I decided to try the ssh:// method, which is a bit more involved.

 % hg push ssh://evilfork@hostname.dreamhost.com/public_html/hg/repos/adium/trunk-only
 ...
 remote: error: device out of disk space

Wow. Just wow. /tmp is out of disk space. Syncing only a couple thousand at a time is still unbelievably slow.

I eventually just gave up trying to push and just scp‘d a tarball of the repository.

Conclusions

Regardless of the hoops I had to jump through, the repository is up. It is for now trunk only, and contains full history back to the very beginning of time. To use it, simply do:

 hg clone http://evilfork.com/hg/adium/trunk-only

A clone of that repository is going to be 923M on disk, or thereabouts. Luckily for you, you don’t have to eat that cost multiple times to have multiple trees, because hg uses hardlinks when doing local clones (when the OS supports). I hope to create a version with only a year of history with hg transplant rather soon, for people who don’t want to download all that ancient history.

Still though, despite that massive amount of history, operations are still speedy. hg tip and hg log -l5 are instantaneous (0.2s), and a full on hg log of 17,000+ revisions takes about 5s (mostly for the text to scroll by).

A number of people have brought up the fact that testers will all leave because they don’t want to have to download all this history. Personally I don’t think they should be using hg or svn at all. Instead we should be offering them nightly or some sort of other regular builds using Buildbot.

For contributors, a reduced history clone would help improve things. However, hgweb also offers the option to allow downloading of tarballs right from the web interface, which is even less disk/bandwidth intensive than the current svn checkout system.

Finally, I hope that by giving folks a chance to play with a distributed SCM people will learn to see some of the benefits. I urge Adium folks to read through the hgbook and to explore the Mercurial wiki. It’s a nice, relatively simple system with some great features (mq is really sweet for patch management if you want to go down that road). I’ll blog about ways to use Mercurial later as well.

If people are interested, I can try getting svnsync and hg convert running on the server every night. It should be much faster since it’s just an incremental update.

Comments

  1. Colin replied on October 11th, 2007:

    Hmm, looks like bzr-svn supports crazy BoredZo commits, but not tags. Whee!