Wednesday, October 10th, 2007
So I took the time today to set up Mercurial. “Why set up Mercurial when I just said we should switch to Monotone?” you might ask. Good question! The things that had been bothering me about Mercurial that I had mentioned on the Adium mailing list turned out to be misconceptions on my part.
So I decided to go ahead with a test import. What a wild ride it’s been.
I decided to use Mercurial’s (a.k.a
hg, as in the chemical symbol for Mercury) excellent
hg convert built-in extension, combined with svnsync to speed up repository access, hopefully making for a quick import. Turns out I was mostly right.
Installing svnsync from MacPorts was easy & painless. So far so good. Figuring out exactly how to get the destination repository to have revprops changeable was a bit tricky. Here’s how you do it:
% cd dest-repo % cat > hooks/pre-revprop-change #!/bin/bash exit(0); ^D
After that, I just did an
svnsync init and then a
svnsync sync, which took roughly two hours. The repository it generated is roughly 1.6GiB.
The next step was a little harder. I had to clone the convert-svn hg repository, because the svn option to
hg convert hasn’t been in a release yet.
% hg clone http://www.cecm.usp.br/~alexis/cgi-bin/hgwebdir.cgi/convert-svn % cd convert-svn % make local
That got me an
hg binary I could run in as
~/convert-svn/hg. Great. This is where it stopped being fun.
% ~/convert-svn/hg convert adium-svn-copy adium-hg ... KeyError: u'svn:dfb16b33-163c-0410-afca-a8323011ef18/branches/adium-1.0@20779'
Um. What? That is not what I wanted to happen at all.
Undiscouraged, I started pawing through the source to the svn converter (with help from the excellent folks in #mercurial on freenode). After a number of hours, I determined two things:
It’s all Peter’s fault. ;)
His habit of doing multi-branch merges screws with the internal data structure of the converter, which expects each commit to only be on one branch (not too unreasonable of an assumption, considering the normal use cases of svn).
I don’t know how to fix it.
The parser is bailing on the adium-1.0 branch, but the output from
--debugleads me to believe that the adium-1.1 branch is horked similarly. Instead of fixing it outright, I decided to make a hack to only do a trunk import. You can pull that here. I kind of failed at using Mercurial there, ignore the odd history, please :)
Sharing Is Caring
Once I got
hg convert running, it only took about an hour and a half to actually convert everything.
I decided to publish the repository on my personal website (which is hosted on Dreamhost), rather than on Adium website because this repo is a) not official and b) I wanted to see what it would be like for a contributor who wanted to host their own hg repository. The results are only kind of encouraging.
Setting up the tool to serve repositories,
hgwebdir.cgi is not too bad. There’s plenty of information on the Mercurial wiki, and I’ll blog about it soon too.
Push It Along
At this point, I’m ready to push my changes. Sweet.
% hg push http://email@example.com/hg/adium/trunk-only ... error: broken pipe
Wait, what? That took almost an hour, and all I get is that there was a broken pipe?
After a bunch of googling around I found that others were having the same problem. Through tests it looked like I could push about 100 revisions at a time via http. This is totally unacceptable, as the repo has about 17,000 revisions, and they all need to go upstream.
I decided to try the ssh:// method, which is a bit more involved.
% hg push ssh://firstname.lastname@example.org/public_html/hg/repos/adium/trunk-only ... remote: error: device out of disk space
Wow. Just wow. /tmp is out of disk space. Syncing only a couple thousand at a time is still unbelievably slow.
I eventually just gave up trying to push and just
scp‘d a tarball of the repository.
Regardless of the hoops I had to jump through, the repository is up. It is for now trunk only, and contains full history back to the very beginning of time. To use it, simply do:
hg clone http://evilfork.com/hg/adium/trunk-only
A clone of that repository is going to be 923M on disk, or thereabouts. Luckily for you, you don’t have to eat that cost multiple times to have multiple trees, because hg uses hardlinks when doing local clones (when the OS supports). I hope to create a version with only a year of history with
hg transplant rather soon, for people who don’t want to download all that ancient history.
Still though, despite that massive amount of history, operations are still speedy.
hg tip and
hg log -l5 are instantaneous (0.2s), and a full on
hg log of 17,000+ revisions takes about 5s (mostly for the text to scroll by).
A number of people have brought up the fact that testers will all leave because they don’t want to have to download all this history. Personally I don’t think they should be using hg or svn at all. Instead we should be offering them nightly or some sort of other regular builds using Buildbot.
For contributors, a reduced history clone would help improve things. However, hgweb also offers the option to allow downloading of tarballs right from the web interface, which is even less disk/bandwidth intensive than the current svn checkout system.
Finally, I hope that by giving folks a chance to play with a distributed SCM people will learn to see some of the benefits. I urge Adium folks to read through the hgbook and to explore the Mercurial wiki. It’s a nice, relatively simple system with some great features (
mq is really sweet for patch management if you want to go down that road). I’ll blog about ways to use Mercurial later as well.
If people are interested, I can try getting
hg convert running on the server every night. It should be much faster since it’s just an incremental update.