Navigation

A Cry for Help Re: Installing and Managing Python Scripts

Thursday, December 9th, 2010

Over the past year or so I’ve started writing scripts again. (Why I started, stopped, and started again is enough for a post of its own.) At first just here and there, one-off things, but I’ve started to accumulate several that I use pretty frequently now.

~/.shellscripts

Previously (circa early-mid 2000s) most of my scripts were written in shell script (occasionally in Perl). I’d just chmod a+x them and put ‘em in ~/.shellscripts (which I’d put on my path).1 This worked great.

Enter Python

Most of my scripts now are written in Python. I like pretty much everything about Python for writing sysadmin-type scripts: the language is focused on being easy to type, the standard library is gigantic, etc. Relevant to this discussion, the argparse library makes building robust and easy to use command line interfaces a snap.

Packaging

However Python’s biggest wart is its packaging problem; I warn anyone who wants to learn the language about this. (For those aren’t aware of the issues, a very brief summary: packaging/install tool development stagnated for a while in the mid-late 2000s and it’s only recently become active again.)

setuptools and wrapper scripts

The standard packaging library (for all intents and purposes) is setuptools.2 One of the nicer features of setuptools is its ability to generate wrapper scripts for you. You tell it which python module + function to use as the entry point for the script and during the install process it will generate a script for you. Unfortunately, there are some problems.

develop vs. install

One of the options when installing a setuptools project is to install it in develop mode. (This mode symlinks installed sources to the ones in your development tree, removing a step from your edit/test/debug cycle.)

However, script wrappers are as far as I can tell broken in develop mode. Specifically, you can’t uninstall them. installing every time should work, then, right? Here I’m less certain, but I have seen some posts that seem to indicate that setuptools has tons of bugs with both 1) uninstalling in general and 2) pave-over installs as you incrememnt versions of your software.

What’s the Alternative?

I have no idea, actually. I’m hoping folks out there on the internet can help. I’m looking for some best practices for installing and maintining my growing flotilla of scripts. Here are some behaviors I’m pretty sure I want, but I’m open to being wrong about these:

  • I need these scripts to be on my path when I start a new shell. I also still want to be able to use the system Python & site-packages. (I believe this rules out virtualenv & virtualenvwrapper, but feel free to contradict me.)
  • I need the sources for these scripts to remain private, at least for now. I don’t feel like any of them are generally useful (some yet, some ever) so putting them on PyPI isn’t an option.
  • I need to able to split my code up into multiple modules. Modules are the main reason I switched to Python for these kinds of scripts in the first place — shebang-style scripts just don’t cut it for me anyomre.3
  • I need to coordinate easily between multiple machines. This one is more vague: I’m not opposed to having to manually upgrade or sync things if it comes to that; however the scripts current live in Dropbox (and I execute them with python ~/Dropbox/path/to/script.py) and it would be nice to maintain that “always up to date” behavior — but it’s certainly not required.

A Cry for Help

If anyone out there has a good solution for this (or if I’m wrong about something), please leave a comment or get in touch with me: my contact information is on the about page.


  1. This is actually a false memory; I don’t remember where I put them, probably in ~/bin or just /usr/local/bin. However, were I doing this today, that’s where I’d put them. 

  2. It doesn’t come with the standard library though, which should give you a glimpse into some of the problems Python faces in this area. 

  3. As I’ve written more and more code professionally, I find that I’m uncomfortable continuing to use scripts that have bad “code smells” day-to-day — and 500+ line Perl or shell scripts usually smell pretty awful. 

Comments

  1. Doug Hellmann replied on December 9th, 2010:

    If you’re going to run develop mode or install the scripts, you can install them into your user tree instead of the site-wide tree by using the —user option to “python setup.py install” . On Unixish systems the installed files (scripts, wrappers, packages, everything) ends up under ~/.local (~/.local/bin, ~/.local/lib/pythonX.Y, etc.).

    I’m not sure why you think a virtualenv won’t work for this, though. Can you elaborate?

  2. Colin Barrett replied on December 9th, 2010:

    Doug: As far as I’m aware, virtualenv requires specifically activating it (with bin/activate). Also, it copies the system Python executable into the virtualenv — I want to be able to use whatever the system Python is (across Mac OS X updates). I don’t think it’s unreasonable to worry about since the virtualenv is going to be long lived, but maybe I’m wrong about that.

  3. Benno Rice replied on December 9th, 2010:

    virtualenv will happily use your system packages unless you tell it not to with —no-site-packages. You can also make your virtualenv relative which makes copying it around easy.

    I do multi-module development by having a common virtualenv and having all the packages operating in develop mode (ie python setup.py develop). We then push changes to git repositories which trigger builds and generate Debian packages off those. The whole system works pretty well. Feel free to drop me an email if you’ve got further questions.

  4. Erik Swanson replied on December 9th, 2010:

    I install every discrete thing I use into a separate virtualenv, then create symlinks to the scripts in a common directory in my path.

    For example, Mercurial is installed under ~/.prefix/mercurial/ and the ‘hg’ command winds up in my path via a symylink from ~/.command/hg -> ../.prefix/mercurial/bin/hg.

    Uninstallation requires nothing more than rm, and for ‘clean’ upgrading you can simply move the old virtualenv out of the way (e.g. `mv ~/.prefix/mercurial ~/.prefix/mercurial_old) then install the new version in a new virtualenv.

  5. Carl Meyer replied on December 9th, 2010:

    Hi Colin,

    There are a variety of ways you could approach this that I think could work well; most of them would involve using pip (http://pip.openplans.org) as a starting point; it’s the best tool currently available for installing (and, in particular, uninstalling) Python packages.

    I think there are a few different workflows that could meet your criteria; which one is best depends on some additional preferences you haven’t clarified here. How about you just ping me on Twitter (@carljm) or IRC (carljm on Freenode); I’d be happy to help you sort through the options and get something that works for you, and then you can post a followup if you feel like it.

  6. Colin Barrett replied on December 9th, 2010:

    Carl: Thanks very much for your offer. I’ll find you on freenode shortly.

  7. Doug Hellmann replied on December 9th, 2010:

    virtualenv does copy the system python, but it is safe to re-run virtualenv to update an existing environment if the system python changes.

    The scripts installed into a virtualenv should be configured to use the python interpreter in that environment, so as long as that bin directory is in your path it should “just work”.

  8. Colin Barrett replied on December 9th, 2010:

    Doug: Thanks a lot. I talked it over with Carl as well and I think I’m gonna give the virtualenv method a shot; we’ll see how it goes.

    Thanks everyone.