Thursday, December 9th, 2010
Over the past year or so I’ve started writing scripts again. (Why I started, stopped, and started again is enough for a post of its own.) At first just here and there, one-off things, but I’ve started to accumulate several that I use pretty frequently now.
Previously (circa early-mid 2000s) most of my scripts were written in shell script (occasionally in Perl). I’d just
chmod a+x them and put ‘em in
~/.shellscripts (which I’d put on my path).1 This worked great.
Most of my scripts now are written in Python. I like pretty much everything about Python for writing sysadmin-type scripts: the language is focused on being easy to type, the standard library is gigantic, etc. Relevant to this discussion, the
argparse library makes building robust and easy to use command line interfaces a snap.
However Python’s biggest wart is its packaging problem; I warn anyone who wants to learn the language about this. (For those aren’t aware of the issues, a very brief summary: packaging/install tool development stagnated for a while in the mid-late 2000s and it’s only recently become active again.)
setuptools and wrapper scripts
The standard packaging library (for all intents and purposes) is
setuptools.2 One of the nicer features of
setuptools is its ability to generate wrapper scripts for you. You tell it which python module + function to use as the entry point for the script and during the install process it will generate a script for you. Unfortunately, there are some problems.
develop vs. install
One of the options when installing a setuptools project is to install it in
develop mode. (This mode symlinks installed sources to the ones in your development tree, removing a step from your edit/test/debug cycle.)
However, script wrappers are as far as I can tell broken in
develop mode. Specifically, you can’t uninstall them.
installing every time should work, then, right? Here I’m less certain, but I have seen some posts that seem to indicate that setuptools has tons of bugs with both 1) uninstalling in general and 2) pave-over installs as you incrememnt versions of your software.
What’s the Alternative?
I have no idea, actually. I’m hoping folks out there on the internet can help. I’m looking for some best practices for installing and maintining my growing flotilla of scripts. Here are some behaviors I’m pretty sure I want, but I’m open to being wrong about these:
- I need these scripts to be on my path when I start a new shell. I also still want to be able to use the system Python & site-packages. (I believe this rules out virtualenv & virtualenvwrapper, but feel free to contradict me.)
- I need the sources for these scripts to remain private, at least for now. I don’t feel like any of them are generally useful (some yet, some ever) so putting them on PyPI isn’t an option.
- I need to able to split my code up into multiple modules. Modules are the main reason I switched to Python for these kinds of scripts in the first place — shebang-style scripts just don’t cut it for me anyomre.3
- I need to coordinate easily between multiple machines. This one is more vague: I’m not opposed to having to manually upgrade or sync things if it comes to that; however the scripts current live in Dropbox (and I execute them with
python ~/Dropbox/path/to/script.py) and it would be nice to maintain that “always up to date” behavior — but it’s certainly not required.
A Cry for Help
If anyone out there has a good solution for this (or if I’m wrong about something), please leave a comment or get in touch with me: my contact information is on the about page.
This is actually a false memory; I don’t remember where I put them, probably in
/usr/local/bin. However, were I doing this today, that’s where I’d put them. ↩
It doesn’t come with the standard library though, which should give you a glimpse into some of the problems Python faces in this area. ↩
As I’ve written more and more code professionally, I find that I’m uncomfortable continuing to use scripts that have bad “code smells” day-to-day — and 500+ line Perl or shell scripts usually smell pretty awful. ↩