Originally posted at https://tech.labs.oliverwyman.com/blog/2018/07/03/psa-many-ways-to-manage-your-python-dependencies/
Having recently learnt about a few more tools, I felt it was worth writing this post. It’s pretty much a PSA, and some of you may well have found most of these, but there are a lot of Python developers out there who haven’t, and the improvements when using some them are considerable, so it’s worth knowing about them.
Let’s start with the big one: Virtualenv. It’s docs say “virtualenv is a tool to create isolated Python environments”. If you haven’t used it yet, you may be going “why do I want that?”. Well, because this lets you solve a big one of the ‘works on my machine’ issues. If you’re working on more than one project, and they have conflicting package requirements, then you’re going to be repeatedly (un)installing things, and it’s just not fun.
Virtualenv solves that by letting you easily swap between the packages for both Project A and Project B without repeated installation. The user guide is pretty good, and well worth a read. Given a few recent conversations I’ve had, some people are still writing Python without Virtualenv, and I’d strongly advise just not to do that.
Ok, so you’ve got Virtualenv setup. You’ve installed a whole bunch of packages for your project, and you’ve probably got as far as finding pip freeze which lets you snapshot the set of local packages, and then running the standard ‘pip install -r requirements.txt’ when you want it installed on someone else’s machine or in CI. However, that requirements.txt will start to fill up over time, and it’s a real mixed bag of ‘stuff I need’ and ‘stuff required by stuff I need’, and trying to work through why something is installed is a lot of a faff, and spotting old dependencies you don’t need any more is often difficult.
The next tool you really need is pip-tools. This now lets you write a ‘requirements.in’ which lists all the packages you actually want, run ‘pip-compile’ to generate a requirements.txt with a bunch of useful comments about where a package got pulled in from, and then run ‘pip-sync’ to (un)install things to bring your environment back in sync with the requirements files. I find myself running ‘pip-sync -n’ (dry-run of pip-sync) a lot, especially when I’ve just been messing around with the current set of packages to work out what I want to keep in my requirements.in.
This is as far as most projects need. In fact, up until very recently, I hadn’t dealt with any of the later tools I’ll now talk about.
You now have your packages pretty much sorted out. However, there’s another aspect of things: what Python version to run with? Most projects tend to just want either 2 or 3 and “any of the not ancient versions” more or less. This is mostly pretty easy. Except when you run into the problem of Travis CI’s most recent environment being a 4-year old version of Ubuntu running Python 2.7.6, Tornado just decided to deprecate Python <2.7.9 and one of my projects needs that.
Pyenv solves this by letting you install multiple versions of Python easily and switch between them. For the Ruby fans, it’s a lot like rbenv. Getting this to work with Travis is fiddly, but luckily there’s a helper script for that (which will hopefully release a new version soon, as it’s got a useful fix in there).
If you’ve got this far, you’re dealing with a whole bunch of tools, and your setup is a little bit more work than you’d probably really like. pipenv is a new and shiny tool intended to deal with all these things in one go. It uses pyenv to setup whatever Python version you want, makes virtualenvs as needed, and builds lock files from your dependencies. It even copes with packages you only need for development (which is doable with earlier tools, but manually).
Having just switched over tachikoma to using it, I’m reasonably happy, but as a brand-new tool that stops using requirements.txt (which most other projects dealing with Python will assume is where packages go) you may have issues. For example, I ran into issues with tox but thankfully my needs were small enough I could remove that entirely.
pyup.io is an extra little tool (well, website mostly) that I’ve found useful recently. It’s of most use to public Github repos (although if you give them some money, you can make it work for private ones as well), as the website can automatically open PRs for any updated packages. They’ve even got a whole interesting security vulnerability checker so you can only update packages with security vulnerabilities if you’d like instead.
There’s probably a few others out there, but that’s the big names, and most of the issues. Please add comments if you’ve got any other Python tools we should all know about!