Scraping Lewishams bin days

7 Sep 2020

Docker  Python  Web 

Lewisham Council have what at first glance appears to be a perfectly good webpage for figuring out what day your recycling is on. Except, there’s no API, and the pages appear to have been designed for the explicit goal of breaking every option for scripting them.

So I’ve build Ordure, a tool that mostly abuses Selenium (because the one thing they probably won’t break is human web browsing of it) to manage to fish data from it into Todoist. It even uses the bank holiday data to cope with those days (because the main page doesn’t).

Horrible things I learnt along the way:

It actively feels like someone spent time and effort actually making it a shittier page. Making this standards compliant would be way easier! Also, they clearly have the data in a nice form, so any chance of an actual API Lewisham?

Previously: Customising Github profile pages Next: Immutable Systems Infrastructure, or how to mashup Kubernetes and Nix