Scraping Lewishams bin days

7 Sep 2020

Docker  Python  Web 

Lewisham Council have what at first glance appears to be a perfectly good webpage for figuring out what day your recycling is on. Except, there’s no API, and the pages appear to have been designed for the explicit goal of breaking every option for scripting them.

So I’ve build Ordure, a tool that mostly abuses Selenium (because the one thing they probably won’t break is human web browsing of it) to manage to fish data from it into Todoist. It even uses the bank holiday data to cope with those days (because the main page doesn’t).

Horrible things I learnt along the way:

• Attempts at posting to the address form seems to require you to have made a magic POST beforehand with other variables to make doing a later GET to get the results
• Part of the page returns a redirect as the page content (and not as headers, as every other site on the web would do)
• Some addresses (mine for example) return zero results, because apparently I have an address that confuses them (despite my having selected it from their list of addresses they know). On the basis they do actually pick up my rubbish, I’ve got it setup to get one of my neighbours results instead.

It actively feels like someone spent time and effort actually making it a shittier page. Making this standards compliant would be way easier! Also, they clearly have the data in a nice form, so any chance of an actual API Lewisham?

Previously: Customising Github profile pages Next: Immutable Systems Infrastructure, or how to mashup Kubernetes and Nix