At GDS, we often talk about user needs: the idea that services should be designed around the needs of the user, rather than around the needs of government. For many of our development teams, this translates to making sure that the service or application they build has a workflow that is logical to those who need to use it, that it doesn't force the user to provide extraneous information and so forth.
I work on the GOV.UK Infrastructure team. It's our responsibility to ensure that the GOV.UK infrastructure platform is stable, available and reliable for our users: GDS development teams, and the millions of users around the world who access GOV.UK each month.
Whilst it's true to say that - due to the nature of the content on the website - GOV.UK receives most of its traffic from within the United Kingdom, we do still have a sizeable number of users from abroad who come to us to find out, for instance, whether they need to apply for a visa, find out about their duty-free allowances, or get details of their nearest British Embassy or Consulate. Our users are everywhere; we need to care for them all.
Delivering content to users
As with many large websites, GOV.UK is fronted with a CDN - a Content Delivery Network. CDNs work by caching copies of a website in geographical locations close to users, rather than forcing users to wait for their request to be processed by servers at the origin - the servers which host the canonical copy of the website. By doing this, they can help alleviate load and make the delivery of content back to the user faster. Only those requests for which the cache node cannot respond are routed back to the origin for processing. Using a CDN also helps keep GOV.UK resilient by protecting against disasters such as origin failures.
Today, the vast majority of requests for content from GOV.UK are responded to by Fastly. In the past, we've also used CloudFlare and Akamai. We continue to re-evaluate our choice of CDN regularly in order to ensure that we're always meeting user needs and getting the best value for taxpayer money.
Content delivery as code
We strongly believe in the idea of infrastructure as code: your infrastructure should be treated in the same way that your applications are, using tenets such as source control and test-driven development. We write our infrastructure, and supporting tools, in Ruby, Python, Go, YAML and Puppet, to name a few. Developers work to a slightly-modified version of GitHub Flow as their pipeline - the key difference is that we deploy the master branch after pull requests have been merged in to it.
As mentioned, we've previously used CloudFlare to deliver assets - things like stylesheets, images, PDFs and so forth - to users. At the time, a colleague wrote a tool to configure CloudFlare using configuration stored in version-controllable JSON files. Internally, we'd been using something quite similar (with a few rough edges) to deploy new configuration to Fastly. It lived in an internal Git repository, and was manually triggered on a Jenkins job whenever a new configuration change needed to be made. There was no real need for it to continue to exist, hidden away, in that repository; after all, if we make it open, we can make it better.
One evening, I was watching a video from Target about how they fronted their API-serving stack with Fastly. The presenter - Eddie Roger - noted how his team didn't yet have a way to manage their VCL (Fastly is built on a modified version of Varnish; VCL is the Varnish Configuration Language) in code and ship it as they would their other infrastructure.
The next day, I spent some time with a colleague - Alex Muller, who also works on the GOV.UK Infrastructure team - working out how to best to open the script up. After reviewing it manually, we decided that the only change we needed to make was to rip out a hash of service-specific variables at the top of the script. In keeping with the idea that code and data should be separate, we moved these variables in to a YAML file, and ensured that we reconstruct the hash at runtime from the data in the YAML file. This made the code much cleaner, since service-specific settings were now called in with only one line of code, like this. Keeping in line with the naming convention already started by Cloudflare-Configure, we christened our new project Fastly-Configure. After some discussion with colleagues in a pull request, the script was merged to the master branch, ready for use.
One of my favourite things about Fastly-Configure is that it runs without issue in a continuous integration environment, such as Jenkins or GoCD. This makes uploading new configuration and maintaining accountability much easier.
Pull requests welcome!
We've opened up Fastly-Configure to the world. You're welcome to use it in your own stack, and my colleagues and I very much appreciate pull requests and issues if things aren't working for you.
If this sounds like a good place to work, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can follow Andrew on Twitter, sign up now for email updates from this blog or subscribe to the feed.
