As you read in the previous post in this series, we have built a frontline router using Go.
If you cast an eye across our many and varied open source projects, the common theme is that they're generally written in Ruby. We have strong Ruby developers and an excellent track record at building and running Ruby services.
So how did we get to choosing Go?
Back to the beginning
Nick has already explained the problem space well, so time for a little history lesson instead.
Back in the alpha and beta days of GOV.UK, we actually had a router. This router had some of the same characteristics desirable of a frontline piece of infrastructure: it offered good performance and resilience. It was also written in Scala.
There were a few issues with this, which I'll enumerate quickly:
- Scala is great for performance, but quite bad at resource usage
- No-one in the core GOV.UK team had a deep knowledge of Scala, and particularly how the old router worked
- It required us to put the JVM onto our frontline cache boxes (currently just Nginx and Varnish), which adds complexity and resource constraints
Whilst the 1st and 3rd point are things we can deal with, it's number 2 that was the real sticking point. We had to have something we could not only maintain but iterate on and build out as required.
So at this point we've ruled out the old Scala router, and in fact it was removed for launch in place of the Varnish configuration described in Nick's article. In order to move forward with a router, we had to go back to first principles.
Performance
As explained above, while GDS as a whole has projects built on a variety of stacks, GOV.UK is primarily a Ruby shop. The performance characteristics we want from a router (high concurrency, minimal latency added per request, low memory usage so we can contain bursts) mean we need to pick a language with a more aggressive performance baseline. This rules out pretty much all scripting languages.
The router is also, by design, largely an HTTP reverse proxy with some extra statefulness - it has minimal features and complexity. So a strong networking library, or pedigree, is desirable.
New and shiny
Programmers like learning new languages (many spend their spare time working on new things), so giving developers the opportunity to learn new things on work time seems like a good move for many reasons:
- It helps the programming team as a whole. The ideas and methods from other languages help us reason about things better, and allow us to approach old problems in a fresh way
- It helps with retention. Programmers might not feel the need to move elsewhere if they can work on different things at work
- It allows us to try out the features of new languages on tightly scoped projects so we can get a flavour for whether or not they'd be a good fit for larger or more complex things. R&D.
Maintainability
Based on the above criteria of performance, friendliness with HTTP, and wanting to introduce a new language, we have a few strong candidates. Erlang, Go, Scala, and Clojure are all good fits and represent the current state of the art. Since we wanted to use a new language here it makes sense for us to pick something that's easy for all of us to learn and easy for future developers on the team to learn and maintain.
We can immediately rule out Scala based on the previous issues we had with the router. We can also rule out Clojure due to unfamiliarity and us not wanting to deploy the JVM on production. We can now rule out Erlang (as much as it pains me - I personally have a soft spot for Erlang and the actor model of concurrency it gives us) because of the unfamiliar syntax and learning curve.
As this leaves us with just Go in the 'new language' bucket, we can be reasonably sure that it will fit well with our needs. But what of the language itself?
How Go specifically fits our stack
GOV.UK is a collection of loosely coupled applications that speak to each other via HTTP. When we put it like that, it's easy to see why a high performing language which speaks HTTP very well would fit in.
Documentation
Go has wonderful documentation for its standard library. All source code is clearly written, well-commented, and is a great source of learning for new and seasoned Go programmers. The combination of documentation and being open source makes it very easy to compose richer functionality out of modules from the standard library. There's even an excellent guide to writing Go code idiomatically, and a tool that formats your code correctly.
Simplicity
As mentioned above it's fairly easy to compose robust new services that use HTTP just by using the Go standard library. This makes testing and performance a breeze as they're both well tested and very fast - we just need to test our own wrapped functionality (more on that to come).
Ease of deployment
To deploy the Router we just compile a binary and ship that single file to our production servers. This is significantly easier than the rsync/symlink/Unicorn dance we have to perform when deploying Rails services, and it's obviously significantly faster and has less moving parts that could fail.
Concurrency is built-in
When writing high performing network servers it's useful to delegate some of the expensive work to other threads - but threads are difficult to reason about and to manage well. Go handles this using goroutines and other native language constructs that mean we never need to worry about such things. An example of how we use goroutines is in our logger module.
The future
Other colleagues will cover other deeper areas of the router project in more detail soon and explore how we really put it through its paces, but it's a testament to the ease with which they took to Go (a new language for all of us) that we built and deployed the Router in front of the GOV.UK website in just 6 weeks. I look forward to working with it again.
If work like this sounds good for you, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can follow Brad on twitter, sign up now for email updates from this blog or subscribe to the feed.
17 comments
Comment by csmuk posted on
Did 6 weeks development time (and post-completion maintenance, deployment, configuration) save you anything over picking something off the shelf? A mid-rate developer costing for this is £6400 excluding the ongoing maintenance and deployment and the desk, systems and electricity he/she is utilising.
How much did the previous N attempts cost over time?
How many products did you evaluate before building your own?
Why did you use Scala to start with when there are COTS solutions to this?
Don't forget the taxpayer is paying for this - this shouldn't be a technology playground.
Comment by James Stewart posted on
We've yet to see an off the shelf tool in this space that does exactly what we want while remaining simple and easy for us to iterate as our requirements develop. This is a rapidly developing area and we're keeping a close eye on what other teams are doing as well as continuing to share our work for others to be aware of, use and contribute to if it's useful to them.
As we'll be talking about in subsequent posts the majority of the time involved in this project was actually spent in testing and integration, which would be a consideration whatever we chose for the core piece of software. We talk more about our approach to choosing technology at https://www.gov.uk/service-manual/making-software/choosing-technology
Comment by Hugh Jass posted on
"No-one in the core GOV.UK team had a deep knowledge of Scala, and particularly how the old router worked"
So.. you had no Scala experts but yet decided to pick a language none of you knew?
This whole blog post just reads like a confused post-hoc justification for a decision you made based on nothing but wanting to work with a seemingly 'cool' language.
Comment by James Stewart posted on
We had one or two scala experts early on and one of them was the main force behind the scala implementation. He quickly moved on to another project and as time went on it became clearer that GOV.UK wasn't going to be doing much else with Scala then or in the near future.
One thing Brad didn't really touch on here was the comparative complexity of the two codebases. The Go codebase is much smaller and simpler than the Scala code ever was, and deeply rooted in the standard library, so it's simpler for someone new to the language to pick up.
Comment by Ben posted on
Love, Go, but curious if HAProxy (http://haproxy.1wt.eu/) couldn't have met your needs. I have great luck with it.
Comment by Brad Wright posted on
That's a good question. We needed a bit more statefulness than what I understand HAProxy offers - in particular we need to store and action redirects as well as offer a reverse proxy function. We also need to soft-delete routes, which we return as a 410. The full features of the router are clearly explained in the first post of this series:
https://gdstechnology.blog.gov.uk/2013/12/05/building-a-new-router-for-gov-uk/
Comment by Victor posted on
I love go but haproxy offer everything to manage a frontline http router. even your redirects and contention(also a god damn performance)
Comment by Rosario posted on
How come you didn't consider Elixir (Erlang based with a syntax similar to Ruby)?
Comment by Brad Wright posted on
Elixir is an abstraction on top of Erlang, which for a low-level networking thing moves us one step further away from understanding the guts of it. Additionally we'd have to install the Erlang virtual machine on our deployment targets which gives us a similar level of deployment complexity to the JVM, but with less of the shared understanding we already have of the JVM.
Comment by John Murray posted on
You're distrust of VMs aside, it seems that Erlang would have made your deployment story even more simple given that Erlang VM can support two versions of any module, allowing applications to upgrade themselves whenever is most appropriate for them.
Just a thought.
Comment by Brad Wright posted on
There's no distrust of VMs - my intent was to say that deploying extra VMs we don't have expertise in adds complexity.
I'm very familiar with Erlang and one thing I do miss in Go is native hot code reloading the way Erlang does it - there are some patterns emerging but nothing as nice as Erlang's way of doing things that I could see.
Comment by Ian Feather posted on
Really interesting. Have you found the R&D side of it to be fruitful? I think the new & shiny part of this post is a really good point, and something more companies should be focusing on to keep developers interested.
We made a similar app in Lua which compiles as an nginx module.
Comment by James Stewart posted on
Interesting to hear you're using lua. I'd worked on a lua extension for nginx to do this too but abandoned it, partly because the Go implementation came together more quickly and elegantly, and partly because of the need to recompile nginx to get the set of extensions I needed. It did feel like an interesting approach, though, and one of our colleagues has also been playing with it to do some lightweight filtering of API requests to protect backend services so we may be working with lua more in future.
Comment by Lelala posted on
OK, actually a first "real live" and "professional" usecase for Go, driven by a government/public entity - something thats very beautiful, since it leverages the potential of Go.
I predict, in the more far future, you will see similar usecases of Go, since the language really has some great aspects. (and, by the way, its far from beeing as rusty as C++ is, for example)
Comment by Guilherme Aiolfi posted on
I'm curious to know why Rust (from mozilla) was not considered.
Comment by Brad Wright posted on
The simple answer is that Go was already on our radar as an interesting language, and it was the language Nick originally chose to spike a new router. Rust is certainly looks like a useful language though.
Comment by John Murray posted on
Rust may not have been a good choice for the team though (although I am a fan of Rust) given that it has been rather volatile. By that I mean it has changed non-trivially from version to version and enough so to really slow down a development team if they want to stay up-to-date with the latest version. I feel Rust will be a better choice when they start to "slow down" in the same way that the Go team promised backwards compatibility for the 1.x series.