This is the fourth piece in a series of the router component we deployed in front of GOV.UK towards the end of 2013. You may also want to read the previous posts in the series.
In developing the router, we needed to be confident that it functioned correctly. We decided to create an integration test suite that was independent from the implementation as far as possible. To do this, it needed to start up some sample backends, populate some routes in the database, spin up the router, and fire some requests at it.
We created some test backends in Go, and used RSpec (a Ruby testing framework) to do the orchestration and testing. There are 3 main areas we needed to cover with the tests - route selection, the reverse-proxy function, and some relative performance tests.
Route selection
To test the route selection, we created a simple backend that could be configured to return a given string to any request. We could then spin up a few of these, configure some routes pointing at them, and make some assertions about where requests were routed. The result is the backend_selection_spec.rb, which ends up with tests like this:
describe "simple exact routes" do
start_backend_around_all :port => 3160, :identifier => "backend 1"
start_backend_around_all :port => 3161, :identifier => "backend 2"
before :each do
add_backend("backend-1", "http://localhost:3160/")
add_backend("backend-2", "http://localhost:3161/")
add_backend_route("/foo", "backend-1")
add_backend_route("/bar", "backend-2")
add_backend_route("/baz", "backend-1")
reload_routes
end
it "should route a matching request to the corresponding backend" do
response = router_request("/foo")
expect(response).to have_response_body("backend 1")
response = router_request("/bar")
expect(response).to have_response_body("backend 2")
response = router_request("/baz")
expect(response).to have_response_body("backend 1")
end
it "should 404 for children of the exact routes" do
response = router_request("/foo/bar")
expect(response.code).to eq(404)
end
# etc...
end
Reverse-proxy function
There are a number of things detailed in the HTTP spec that a reverse-proxy must do. A good summary of them is here.
To enable us to make assertions about the details of the request that arrived at a backend we created an echo backend that encodes details of the received request as json, and returns that as the response body. The resulting tests cover lots of details around header handling, request and body handling. For example testing the population of the Via header:
describe "setting the Via header" do
# See https://tools.ietf.org/html/rfc2616#section-14.45
it "should add itself to the Via request header for an HTTP/1.1 request" do
response = HTTPClient.get(router_url("/foo"))
headers = JSON.parse(response.body)["Request"]["Header"]
expect(headers["Via"].first).to eq("1.1 router")
response = HTTPClient.get(router_url("/foo"), :header => {"Via" => "1.0 fred, 1.1 barney"})
headers = JSON.parse(response.body)["Request"]["Header"]
expect(headers["Via"].first).to eq("1.0 fred, 1.1 barney, 1.1 router")
end
it "should add itself to the Via request header for an HTTP/1.0 request" do
headers, body = raw_http_1_0_request(router_url("/foo"))
headers = JSON.parse(body)["Request"]["Header"]
expect(headers["Via"].first).to eq("1.0 router")
headers, body = raw_http_1_0_request(router_url("/foo"), "Via" => "1.0 fred, 1.1 barney")
headers = JSON.parse(body)["Request"]["Header"]
expect(headers["Via"].first).to eq("1.0 fred, 1.1 barney, 1.0 router")
end
# etc...
end
An interesting edge-case that came out of the testing relates to the User-Agent header. The Go HTTP client library will add a User-Agent header of "Go 1.1 package http" to outbound requests that don't already have one set. This is not something we want a reverse-proxy to do, so we had to add some specific code to handle this.
Relative performance testing
The third main area of testing was around performance. Our goal here was to have some tests that could run as part of our automated test suites and catch any significant performance regressions. These would not be exhaustive performance tests (this testing has been covered separately as Dan has already written about).
For this testing we were interested in how much latency the router added to a request, not the raw request latency. To measure this we used Vegeta to run a load test against the router, and simultaneously against a backend directly. Using this approach we can then make assertions about the added latency under different conditions. The resulting tests cover things like ensuring the router doesn't delay requests to fast backends when another backend is responding slowly, or when a backend is down.
Why RSpec?
There were a couple of reasons for choosing RSpec:
Firstly, it forces us to keep the integration tests independent from the implementation. Because the tests are in a different language, and running in a separate process, they are constrained to interacting with the exposed interfaces of the system (the HTTP endpoints and the database). This means that if we wanted to experiment with another router implementation, the existing test suite could be used with minimal modification.
Secondly, given the time constraints around this project, using a familiar set of testing tools reduced the number of things to learn. Additionally, the fact that these tests are written in Ruby will help the wider development team understand the router as they are already familiar with this style of testing. It was important that the router gained acceptance across the development team, and didn't become something that people were afraid to touch (that was one of the reasons the original Scala router wasn't put live).
Closing thoughts
Using RSpec enabled us to write some behavioural tests that were well-structured, and very readable. Go's HTTP library made it very simple to create the backends we needed. The combination of these behavioural tests, and the extensive performance testing (that's been previously covered) gave us confidence that the router would perform correctly when put live.
If work like this sounds good for you, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can sign up now for email updates from this blog or subscribe to the feed.
