Oliver Spindler, Tech Lead for front end development of the Performance Platform, explains how www.gov.uk/performance developed an innovative JavaScript workflow to allow performance data to be visible to everyone, while those on high-end browsers could have more powerful interactions.
GOV.UK Performance provides a growing number of dashboards for government services. These dashboards track and show how well services are doing. This is vital information that helps those providing the services to improve further. The dashboards present data in many ways, including tables and interactive charts.
The application which renders these dashboards, Spotlight, is a pure JavaScript application. With a few exceptions, the app reuses the same code in both server and client. This setup provides us with great flexibility combined with the least code redundancy. We can minimise the development effort needed to deliver a service that provides a good experience for all users.
This post explains why we developed Spotlight.
A bit of history
We have been developing GOV.UK Performance since early 2013. Originally, we developed the frontend as a Ruby on Rails project. Rails is an integral part of the GOV.UK software stack, which allowed us to deploy the Performance Platform easily. That rails app acted as the main client of our backend data API.
We combined this with a large client-side JavaScript layer responsible for creating all the data visualisations. The JavaScript framework queried our data API directly, bypassing the Rails server completely. Doing a lot of the heavy lifting on the client-side meant that we only had to build one way of retrieving and manipulating data and being able to provide interactive visualisations using the D3.js library.
Unfortunately, this approach meant that you did not have access to our content if you:
- used an older browser
- had JavaScript disabled
- had additional accessibility requirements
That may have been okay for a prototype but as we started to roll out dashboards for real users it wasn't good enough. We were breaking:
- a fundamental principle of good software for the web – that content should be available to as many people as possible
- our own design principles
Worse still, our primary users – service managers, people responsible for the success of the services - are often primarily using legacy browsers and so could not access all our content. This meant that we had to rethink how we made our data available to the world.
We had learned more about the needs of our users, and how we should meet them. But we did not want to:
- lose the flexibility of the existing solution
- increase our development effort by developing a completely separate server-side layer
Given the investment we had already made in our client-side JavaScript we decided to explore using it on the server too.
Fight fire with fire
We set up a new frontend application, Spotlight. This is a pure JavaScript application running on the Node.js platform. The new app inherited the existing client-side JavaScript components. Additionally, Spotlight executes the same code on the server. This lets us generate static versions of our data visualisations on the server. After the browser has rendered the static page, the same components runs client-side to make the visualisation interactive.
The code is built on top of the Backbone.js library and follows an MVC-like structure. We call our core abstraction a module, with each module providing a specific type of visualisation. A collection of modules makes up a dashboard. The typical rendering process for a Spotlight page looks like this:
First, find out which modules constitute the requested dashboard and instantiate each module. Next each module retrieves the required data from the data backend and renders the visualisation.
The event-driven nature of Node.js has proven a great match with this approach. Once all modules have finished rendering, the server can send the page back to the client.
The page delivered to the client provides a basic, static experience, but aims to be as useful as possible. We use progressive enhancement to enhance the experience on the client. If JavaScript is available, we use the same modules as the server. On the client, the modules can behave differently to the server, depending on the requirements of the modules. For instance, a module can choose to register event handlers that provide interactivity to the user.
An extra benefit of this structure is that modules are standalone things in their own right. This gives us more flexibility in the ways we visualise data. For instance, it's now much easier for us to show a summary version of a module on a dashboard and then link to a page where the module expands out to show the concept in question in greater detail. This is something we're looking to build on over the coming months.
Lessons learned
A few minor modifications were necessary to make the existing client-side code compatible with Node.js.
- First, our JavaScript codebase uses the AMD convention while Node.js uses the CommonJS module format. However, the Node.js version of the RequireJS library allows you to use AMD code in Node.js without any modifications.
- Second, our rendering code assumes a browser to be available to build up DOM elements. While Node.js does not provide a DOM implementation out of the box, the jsdom fills this gap very well.
Almost all our code works on both server and client, and we ensure this by running our Jasmine-based test suite in Node.js and PhantomJS as part of our continuous integration process. There are only few places where we had to create explicit client or server specific branches. The biggest difference between client and server code is that we need to react to user interaction on the client. On the server, we register these event handlers regardless but they simply do not get triggered.
Using the same code on both client and server side is still an emerging approach. It seemed promising enough to be worth some effort and we were pleasantly surprised by how straightforward it was to achieve the transition. We released our first Spotlight-based dashboard in November and are converting our other dashboards. We're pleased with our experience so far and would encourage others to consider this approach.
Visit GOV.UK Performance, especially the dashboard for Carer's allowance to see Spotlight in action. Like most of the software created by GDS, the project is publicly available under an MIT licence.
You can read more about principles like progressive enhancement and accessibility in the GOV.UK Service Manual.
Oliver Spindler has now left GDS but kindly wrote the above in order to document and explain the work he did whilst he was here.
If this sounds like a good place to work, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can follow Oliver on Twitter, sign up now for email updates from this blog or subscribe to the feed.
5 comments
Comment by David Read posted on
Where can I download the full data? I see you have web tables, but I was thinking something more suitable for analysis, like CSV, covering a longer time period than your graphs show.
Comment by James Abley posted on
It's great to hear that people want that data. We will be writing more here and on the GDS Data blog when we have a published API that we will support. Currently it's still an alpha project, and how we expose full data sets is a question that we haven't addressed yet.
Comment by David Read posted on
It's excellent that you've committed to publishing your data properly. I'm also pleased to see an API is required as key part of the software architecture:
https://www.gov.uk/service-manual/making-software/apis.html
Comment by Jae Task posted on
James, I am working on another government project and have been asked to justify using backbone. I have experience of it at the BBC and can see exactly how it would benefit us, Did you have to do a key design document, to justify its use? If so, would you mind sharing? It would save me having to go through the process myself.
Many thanks
Comment by James Abley posted on
Jae, we didn't have to create any document to justify our technology decisions. How we choose technology is described in the service manual. We started with some pretty standard outcomes that we wanted, which included:
extensible, maintainable code
people who knew or were comfortable picking up Backbone
We're very happy with our choice, which works for us in our context.