Other teams around government frequently ask us about our use of GitHub, and rather than bury those answers in email it seemed time to follow Mike Bracken’s directive and publish (not send) our answers.
I’ve made some assumptions here about readers’ familiarity with a few concepts about version control and deployment. Hopefully between the post, some links and the comments we’ll answer many of the more frequent questions. For a deeper look at git check out the freely available book Pro Git.
Version control and collaboration
The Government Service Design Manual has a section on version control. An early draft of that talked about git and GitHub specifically, but we revised it to focus on the important things:
- You should be able to track fine-grained changes across your codebase so that you can see the history and step back to known good versions. We believe this is vital for building high quality software
- The team should have tools to collaborate on code, to discuss changes and to review what’s happening. This too leads to higher quality code, both because the more eyes we get on the code the more likely we are to spot errors and because it provides a clear way for everyone to review the context of past changes.
For GDS teams making it easy to code in the open is also a very high priority, work is only ever merged into master rather than committed directly (we work in feature branches), and we release from master as often as we can to keep big bang releases to a minimum.
The version control system that best suited the way we worked at GDS was git. Most of our developers were already familiar with it, it makes working in branches straightforward, and the distributed model provides a lot of flexibility. Whenever we’ve looked at the options, the set of tools provided by GitHub have been the best match we’ve yet found for the way we want to work and layer nicely on top of git.
Thanks to that layering, one of our most important assets (the code) can be easily moved if we wanted, and there are lots of options for applying other tools where necessary. That means that “we can change our minds”. We’re very keen to see how other options (for example bitbucket, gitlab and gitorious) develop and it’s good to see some other teams around government experimenting -- I spotted some DVLA teams using gitlab when I visited the other day.
Collaboration
Each GDS team has a slightly different process for reviewing code, dependent on what they’re working on, and who’s in the team at a given time. The really important things for us are that:
- every change gets at least two pairs of eyes on it before it hits master
- we can support continuous deployment and rapid bug fixes
Most teams have adopted a practice of using short-lived feature branches and generating a pull request when that’s ready to merge (or just when you want wider input).
Pull requests are merged by someone who didn’t work directly on the code and they should have reviewed the code and run the tests. Many teams are using Travis CI to have tests run automatically on their pull requests, giving an extra layer of confidence.
Keeping secrets
As with every aspect of how we work it’s really important that we consider the risks associated with how we make changes and manage our code. The detailed risks vary from system to system, but every project will need to know exactly who has made changes to the code before we deploy it and how we’re striking the balance between being open and handing the keys to our system over to anyone who looks.
We work very hard to keep detailed configuration out of our code, with specific configuration repositories holding any details (such as credentials) that need to be provided to apps when they’re deployed. Anyone who’s making a deployment is expected to know details of what’s changed (based on git revisions).
We initially managed those configuration repositories as private repositories on github.com but as our work has become higher profile and more critical we decided to bring those entirely in-house. To support that we have a Github Enterprise installation running in a controlled environment--we could have used other git tools here, but we decided to prioritise a consistent experience for our teams. Anything that contains something we can’t make public lives there. There’s an extra hop to access it if you’re outside the office network but otherwise the day-to-day experience is identical.
Optimise for change
We’ve optimised our ways of working for momentum and the flexibility to change. We’ve even enshrined the principle behind that in the Digital By Default Service Standard:
14. Make sure that you have the capacity and technical flexibility to update and improve the service on a very frequent basis.
https://www.gov.uk/service-manual/digital-by-default#criterion-14
Having a workflow that supports collaboration is a big part of that, but it’s also meant that we need to think about how we might keep the ability to change in less usual circumstances, such as when we need to make a change very quietly (for security reasons) or to deploy an important update when GitHub.com is unavailable.
That’s the other place where the distributed nature of git helps us out. We can easily keep up to date copies of our code in both places and deploy from either. A more sensitive change can be made on the internal clone of the repository, deployed, and then later merged into the public clone.
A diversity of approaches
The way we’re set up works really well for our development teams and we’re keen to see how other teams around government are working (and of course to see how the wider community’s practices develop).
Whether decisions are affected by the preferences of a team, the risks around the work they’re doing, or the emergence of new tools, there are going to be plenty of chances to learn and improve.
If this sounds like a good place to work, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can follow James on twitter, sign up now for email updates from this blog or subscribe to the feed.
2 comments
Comment by Gareth Rees posted on
Do the private repos just store YAML / JSON files with the credentials, or something different? Are they dynamically pulled in to environment configuration like Heroku https://discussion.heroku.com/t/how-does-heroku-persist-config-across-builds/410?
Comment by James Stewart posted on
It's a bit of a mixed bag at the moment. Some repos work that way and we try and push a lot into environment variables. The heroku link is useful as we need to work out a few details to simplify how we do this.
There are also some areas where we will add specific rails initializers on deployment. Most of that could be replaced by environment variables but at the moment those are ruby files in the deployment repo.
Others are things like the core puppet code that glues together the individual modules. We try and open up as many of the modules as we can but they're tied together with private code. Tools like puppet-librarian make that much easier than it used to be.