At GDS, most of our code is publicly available in our alphagov organisation on GitHub. We call this “Coding in the Open” rather than “Open Source”. As James explained in a previous blog post this is because for most of our software, we are not in a position to offer the support that true Open Source requires.
However, we do also have some Open Source Software, in our GDS Operations organisation on GitHub. When we started building vCloud Tools, they were “Coded in the Open”, and we wanted to make them into Open Source Software. Here are the steps we took to do that.
Ensuring the project has a long term home
For the first several months of vCloud Tools’ development, the four of us working on vCloud Tools were a separate team with a separate backlog. However, once the tools were ready to be used, we wanted to bring them back into the GOV.UK Infrastructure Team. Mike Pountney and I rejoined the Infrastructure Team with the vCloud Tools backlog, and our colleagues who hadn’t been working on vCloud Tools started picking stories up.
Initially this was mostly pair programming in order to share the knowledge. A couple of people who were particularly interested took more ownership, but everyone in the Infrastructure Team contributed, including the Head of Infrastructure, Carl Massa, who merged the change that published vCloud Tools to RubyGems.
Releasing each tool as a Ruby gem
Packaging the code as a Ruby gem makes it easier for others to get started with using each tool, using a commonly installed packaging tool. It also ensures that the code follows a relatively consistent structure. Most people used to working with tools developed in Ruby will quickly know how to install it, how to use it, and where to start if they want to make modifications.
I explained as part of a previous blog post the steps we took to release each tool as a gem.
Learning from others’ experience
We already had several Open Source projects, mostly Puppet modules and our own Open Source guidelines, but we were interested in learning from others who knew about maintaining larger projects. One of the main things I learned while talking to people with experience in this area was that when running an Open Source project, you want to optimise for bringing people into your community.
Because of that, we made some changes, one of which was to add contributing guidelines with the specific aim of being helpful to new contributors. We wanted to make it easy for people who were unfamiliar with what would be involved, so we included explicit detail about how to raise a pull request and what we are looking for. You can see from the contributing guidelines on vCloud Core that we say pull requests must contain tests, but we offer to help users write them.
To make it easier for contributors and reviewers to see the quality of submissions, we also added Travis configuration so that lint and unit tests and are run against all pull requests.
Guidelines for communication with contributors
At the same time we also set some ground rules amongst ourselves, for example how quickly we should make sure we respond to pull requests. We drew up some guidelines for reviewing external pull requests. These were on our internal operations manual, but I have now added these to our public styleguide for reviewing PRs.
Putting vCloud Tools with our other Open Source projects
As I mentioned, we have other Open Source projects in our GDS Operations organisation, and once vCloud Tools had met all our Open Source guidelines, we wanted to move the tools to this organisation.
Since there are six gems and they are related, we discussed moving them to an organisation called vCloud Tools. This would have some advantages, for example it would make it easier to add core maintainers who do not work at GDS. However, after some discussion, we felt that it was important to keep the link to GDS so that people who wanted to find out what work government was doing would be able to find their way to vCloud Tools along with other projects. This is also in keeping with what other organisations who have a lot of Open Source do, for example Etsy and Netflix.
We also created a website to showcase our Open Source, with a lot more detail on vCloud Tools.
Moving issues to GitHub
Another thing that came out of early discussions with other Open Source practitioners was the importance of public issue tracking. We had features and bugs in our own issue tracking system, but having them on GitHub issues is better for users because it means they can find them. For example, if a user experiences a bug, they can search for it and may find that we’ve fixed it in a later version or at least that we are aware of it. It also has the advantage that people might fix our issues for us!
At this stage we also added a ‘newcomer friendly’ label, for issues that don’t involve an in-depth understanding of the tools and don’t require access to a vCloud Director environment. This is to offer people an easier way to get involved in contributing.
Encouraging early users
Because we had been coding in the open from the start, we already had some early users. This was really useful, because we found they picked up things we had missed. For example, people coming to the project for the first time will notice onboarding details that you may have forgotten, and can help you clarify your documentation.
Users of your software also help by finding issues that you might have overlooked. For example, one of our early users noticed a discrepancy between our documentation and our code which turned out to be a bug. This was really useful because the bug was in a feature that we don’t use on GOV.UK, so if they had not raised it, we might never have noticed it ourselves.
In order to encourage users, we visited and spoke to teams both inside and outside government. This early contact helped us make sure we were focusing on features that would be useful for more than just ourselves, and in some cases our users even added these features themselves.
Sharing our experience with others
We’ve worked closely with some of our colleagues in government, and currently teams at the Ministry of Justice, HMRC and DVLA as well as external companies are using vCloud Tools for some of their provisioning.
However, visiting individual teams wouldn’t scale for every team that’s interested, so in order to share more widely we started talking a bit more about what we were doing with vCloud Tools publicly. We wrote some blog posts, including a one by Matt Bostock which shows how you could use vCloud Tools to provision your environment. Matt and Dan Carley gave a talk at the Ministry of Justice, and I gave two talks at conferences.
What happens next?
We still have work to do. One thing we’ve found difficult is when a contributor raises a pull request for a feature that is not something we need at the moment. There is work involved in reviewing the code and testing it, and when the feature is not a priority for GOV.UK we’ve found that it’s hard to make a case for doing that work ahead of other things we need to do.
We are really keen to support our Open Source projects, and including contributors’ code even when it’s not something we will use immediately helps the tools be more robust and useful. We’ve discussed some ideas around this, and our current plan is to allot some time every week specifically for working on Open Source, so that those kinds of features will be a priority during that time. We will see how we get on and report back.
We’re also aware that a wider range of contributors leads to other considerations and what that means about things like how we license code is another post in itself. Watch this space, and as always, your comments are very welcome.
If work like this appeals to you, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can follow Anna on Twitter, sign up now for email updates from this blog or subscribe to the feed.
