We recently moved GOV.UK to a new and improved platform. We used vCloud Tools, open source software developed in-house at GDS, to create the virtual machines and networks required as well as configure the firewall, NAT rules and load balancers.
Why the tooling was useful
Using Infrastructure as a Service (IaaS), or the 'cloud', it is possible to define your infrastructure in code. This has a number of benefits; mainly that it enables us to consistently reproduce the same environment across different providers and provides a record of how any particular environment was configured at a given time. Using tools such as these also help to avoid human error by enabling us to automate these tasks.
The tools were particularly useful when moving GOV.UK to the new platform as they enabled us to ensure that the new platform was consistent with the old platform.
How we used the tools (and how you can, too)
The vCloud Tools meta-gem will install each of the dependent gems that vCloud Tools consist of. These are:
- vCloud Walker: 'Walks' an existing environment and outputs what it finds in JSON or YAML format.
- vCloud Net Launcher: Creates new network subnets within a vCloud Virtual Data Centre (vDC).
- vCloud Launcher: Creates new virtual machines and configures their resources including CPU, memory, IP address(es), disks and storage profiles.
- vCloud Edge Gateway: Configures vShield Edge Gateway load balancers, NAT and firewall rules.
The vCloud Tools meta-gem also installs vCloud Core, which the other tools depend on, and vCloud Tools Tester, which is a helper gem for our integration tests.
To use these tools, you'll need Ruby installed. The fastest way to download the tools is to install the vCloud Tools meta-gem by running the following command in a console:
gem install vcloud-tools
Defining the environment
Before we could start to provision the new environment, we had to define what we needed; what networks, virtual machines and firewall rules we would use.
The vCloud Walker gem was invaluable for this task as it enabled us to programmatically query the existing platform we were using, which had been provisioned using a variety of tools and some manual intervention, and output an inventory of the existing platform.
For example, to see what networks were configured on the old platform, we ran the following on the console:
FOG_CREDENTIAL=xyz vcloud-walk networks > old_platform_networks.json
You'll notice that we have also specified the
FOG_CREDENTIALenvironment variable. This is necessary so that fog, which the vCloud Tools are based upon, can authenticate with the vCloud Director API. You can find detailed instructions on how to configure your credentials in the vCloud Core README.
The output of vcloud-walk helped us to audit our existing configuration before we wrote the new configuration for use with the vCloud Tools.
vCloud Walker can output information about all of the entities in a vCloud environment, such as networks, Edge Gateway devices and machine images.
Provisioning networks with vCloud Net Launcher
The vCloud Net Launcher gem creates networks as defined in a configuration file that you specify on the command-line. As with all of the vCloud Tools, the configuration file may be YAML or JSON-formatted.
An example network configuration might look like this:
---
org_vdc_networks:
- name: Management
description:
vdc_name: GOV.UK Management
edge_gateway: GOV.UK Production
fence_mode: natRouted
gateway: 192.168.1.1
netmask: 255.255.255.0
dns1: 192.168.1.1
dns2: 192.168.1.2
dns_suffix: management.production
ip_ranges:
- start_address: 192.168.1.3
end_address: 192.168.1.254
vcloud-net-launch will only create networks if they do not already exist; running it multiple times with the same configuration will have the same result each time.
The configuration file may contain one or several networks to be created. YAML anchors are also supported to help prevent repetition of common values such as the DNS servers.
You can find example configuration files in the vcloud-net-launcher git repository.
To create the networks, assuming that your networks are specified in networks.yml, run the following in your console:
FOG_CREDENTIAL=xyz vcloud-net-launch networks.yml
vcloud-net-launch will check for each of the networks you specified in turn and create them if they do not exist.
Provisioning virtual machines with vCloud Launcher
The vCloud Launcher gem creates new virtual machines (VMs) in your vCloud environment according to a specification you provide in a configuration file. It will only create VMs if they do not already exist.
Create the virtual machines by running the following commands on the console. This example assumes that your configuration file is named 'my-vms.yml':
FOG_CREDENTIAL=xyz vcloud-launch my-vms.yml
Our configuration files for defining virtual machines look like this:
---
vapps
- name: example-1
vdc_name: GOV.UK Management
catalog: ubuntu_precise
catalog_item: ubuntu_precise
vm:
hardware_config:
memory: '2048'
cpu: '1'
network_connections:
- name: Management
ip_address: 192.168.1.100
bootstrap:
script_path: 'preamble.sh'
vars:
role: client
puppetmaster_ip: 192.168.1.3
The configuration file passed to vcloud-launch can define just one or multiple virtual machines. Currently, we group our machines into organisation vDCs such that we have one configuration file for each vDC, but you could just as easily store all of your machine definitions in one file. You can find example configuration files in the vCloud Launcher git repository.
That's it! vcloud-launch will start creating the machines and show the progress as each one is created.
Missing features in the vCloud Edge Gateway gem
The vShield Edge Gateway (vSE) that fronts the GOV.UK stack acts as a firewall but also enables us to configure virtual load balancers.
The vCloud Edge Gateway gem configures a vSE. We weren't able to use it at the the time of the migration, however, as it didn't support the ability to create our configuration from templates.
Templating is very useful to us as it means that we can define a 'common' configuration that can be shared between multiple environments, thus ensuring that the firewall configuration is consistent across the environments.
As the vCloud Edge Gateway gem did not support templating at the time, we used the vcloud-net-spinner gem that we had used previously.
Since then, we have implemented templating for vCloud Edge Gateway. Now that this feature is supported, we can use vCloud Tools exclusively when provisioning GOV.UK environments in future.
Using the vCloud Edge Gateway gem
The vCloud Edge Gateway gem supports the configuration of firewall rules, load balancers and Network Address Translation (NAT) rules.
Each of the above are defined as a 'service', and we can define one or all of these services in a configuration file that we pass to the vcloud-configure-edge command provided by the gem. For complex configurations, you may wish to define each service in a separate configuration file.
The configuration for these services vary too much to show examples of each in this blog post, but you can find example configuration files in the vCloud Edge Gateway git repository.
To configure the services defined in your configuration file on the vSE, run the following command:
FOG_CREDENTIAL=xyz vcloud-configure-edge edge-services.yml
Now that this gem supports templating, we can pass an optional parameter specifying a YAML-formatted file containing variables when we invoke vcloud-configure-edge. When we pass the optional parameter, the file specified as the first is assumed to be a Mustache template and the Mustache tags in the template are populated by the variables from the YAML file. See the 'Usage' section in the vCloud Edge Gateway README for more information.
Answering a user need
During the migration of GOV.UK, we automated the provisioning of 150 machines, 14 networks, configured dozens of firewall and NAT rules and several load balancers using these tools. We are also seeing interest (and code contributions) from other government departments and the general public.
If you're interested in knowing more about the rationale behind writing these tools, please read the blog post by my colleague Anna Shipman.
If you'd like to learn more about using any of the tools, please see the vCloud Tools README. Comments, suggestions and pull requests welcome!
If this sounds like a good place to work, take a look at Working for GDS - we're usually in search of talented people to come and join the team.
You can follow Matt on Twitter, sign up now for email updates from this blog or subscribe to the feed.
