Our web applications and services are hosted on the GOV.UK PaaS. The GOV.UK PaaS is a customised CloudFoundry deployment hosted on AWS open for use by government development teams.
Developers should generally NEVER have to do the things detailed here on a day-to-day basis. Our deploy pipeline is handled by our pipeline release jobs in Jenkins and you should use those. The details here are for information and extreme emergencies only.
The GOV.UK PaaS documentation explains the steps required to set up and login to the Cloud Foundry CLI.
Environment for experiments (remember to clean up after yourself!)
Environment for apps which monitor other apps
Environment for user research
Environments for different stages
Setting up the target organisation and space:
cf target -o digitalmarketplace -s <space>
All Digital Marketplace apps run in Docker containers on the GOV.UK PaaS. Each application has an image tagged with each release. Images can be found on Docker Hub.
The digitalmarketplace-aws repo has a Makefile with rules for generating manifests for apps, and then pushing them to the GOV.UK PaaS. The most important being:
make deploy-app APPLICATION_NAME=<docker repository> RELEASE_NAME=<docker tag>
This is the easiest way to deploy an application. It pulls in variables from
the credentials repo to populate the manifest template, and takes
RELEASE_NAME variables to make sure the correct
Docker image is used.
Application manifests describe the application instances, hosts, launch commands and environment variables used. Secret env variables are pulled in from the credentials repo ‘vars’ directory when the manifest is generated.
The manifest also configures the routing for the applications within GOV.UK
PaaS (e.g. requests with the
/suppliers prefix are routed to the
supplier-frontend app). The AWS repo has the full list of routes.
A healthcheck url for each app is defined in the base manifest, by appending
_status?ignore-dependencies to the first item in the app’s
Head to the ‘Debugging PaaS App Instances’ section of the manual for tips on debugging app instances on GOV.UK PaaS.
We currently have services for our database,
cdn_router. Creating new services is only required when setting up a new
space or creating a separate environment/instance of the API. For example:
cf create-service postgres "M-dedicated-9.5" <database_name>
The created database name can then be used in the application manifest services list.
cf services will display a list of all services in the current
space, and which apps have a binding to them. For example:
+---------------------------+------------------+--------------+------------+ | name | service | plan | bound apps | +---------------------------+------------------+--------------+------------+ | search_api_elasticsearch | elasticsearch | small-ha-5.x | search-api | +---------------------------+------------------+--------------+------------+
The credentials for connecting to GOV.UK PaaS services are automatically
provided in the app environment
cf env <application_name> (i.e. these are
not in our credentials repo). The credentials are regenerated when a new
instance is created. If you need to recycle the credentials following a leak,
you must destroy and rebuild the service. First you must unbind any apps that
use the service, and delete any service keys. Then create the new service
instance, rebind the apps and re-add the service keys.
For example, to destroy/create a new elasticsearch instance:
cf unbind-service <application_name> <service_name> cf delete-service-keys <service_name> <key_name> cf delete-service <service_name> cf create-service elasticsearch small-ha-5.x <service_name> cf add-service-key <service_name> <key_name> cf bind-service <application_name> <service_name>
You may need to redeploy the application via Jenkins, in order to use the service’s new environment variables.
cf delete-service and
cf create-service commands will take several
minutes each. To avoid downtime, you may need to create the new service with a
different name beforehand, bind the app to it, then delete the old service.
If there were previously any custom settings for the service (e.g. our Elasticsearch instances have a larger memory allocation than the default) these will have to be re-applied via a request to GOV.UK PaaS support.
If destroying/rebuilding Elasticsearch, remember to recreate the indexes, aliases and populate the data for both services and briefs!
None of the main Digital Marketplace apps use buildpacks. This should only be used for other apps, such as the prototype.
To deploy an application using an existing manifest,
cd into the
application directory and run:
cf push -f <path_to_manifest> <application_name>
If your directory names match the repository names then you can use
$(basename $(pwd)) as the application name to always deploy
current directory to the corresponding application.
When started without <application_name> CloudFoundry will deploy the current directory to all applications in the manifest. If this happens, you need to redeploy each application separately from the correct application directory.
Cloud Foundry will push the contents of the current directory as the new
application version. This isn’t tied to files tracked by git, so any
uncommitted changes or files ignored by git will be pushed (this usually
Make sure there are no secrets or confidential information (eg production logs, API tokens or database dumps) in the current working directory before pushing
Our PaaS apps can be scaled horizontally (increasing the number of instances) or vertically (increasing the memory or disk space).
During a deployment, we will briefly have twice as many apps as normal running. When scaling ensure that 50% of our PaaS quota is still free. Otherwise you will break deployments.
To scale up or down:
vars/<environment>.ymlfile in the digitalmarketplace-aws repo with the number of instances, memory and disk space for each environment.
Create a pull request and explain the change in the description. Get it approved and merged (see this example PR for scaling G-Cloud 12).
Alert the team in the
Run the Jenkins Re-release all apps job for the right environment. This uses the currently deployed images so you don’t have to go through the full pipeline.
If you urgently need to increase the number of instances, this can be done with a simple CloudFoundry command:
cf scale <application_name> -i 10
You will still need to commit and merge the changes in the AWS repo afterwards.
Only use the CloudFoundry command for increasing the number of instances. If scaling down apps, or increasing/decreasing memory or disk space, this must be done via a re-release.