- Maintenance Mode
Sometimes we need planned downtime, for example when doing a database restore.
The goal is to stop users accessing the site, preventing them from seeing incorrect information or making changes that may be lost.
To do this we put the Digital Marketplace into ‘maintenance mode’. In this mode, any visitor to the Digital Marketplace will be served a static html page informing them of this down time.
Maintenance mode was first used in 2017 when we moved our underlying infrastructure from ElasticBeanstalk to the PaaS.
When maintenance mode is active, no-one is able to access our applications via their normal domains (preview.marketplace.team, staging.marketplace.team, and digitalmarketplace.service.gov.uk). Users see a static page served by the router app.
We also have a ‘recovery’ mode that allows developers access to the API apps, while still blocking user access to the Frontend apps. Recovery mode allows scripts such as the Jenkins re-indexing jobs to access the APIs in the normal way.
Developers can also bypass the router, accessing the apps via the Cloud Foundry/PaaS ‘internal’ domain,
Note that the frontend applications are hosted at
dm-<stage>.cloudapps.digital with their specific path
suffixes, and are protected by an extra level of Basic Auth which will need to be supplied when accessed
through this method (see the credentials repo).
There are two ways to activate maintenance mode:
- With the Jenkins job
Toggle maintenance mode
- Manually deploying a new version of the PaaS router application.
This is the preferred way to enable or disable maintenance mode and should be used in all cases where our infrastructure is running under normal conditions.
- Start a new build of the pipeline job
Toggle maintenance modeon
ci.marketplace.team. Select the appropriate target stage (preview, staging, production) and mode (maintenance, recovery or live).
- Once the build has started, a new Pull Request will be generated against the
digitalmarketplace-awsrepository. Review, approve, and merge this to master.
- After it has been merged to master, continue the Pipeline job in Jenkins by clicking ‘Proceed’ on the two input boxes on the current Pipeline stage.
You should only need to manually deploy maintenance mode if some part of our infrastructure and deployment pipeline has failed, for example, Jenkins or Github are down.
Update the target stage’s variables file that is used in our PaaS manifests -
https://github.com/alphagov/digitalmarketplace-aws/blob/master/vars/<preview|staging|production>.yml. To turn maintenance mode on, set
You will need access to the PaaS space you wish to deploy to (likely production). You will need to deploy a new dockerised app. To deploy the router, run:
STAGE=production APPLICATION_NAME=router RELEASE_NAME=<release_tag> make deploy-app
release_tagis of the format
release-###and can be discovered by using
cf app routerand looking at the currently-deployed app’s docker image, which looks something like:
docker image: digitalmarketplace/router:release-18
The release should only take a few minutes. After the release has completed, maintenance mode will be enabled.
DM_MODE on the manifest will pass it as an environment variable to the router app.
There are three modes:
DM_MODE = 'maintenance': healthcheck and metrics endpoints for the router only, all other routes will be directed to a static maintenance page (served by the router) and return a 503 status code.
DM_MODE = 'recovery': healthcheck and metrics endpoints for the router only, plus API apps. All other routes will be directed to a static maintenance page.
DM_MODE = 'live': all apps served as normal
This is achieved by changing which config files are loaded by the router based on the
DM_MODE environment variable.
See the nginx start up script in the router app for details.