API keys for the Data API, Search API and Antivirus API are stored in the digitalmarketplace-credentials repository. Our applications (both API and frontend), other Digital Marketplace components (eg scripts), and developers use these keys to talk to the APIs.
api: auth_tokens: - A-first-token-is-used-by-other-applications - J-second-token-is-used-by-jenkins - D-third-token-is-be-used-by-developers
In the event that a key is leaked then it should be rotated. If a developer leaves the Digital Marketplace, then
production tokens (at least) should be rotated. A Jenkins job called
Rotate API tokens exists to generate new tokens
and push them out to our applications, following the correct process for a seamless transition. We have three tokens
for each API:
An application token that starts with ‘A’
A Jenkins token that starts with ‘J’
A developer token that starts with ‘D’
The Jenkins job is a Pipeline that has five main stages:
The pipeline will generate a new Pull Request against the credentials repository to add three new tokens each for the Data API and Search API. A developer will need to manually merge this.
Go back to the pipeline and proceed through the next two dialogue boxes to deploy the Search API, then deploy the Data API, and then deploy all other apps simultaneously.
After all of the apps have released and the pipeline is paused at the ‘Update Jenkins config’ stage, a developer needs to update Jenkins config. To do this, prepare Jenkins for shutdown (Manage Jenkins -> Prepare for Shutdown) to stop any new jobs from being started. Wait for existing jobs to finish running and notify developers on #dm-release that Jenkins will be restarted shortly, then run
make jenkins TAGS=configfrom the
digitalmarketplace-jenkinsrepository to push the updated tokens for Jenkins to inject into the environment for scripts/etc. You must have DM_CREDENTIALS_REPO defined and it must point at a version of
digitalmarketplace-credentialsthat contains the newly-added tokens. After Jenkins has restarted, continue the Pipeline by clicking ‘Proceed’ twice.
Pipeline stage 4 will generate another Pull Request against the
digitalmarketplace-credentialsrepository to remove the old tokens from the Data API and the Search API. A developer will need to manually merge this.
Finally, continue the pipeline again by clicking ‘proceed’ twice to deploy the Search API, then the Data API, then all other apps (apps don’t access the Antivirus API directly, so it’s unimportant when the Antivirus API is restarted).
Jenkins has its own Docker Hub account which it uses to publish Docker images that it builds. This account is protected by two-factor authentication, and is accessed by Jenkins using a Docker Hub personal access token.
The account token is kept in the digitalmarketplace-credentials repo, in the file
jenkins-vars/docker_credentials_env.enc, as an environment variable called
This is a manual process during which releases/merges should be paused.
Post a message in the #dm-release Slack channel warning the team not to merge or release.
Using the details for the Jenkins user in the credentials file
pass/docker.com/jenkins-ci, log in to the Docker Hub console. To do this you will need to reset the password (generate a new one with your choice of reputable token generator).
Create a new access token following the instructions in the Docker manual. Make sure to give the token a memorable name, like “jenkins (new)”. Copy the token into the credentials file
Delete the old access token and rename the new access token (which should now be the only access token) to “jenkins”.
Log out of Docker Hub, and throw the account password away.
Commit the new token to digitalmarketplace-credentials and open a pull request. Get the PR reviewed and merged.
update-credentialsjob in the Jenkins console.
SSH into Jenkins and delete the
/var/lib/jenkins/.dockerfolder to remove the auth token.
Still in the Jenkins box, log in with the new token:
sudo -u jenkins docker login
Test with a Docker image job (e.g. run the Refresh Docker base images job).
Give the all clear in the #dm-release Slack channel.
We should normally only have two “Live” Notify API keys at a time: one for the (production) applications and one for Jenkins. These are the only ones it’s particularly useful to rotate: “Test” and “Team and Whitelist” tokens are relatively powerless.
- Create two new “Live” Notify API keys using the Notify dashboard.
One named in the format
APPLICATION-PRODUCTION-<date>. Use this key to replace the
notify_api_keyvariable in the digitalmarketplace-credentials repo’s
One named in the format
JENKINS-LIVE-<date>. Use this key to replace the
jenkins_env_variables.NOTIFY_API_FUNCTIONAL_TESTS_KEY_PRODUCTIONvariables in the digitalmarketplace-credentials repo’s
Commit these changes and get the resulting PR approved and merged.
Ensuring you have this latest credentials revision checked out in your local credentials repository, prepare Jenkins for shutdown (Manage Jenkins > Prepare for Shutdown), Wait for existing jobs to finish running and notify developers on
#dm-releasethat Jenkins will be restarted shortly, then run
make jenkins TAGS=configfrom the
Run the “Re-release all apps” jenkins job against
productionto ensure all the apps have picked up the new notify key.
It’s probably best to take a look at the “Message log” in the Notify dashboard (under “API integration”) and check you can see the new keys in use (and that you can’t see the old keys still being in use).
Delete the old keys in the Notify dashboard.
The apps hosted on PaaS use an AWS Access Key to connect with S3 buckets and other AWS resources. There is an access key for each AWS account (development and production).
To rotate an access key, do the following steps for each account:
In the AWS console, switch role (to either
adminrole) for the relevant account.
Create new key for the
paas-appuser for that account in AWS console: IAM > Users >
paas-app> Security Credentials > Access Keys Each user can have a max of 2 keys.
Add the new
aws_secret_access_keyto the credentials repo in
vars/<stage>.yml. Commit, review and merge the PR.
Re-release all apps via Jenkins to inject the new
AWS_SECRET_ACCESS_KEYenvironment vars into the PaaS apps.
Try accessing a restricted document on Digital Marketplace for the relevant stage(s). For example, log in as a supplier and download/upload a service document.
In the AWS console, check that the new key has been used (and that the old key has not been used since the release). This information may take up to 5 minutes to appear in the AWS console. If it doesn’t appear, check if logs for the stage are appearing in CloudWatch (the PaaS apps use the AWS access key to send logs to CloudWatch). If there are no logs in CloudWatch, the key is probably invalid/misconfigured and you will need to start again.
Once you’re confident the new key is being used, make the old key inactive in the AWS console (this is reversible and it doesn’t need to be deleted straight away).