Digital Marketplace Runner (‘dmrunner’)

Running the Digital Marketplace code locally involves running 3 backing services (nginx, Elasticsearch and Postgres), 2 APIs (data and search), and 6 frontend apps (admin, briefs, brief-responses, buyer, supplier, user). It has a further set of mostly decentralised/loosely documented dependencies such as Node, NPM, AWS CLI, etc. This can make it a fairly laborious and inconsistent task for a new developer to quickly know what to install and how to get up and running.

The dmrunner was developed during a firebreak to try to make it easier to get new developers up and running, sanity-check certain dependencies, and make it simpler to start and maintain the full suite of Digital Marketplace code.

Repository

The code for the runner can be found at: https://github.com/alphagov/digitalmarketplace-runner

This has been in use since Q2 2018 and, after the initial setup, has presented few issues for ongoing use. The initial setup can be slightly error-prone, but most of the problems should have been resolved by now.

Setup

READ THE REQUIREMENTS IN THE REPOSITORY’S README.MD

In theory, a new developer should be able to clone this repository locally (e.g. to ~/gds), install the requirements, and then run make setup. This should run through some pre-flight checks to validate that the correct dependencies are available. If they aren’t, the user should be told what they need to install. It will then download all the core Digital Marketplace code and bootstrap them by installing dependencies, setting up backing services with the data they require (e.g. a database dump for Postgres and indexing for Elasticsearch), and terminate once everything is good to go. The developer can then either fall back to running their code manually, or simply run make to have the dmrunner maintain services and apps for them.

Commands

After the dmrunner has started all of the Digital Marketplace apps, you will enter command mode, which exposes a number of shortcuts for maintaining the running processes. These can be listed by typing ‘h’ or ‘help’.

This is a summary of the main commands available:

 h /     help - Display this help file.
 s /   status - Check status for your apps.
 b /   branch - Check which branches your apps are running against.
 r /  restart - Restart any apps that have gone down (using 'make run-app').
rb /  rebuild - Rebuild and restart any failed or down apps using 'make run-all'.
 f /   filter - Start showing logs only from specified apps*
fe / frontend - Run 'make frontend-build' against specified apps*
 k /     kill - Kill specified apps*
 q /     quit - Terminate all running services and apps and drop back to your shell.
 e /      env - 'delete', 'set', 'list' environment variables for apps (requires app restart for effect)

Where a command is annotated with an asterisk (*), you can provide a space-separated list of strings to match against app names. For example, the command kill buyer briefs search will match the buyer-frontend, briefs-frontend, and search-api, then kill those three apps. If no match string is specified, all apps will match.

Debugging

The dmrunner supports dropping into pdb (but not ipdb) shells. Edit your code and insert import pdb; pdb.set_trace() and wait for the app to be reloaded, then trigger the relevant code (e.g. by visiting the appropriate page in your browser) and wait for dmrunner to attach you to the process. While an app has a PDB shell active, you will no longer be in dmrunner command mode and your input will instead go directly to the debugger. Use normal pdb commands as you will; when the debugging session terminates (either by quitting out, or by changing the code in the app and triggering a reload) you should drop back to dmrunner’s command mode.

Possible improvements

It should be possible to integrate nix so that all of the apps use the default.nix files to run. This would remove the requirement for a number of dependencies to be installed globally on the host system (e.g. Postgres, …) and should provide a more reproducible and generally-simpler setup. The end goal would be that users only need to clone the dmrunner repo, install Nix, Docker for Mac, and they’d be good to go.

The code could use some refactoring. As mentioned above this was developed in a firebreak and initially intended to be a personal utility. As more people come to rely on the code refactoring would become useful. DMRunner, in its current incarnation, is likely to require exponentially more support as more users rely on it. This means that the value in refactoring to make the code easier to understand grows at a similar rate.