Digital Marketplace Runner (‘dmrunner’)¶
Running the Digital Marketplace code locally involves running 3 backing services (
Postgres), 2 APIs (data and search), and 6 frontend apps (
user). It has a further set of mostly decentralised/loosely documented dependencies such as
AWS CLI, etc. This can make it a fairly laborious and inconsistent task for a new
developer to quickly know what to install and how to get up and running.
The dmrunner was developed during a firebreak to try to make it easier to get new developers up and running, sanity-check certain dependencies, and make it simpler to start and maintain the full suite of Digital Marketplace code.
The code for the runner can be found at: https://github.com/alphagov/digitalmarketplace-runner
This has been in use since Q2 2018 and, after the initial setup, has presented few issues for ongoing use. The initial setup can be slightly error-prone, but most of the problems should have been resolved by now.
READ THE REQUIREMENTS IN THE REPOSITORY’S README.MD
In theory, a new developer should be able to clone this repository locally (e.g. to
~/gds), install the requirements,
and then run
make setup. This should run through some pre-flight checks to validate that the correct dependencies
are available. If they aren’t, the user should be told what they need to install. It will then download all the core
Digital Marketplace code and bootstrap them by installing dependencies, setting up backing services with the data they
require (e.g. a database dump for Postgres and indexing for Elasticsearch), and terminate once everything is good to go.
The developer can then either fall back to running their code manually, or simply run
make to have the dmrunner
maintain services and apps for them.
After the dmrunner has started all of the Digital Marketplace apps, you will enter command mode, which exposes a number of shortcuts for maintaining the running processes. These can be listed by typing ‘h’ or ‘help’.
This is a summary of the main commands available:
h / help - Display this help file. s / status - Check status for your apps. b / branch - Check which branches your apps are running against. r / restart - Restart any apps that have gone down (using 'make run-app'). rb / rebuild - Rebuild and restart any failed or down apps using 'make run-all'. f / filter - Start showing logs only from specified apps* fe / frontend - Run 'make frontend-build' against specified apps* k / kill - Kill specified apps* q / quit - Terminate all running services and apps and drop back to your shell. e / env - 'delete', 'set', 'list' environment variables for apps (requires app restart for effect)
Where a command is annotated with an asterisk (*), you can provide a space-separated list of strings to match against
app names. For example, the command
kill buyer briefs search will match the buyer-frontend, briefs-frontend, and
search-api, then kill those three apps. If no match string is specified, all apps will match.
The dmrunner supports dropping into
pdb (but not
ipdb) shells. Edit your code and insert
import pdb; pdb.set_trace() and wait for the app to be reloaded, then trigger the relevant code (e.g. by visiting
the appropriate page in your browser) and wait for dmrunner to attach you to the process. While an app has a PDB shell
active, you will no longer be in dmrunner command mode and your input will instead go directly to the debugger. Use
pdb commands as you will; when the debugging session terminates (either by quitting out, or by changing
the code in the app and triggering a reload) you should drop back to dmrunner’s command mode.
It should be possible to integrate nix so that all of the apps use the
default.nix files to run.
This would remove the requirement for a number of dependencies to be installed globally on the host system
Postgres, …) and should provide a more reproducible and generally-simpler setup.
The end goal would be that users only need to clone the dmrunner repo, install
Docker for Mac,
and they’d be good to go.
The code could use some refactoring. As mentioned above this was developed in a firebreak and initially intended to be a personal utility. As more people come to rely on the code refactoring would become useful. DMRunner, in its current incarnation, is likely to require exponentially more support as more users rely on it. This means that the value in refactoring to make the code easier to understand grows at a similar rate.