Support Tasks

There are a number of common support requests we receive that admins cannot carry out themselves.

Support process

Support requests from CCS should be sent to the developer team via ccsrequests@digitalmarketplace.service.gov.uk. This gets forwarded to the Digital Marketplace 2nd line google group. When you receive a support request, check the following before creating a ticket on the 2nd line board:

  • Did it come in to the ccsrequests/DM 2nd line email group? If the request was sent to you individually, push back asking to resend to correct address. This is important for team visibility and tracking how many requests we receive.
  • Is the request asking for help with something admins can already do? Check the permissions for different admin roles.
  • Check whether the request is something we should be handling (see https://trello.com/c/Uji4WJ1A/1001-handling-ccs-support-requests). If you’re not sure or need to escalate a serious issue, ask a senior developer or product/delivery manager.
  • Has someone already created a ticket for this support request? Check the 2nd line board to avoid duplicates.

If the request meets the criteria above, create a new Trello card using the template, copying any relevant info from the email into the description and adding attachments if provided. Remember to tag the card ‘Support’ and include at least the subject and the ‘from’ address of the original email.

Once the investigation/change is done, reply-all to the original email (so other devs have visibility that the task has been picked up) and move the card to Done.

If you’re not sure about any of the above, ask! Talk to a Product or Delivery Manager (or your friendly senior dev).

Users

Changing a user’s name

User names cannot currently be edited by the user themselves, nor an admin. The name isn’t surfaced anywhere except audit events, CSV exports and list of supplier contributors. However sometimes we get requests to change the name:

POST `/user/<int:user_id>`
{
    "updated_by": "developers.email@digital.cabinet-office.gov.uk",
    "users": {
        "name": "Jane Doe"
    }
}

or:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.update_user(<user_id>, name="Jane Doe", updater="developers.email@digital.cabinet-office.gov.uk")

Changing a user’s email

Buyers cannot change the email address on their account. We recommend two options for buyers:

  • Create a new account with their new email address. They can still log in to their old account (e.g. to view past opportunities) as long as they know the password for it.
  • Ask their own IT department to set up email forwarding to their new email address, so that they still receive anything sent to the old address.

Suppliers can add a new email address to their account by inviting a contributor. They can remove any inactive email addresses without needing to raise a support request, however CCS admins can also invite/remove contributors if needed.

Admin accounts must be invited/removed by a developer, using the admin-manager account.

User account keeps becoming inactive

Sometimes you’ll be made aware of an account that is repeatedly becoming inactive. This is likely because when we send a Notify email to a user and the mailserver responds with an undeliverable error (also commonly referred to as a hard bounce) we deactivate the account.

This is the correct behaviour because emails, particularly those sent by Notify, are an integral part of our service.

The actual mechanism by which this is achieved is a Notify callback which POSTs to a view on the api which deactivates the user.

The method to check whether or not this is the cause of the user becoming inactive is helpfully detailed above in the Checking audit events section.

Something like the below will find this type of audit event:

  digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.find_audit_events_iter(object_id=<user_id>, object_type='users', audit_type=AuditTypes.update_user)

Removing personal data

As per the Digital Marketplace’s commitment to upholding GDPR regulations our users have the right to request that their personal data be removed from our datasets. This is detailed in our GDPR documentation.

Note

Note that their data will still exist in our backups but the removal will be propagated over 2 years. This is inline with the policy outlined in our GDPR documentation.

For all user types complete the following:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.remove_user_personal_data(<USER_ID>, 'actor.email@digital.cabinet-office.gov.uk')

Go to Mailchimp lists and click the magnifying glass icon to conduct a new search

Be sure to select ‘Contacts’ from the drop down to search for all contacts:

../_images/search.png

Select ‘View profile’ then from the ‘Actions drop down select ‘Remove’. Finally confirm ‘Permanently delete’ and confirm the action:

../_images/remove.png

Removing personal data from a supplier

If the user is a ‘supplier’ user then they may have been entered as a contact for the relevant supplier. This can be deleted by issuing the following:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.get_supplier(<SUPPLIER_ID>)
data.remove_contact_information_personal_data(<SUPPLIER_ID>, <CONTACT_ID>, 'actor.email@digital.cabinet-office.gov.uk')

If there is a request to remove all personal information from a supplier, and the supplier is not on any framework, then the steps above should be followed for all users and contact information on the supplier object.

You should also check whether the supplier has an application in progress for any frameworks that might happen to be open at the time of the request. If they have started an application you should remove the supplier declaration:

data.remove_supplier_declaration(<SUPPLIER_ID>, <FRAMEWORK_SLUG>, 'actor.email@digital.cabinet-office.gov.uk')

Updating DOS Briefs

Updating incorrect Brief award details

Buyers must supply details of their awarded contract value and start date. If they enter the wrong award details (e.g. ‘£10.000’ instead of ‘£10,000’) then they must raise a support request. A developer then needs to update the details via an API request.

Award details may also need to be manually updated if the buyer account is inactive (e.g. the user has left the buying organisation).

To update the award details:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

brief_id = 12345
brief_response_id = data.get_brief(brief_id)['briefs']['awardedBriefResponseId']
award_details = data.get_brief_response(brief_response_id)['briefResponses']['awardDetails']

# Make your changes to award_details...
award_details['awardedContractStartDate'] = '2525-01-01'
award_details['awardedContractValue'] = "20000.01"  # £20,000.01

data.update_brief_award_details(
    brief_id,
    brief_response_id,
    award_details,
    updated_by='developers.email@digital.cabinet-office.gov.uk'
)

or:

POST `/briefs/<int:brief_id>/award/<int:brief_response_id>/contract-details`
{
    "awardDetails": {
        "awardedContractStartDate": "2525-01-01",
        "awardedContractValue": "20000.01"
    },
    "updated_by": "developers.email@digital.cabinet-office.gov.uk"
}

If the buyer has not awarded the brief at all, the status on the winning BriefResponse must be changed:

POST `/briefs/<int:brief_id>/award`
{
    "briefResponseId": 12345,
    "updated_by": "your.email@digital.cabinet-office.gov.uk"
}

To mark a brief as unsuccessful or cancelled:

POST `/briefs/<int:brief_id>/<cancel|unsuccessful>`
{"updated_by": "developers.email@digital.cabinet-office.gov.uk"}

Make sure to check the public brief page to confirm the correct outcome is displayed.

Making changes to a Brief

Warning

You should seek explicit CCS DOS Category approval to perform any of the following actions

Briefs are designed not to be changed once published as that may constitute a breach of procurement regulations. Changing a brief part-way through a procurement could unfairly benefit one supplier over another.

However a brief is not a binding contract to actually exchange money or sign a contract (commonly referred to as a call-off contract).

A buyer may decide they no longer want whatever it was they were procuring so we allow buyers to:

  • withdraw a brief while it is live and suppliers are making bids
  • cancel a brief once its tender period has ended

These states signify that the buyer will not be signing a contract based on the outcome of the brief.

Sometimes a supplier needs more information than the buyer provided on the original brief. Because we can’t allow the buyer to change the brief, and because we don’t want to unfairly benefit one supplier over another by giving them more information than everyone else, this information is given by the clarification question process.

The supplier can ask a question about the information they think should have been included in the brief, and the buyer publishes a response to that question openly on the brief. This supplementary information is provided transparently and all suppliers are notified that a question has been answered and supplementary information is available.

This all means that it is very rare that a developer should have to withdraw or remove a brief but is has been discussed as part of threat modelling scenarios.

  • A buyer wants to legitimately withdraw an opportunity but has lost access to their email address
  • A malicious buyer account posts defamatory information
  • A buyer accidentally posts a brief containing security-breaching data or personally identifiable information

Withdrawing

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.withdraw_brief(<BRIEF_ID>, 'actor.email@digital.cabinet-office.gov.uk')

Removing

Briefs cannot be deleted, or unpublished, because brief responses may be orphaned as a result.

Removing data from public view would have to be done directly in the production database. Publicly visible brief details are stored in the briefs table data JSON column. The current accepted wisdom would be to overwrite the offending field of this data blob with a string such as '<REMOVED>'

You should then create a new briefs index and redirect the briefs-digital-outcomes-and-specialist alias to your new index.

Finally, extract a list of suppliers who have responded to the brief and ask CCS to message them concerning the removal.

Supplier queries

Updating Supplier DUNS numbers

A supplier’s DUNS number should only be done if CCS have raised a specific request, giving the supplier ID(s).

To update a DUNS number for one supplier, first check that the new number isn’t already in use:

GET `/suppliers?duns_number=123456789`

or:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.find_suppliers(duns_number=<DUNS_NUMBER>)

Update the DUNS number:

POST `/suppliers/<int:supplier_id>`
{
    "updated_by": "developers.email@digital.cabinet-office.gov.uk",
    "suppliers": {
        "dunsNumber": "123456789"   # Must be a string, not an integer
    }
}

or:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.update_supplier(<SUPPLIER_ID>, {'dunsNumber': '<DUNS_NUMBER>'}, "developers.email@digital.cabinet-office.gov.uk")

Any declarations for open applications will be updated, but any other declarations won’t.

To swap DUNS numbers for two suppliers, run oneoff/swap-supplier-duns.py from the root of the scripts repo:

scripts/oneoff/swap-supplier-duns.py <SUPPLIER_ID_1> <SUPPLIER_ID_2> --updated-by=<YOUR_WORK_EMAIL> --stage=production

The two keyword options are not optional and must to be included.

Novations

This is when one supplier takes over another supplier, and wants to sell their services.

See Novations for more details.

Regenerating/replacing a supplier’s framework agreement

Note

If you are unfamiliar with framework agreements and how they work, see the Framework Agreements manual page for detailed info.

Suppliers often have issues uploading their signed framework agreement pages:

  • the supplier may have made a mistake when signing the PDF itself
  • the supplier may have made a mistake in the accompanying signer details (signer name/date/role)
  • the supplier may have had their services suspended for not returning their signed agreement before the cutoff date
  • the framework agreement PDF may have the wrong details (e.g. registered company number)

For most cases the supplier can simply start the upload process from scratch, by logging into their account and visiting /suppliers/frameworks/<framework_slug>/agreement. This process will create a new FrameworkAgreement object in the database for that supplier; the old version are kept for audit purposes.

If the supplier details on the agreement PDF are wrong, the CCS Data Controller will need to correct the supplier data in the admin. Once this is done, a new agreement PDF will need to be regenerated by a developer and uploaded to S3.

  • Use the scripts/framework-applications/generate-framework-agreement-signature-pages.py script, with the supplier ID in a text file.
  • Upload the PDF to the supplier’s folder in the digitalmarketplace-agreements-<stage>-<environment> S3 bucket.

Adding or updating a Modern Slavery statement

Once framework applications have closed, suppliers can’t edit their Modern Slavery statement themselves, so they will have to raise a support ticket. It’s easy enough for a developer to change:

To upload the document:

  • CCS Support should provide the supplier name, ID and the new document
  • Check that the document is under 5MB and is either Open Document Format (ODF) or PDF/A (eg .pdf, .odt)
  • Rename the file if necessary, using a URL-friendly name
  • In the AWS console switch role to admins in the digitalmarketplace-production account
  • Upload the new document to the documents S3 bucket for that supplier and framework at digitalmarketplace-documents-<stage>-<environment>/<framework_slug>/documents/<supplier_id>/ and grant read-only access to the public/everyone.

The framework declaration also needs to be updated using the api:

  • Get the supplier declaration object using GET /suppliers/<int:supplier_id>/framworks/<framework_slug>/declration or data.get_supplier_declaration(<supplier_id>, <framework_slug>).

  • Look at the supplier declaration object to see whether they have a modernSlaveryStatement or modernSlaveryStatementOptional key. If they don’t have an existing modern slavery statement key then the key should be modernSlaveryStatementOptional.

  • Update or add the modern slavery statment URL to the supplier’s framework declaration, using the correct key. The URL should use the ‘assets’ domain (https://assets.digitalmarketplace.service.gov.uk), not the S3 path.

    For example:

    PATCH `/suppliers/<int:supplier_id>/frameworks/<framework_slug>/declaration`
    {
        "updated_by": "developers.email@digital.cabinet-office.gov.uk",
        "declaration": {
            "<key>": "https://assets.digitalmarketplace.service.gov.uk/<framework_slug>/documents/<int:supplier_id>/<filename>"
        }
    }
    

    or:

    digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview
    
    data.update_supplier_declaration(
      <supplier_id>,
      <framework_slug>,
      {"<key>": "https://assets.digitalmarketplace.service.gov.uk/<framework_slug>/documents/<int:supplier_id>/<filename>"},
      user="developers.email@digital.cabinet-office.gov.uk"
    )
    
  • Check one of the supplier’s G-Cloud services to make sure the new document is visible.

  • For DOS suppliers, statements are not publicly available, however the support team may request them, so we should update them anyway.

Adding or updating a service document

Suppliers can replace their own Service Definition Document once the framework is live, but they cannot remove it completely. Occasionally it’s necessary to remove the Service Definition Document (e.g. following a novation), however from G-Cloud 12 onwards the document is mandatory, so a replacement must be provided.

To remove a Service Definition Document:

data.update_service(<service_id>, {'serviceDefinitionDocumentURL': None}, 'developer.email@digital.cabinet-office.gov.uk')

To update a service document, ensure the new file is present in the production S3 documents bucket. The filename should be in the format <service_id>-<document-type>-<datestamp>.pdf, for example 12345678901-pricing-document-2020-01-01.pdf.

Then update the service with the full https://assets.digitalmarketplace.service.gov.uk/… path:

data.update_service(<service_id>, {'serviceDefinitionDocumentURL': 'https://assets...'}, 'developer.email@digital.cabinet-office.gov.uk')
data.update_service(<service_id>, {'sfiaRateDocumentURL': 'https://assets...'}, 'developer.email@digital.cabinet-office.gov.uk')
data.update_service(<service_id>, {'termsAndConditionsDocumentURL': 'https://assets...'}, 'developer.email@digital.cabinet-office.gov.uk')
data.update_service(<service_id>, {'pricingDocumentURL': 'https://assets...'}, 'developer.email@digital.cabinet-office.gov.uk')

Change supplier details on DOS service CSVs

When buyers start a DOS opportunity on a particular lot, they have access to a CSV download of supplier services for that lot. These CSVs are generated by a developer when the framework opens and uploaded to a public S3 bucket (see more info on this process at making DOS services live).

Sometimes suppliers need to change the details of their services during a framework, for example adding a new specialist role or lowering day rates. These service changes can be made by a CCS Category admin, however they are not automatically reflected in the public CSVs available to buyers.

The DOS service export scripts take a long time to run, and need ‘cleaning’ before they’re suitable for publishing to a public bucket to remove any sensitive information. For ad hoc change requests it’s quicker and easier to manually edit the CSV instead. Make sure to follow the same upload process on S3, remembering to:

  • update the ‘Last updated’ line at the top of the CSV
  • follow the existing file name pattern
  • set the Content-Disposition header to attachment
  • set the Cache-Control header to max-age=3600
  • make the file readable to the public

Supplier querying their DOS brief submission

It is important to suppliers that their brief responses are received by the buyer but there is some evidence that suppliers can neglect to either complete their response or to hit submit before the deadline. In these cases we often recieve support requests from suppliers, via CCS, who want to find out if there was any platform instability that could have impacted the submission of their brief.

Digital Marketplace has robust Monitoring, Alerting and Logging and we use this to investigate every error returned from the platform. The case of user not being able to submit a brief would constitute a P1-P3 incident in line with our incident grading scale detailed in the Incidents section of this manual. This is significant because the steps taken in managing such incidents include proactively contacting CCS Support to notify them of impact to users and degradation of service.

What this all means is that if a user is served an error from a Digital Marketplace server whilst submitting a brief CCS would be aware of it.

If you have received a support request from CCS asking whether Digital Marketplace has experienced any instability that could affect the submission of brief responses or if you can investigate whether any users were served errors whilst submitting a brief response please reply with the standardised response on the template card on the Digital Marketplace 2ndLine Trello Board.

In the case of such instability it is possible to check whether:

  • the user made a request to submit their brief response
  • whether that request actioned the submission of their brief response

You will need the supplier id, brief id, access to credentials and the scripts repo and access to Kibana:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

supplier_id=<SUPPLIER_ID>
brief_id=<BRIEF_ID>

from dmapiclient.audit import AuditTypes

brief_response_id = data.find_brief_responses(
    supplier_id= supplier_id,
    brief_id=brief_id,
    status='draft,awarded,pending-awarded,submitted'
)['briefResponses'][0]['id']

audits = data.find_audit_events(
    object_type='brief-responses',
    object_id=brief_response_id,
    data_supplier_id= supplier_id,
    audit_type=AuditTypes.submit_brief_response
)['auditEvents']

print('"/brief-responses/' + str(brief_response_id) + '/submit"')
print(audits)

The above will print a string containing the brief_response_id of the brief response (like "/brief-responses/12345/submit"). You can then search for this in Kibana which will tell you if we received a request from the user to submit their brief response.

It will also print whether we have an audit event record of the request having actioned the submission of their brief response.

Note

If you want to look up the suppliers updates to the brief response you can change the audit_type passed to find_audit_events to AuditTypes.update_brief_response.

General investigation tips

Checking logs

Our platform logs are available in Kibana for 30 days. For logs going further back, use AWS Cloudwatch. See the Logging and Monitoring manual page for more information.

Notify email logs are stored for 7 days. See the Email integration manual page for more information.

Checking audit events

Occasionally we’ll need to get a list of supplier actions from the audit events, (for example, to check whether a supplier completed part of their framework application).

One way to do this is via the API client. The easiest way to initialise it is via scripts/api-clients-shell.py in the scripts repo. This will initialise the DataAPIClient in a Python shell, fetching the credentials from your local environment rather than having to copy and paste the key into the shell.

For example, to find all the update draft service events for supplier ID 12345 (using the API client AuditTypes):

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

> audits = data.find_audit_events_iter(data_supplier_id=12345, audit_type=AuditTypes.update_draft_service)
> for audit in audits:
>    print(audit)

{'acknowledged': False, 'createdAt': '2019-01-23T00:12:23.456789Z', 'data': {'draftId': 654321, 'serviceId': None, 'supplierId': 12345, 'updateJson': {'serviceManagerPriceMax': '890'}}, 'id': 5123456, 'links': {'self': 'https://api.digitalmarketplace.service.gov.uk/audit-events'}, 'objectId': 123456, 'objectType': 'DraftService', 'type': 'update_draft_service', 'user': '12345@example.com'}
{'acknowledged': False, 'createdAt': '2019-02-23T00:12:23.456789Z', 'data': {'draftId': 654321, 'serviceId': None, 'supplierId': 12345, 'updateJson': {'technicalArchitectLocations': ['Offsite']}}, 'id': 5123457, 'links': {'self': 'https://api.digitalmarketplace.service.gov.uk/audit-events'}, 'objectId': 123456, 'objectType': 'DraftService', 'type': 'update_draft_service', 'user': '12345@example.com'}

Make sure to use find_audit_events_iter(), otherwise only the first page of audit events will be returned.

Other useful API client filters:

> client.find_audit_events_iter(object_type="suppliers", object_id=12345)
> client.find_audit_events_iter(user="12345@example.com")

Note that the list of queryable object types is found in the API repo’s audits.py view.

Requests for sharing users information or exporting data

Ensure first that any requests for user data meets our privacy policy.

You can find all buyers under a certain email domain by running:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

> domain = 'my.domain.com'
> users = [u['emailAddress'] for u in data.find_users_iter(role='buyer') if u['active'] and u['emailAddress'].split('@')[-1] == domain]
> print(users)