On this page

Operations and Monitoring#

DevStride is monitored with AWS's own tools, chiefly Amazon CloudWatch, plus checks of our own built into the product and the release process. Alarms are defined in code with everything else, so every new function, queue and scheduled job gets them automatically.

Alarms#

On every function. Every Lambda function has an alarm that fires when a run reaches 80% of its time limit. This catches work that is slowing down before it starts failing. Production also has account-wide alarms on function errors and throttling.

On every scheduled job. Every scheduled job has three alarms:

  • it failed;
  • it missed a run, because nothing ran within twice its interval;
  • its schedule could not deliver the trigger to it.

On every queue. Every dead-letter queue, where messages that keep failing end up, has two alarms. One fires on how many messages are waiting. The other fires on new arrivals, so a fresh failure is noticed even while older ones are still being looked at.

Custom signals. The product reports more than twenty measures of its own health, each with an alarm. Examples include messages being retried over and over, real-time updates being refused, an email or Teams self-test missing its run, and clean-up jobs stalling.

Security. GuardDuty findings, root-account use, and security changes made outside the deploy pipeline all raise alerts. See Security and Identity.

Where alerts go#

  • Alarms are posted to an engineering alerts channel in Slack and sent by email.
  • In production, a second email path with no application code in it acts as a backup, so an alert still arrives if the application's own email sender is down.
  • An alarm that stays on is re-announced every few hours until it is dealt with.
  • An alarm also watches the alerting itself, and fires if no alert has reached Slack in a day.
  • Billing alarms and cost alerts go to their own channel and a daily digest.
  • A failed deploy opens a GitHub issue.

Logs, metrics and tracing#

  • Logs. Every function writes structured logs to CloudWatch Logs, kept for one year. API access logs are kept for one year too.
  • Metrics. Functions publish custom metrics through AWS Lambda Powertools.
  • Error dashboard. A CloudWatch dashboard shows errors across the system, and an hourly digest summarizes them.
  • Tracing. AWS X-Ray is available on every function. It is switched on for a function while a problem is being diagnosed, rather than all the time.

Canaries#

Self-tests that run on a schedule against the live system:

  • Microsoft Teams, hourly: checks every link in notification delivery to Teams.
  • Inbound email, every ten minutes: sends a real email through the public mail route to a test mailbox and checks that it became a Service Desk ticket. A daily reconciliation in production also checks that every email received was processed.
  • Customer sending domains, hourly: rechecks the domains customers send from.

After every deploy#

Every production release ends with a read-only health check, which must pass before the next release can go out. It checks that:

  • the code running in production is exactly the code that was merged;
  • no alarm has gone off since the deploy;
  • no dead-letter queue has grown;
  • inbound email is healthy, from mail routing through to the canary;
  • every file of the web app reached hosting;
  • every background consumer starts;
  • the Teams canary passes on every link.

A failure goes to a person to decide whether to roll back. A check that could not run is reported as not run, never as a pass.

Incident response#

When an alarm fires, the on-call process is:

  1. Gather the evidence: the alarm, the dead-letter queues, the logs and recent deploys.
  2. Name a single root cause, keeping what is proven apart from what is suspected.
  3. Take only read-only steps without asking. Any step that changes production, such as re-sending messages, editing data or deploying, needs explicit approval first.
  4. Record the alarm's signature and its fix in a playbook, so the same problem is resolved faster next time.

What is not used#

There is no third-party monitoring or tracing service (such as Datadog, Sentry or Lumigo) in the system. Earlier ones were removed in favour of the AWS-native setup above. There is currently no browser-side error tracking.

Powered by DevStride