Sample only. Example Health Software Ltd. is fictional, and so is every number below. A real report covers your application and your evidence.

Quick Scan report

Missed reminder jobs after releases

Prepared for
Example Health Software Ltd., 26 staff
Application
Clinic appointment scheduling
Prepared by
Emmanuel Comia, Laurentia Systems

1. Summary

Over the last six months, 7 of 28 releases to production were followed by a missed run of the nightly appointment reminder job. Each time, clients missed reminders and the team learned about it the next morning from support tickets.

All seven share one cause. The job only starts when an environment variable is set, and production gets that variable by hand rather than from the pipeline. Releases that rebuild the container remove it. Fixing the cause takes about half a day. Two further changes would make any repeat visible within minutes.

2. What was investigated

  • Pipeline configuration for staging and production
  • The last 28 deployments and their release notes
  • Support tickets and incident notes from the same period
  • Alert rules and the monitoring dashboard
  • Conversations with two engineers and the support lead

3. Findings

Finding 1: production loses a setting that staging keeps

The reminder job registers itself only when REMINDER_QUEUE is set. Staging sets it in the pipeline. Production sets it by hand on the server. Releases that change the base image or dependencies rebuild the container and drop the setting. Seven of the 28 releases did exactly that, which is why the failures looked random.

02:00:00  scheduler started
02:00:00  job registry: 0 jobs registered (REMINDER_QUEUE not set)
02:00:01  scheduler idle

Finding 2: nothing alerts when a job does not run

Monitoring confirms that the web application responds. It does not confirm that scheduled jobs ran. The median time from a missed run to someone noticing was 14 hours, and it was always a support ticket that raised it.

Finding 3: the release checklist is out of date

The checklist is a wiki page last edited 14 months ago. Two of its eleven steps no longer apply, and the manual variable step is not on it.

Finding 4: rollbacks are slow

Database migrations cannot be reversed, so a rollback means restoring from backup. The last one took 40 minutes.

4. Recommendations, in order

PriorityChangeEffortWhat it fixes
1Move REMINDER_QUEUE into the pipeline configuration, the same way staging does it.Half a dayThe repeat incident
2Add a heartbeat alert for each scheduled job. Page someone if a run is missed by 15 minutes.One dayA 14-hour wait to notice becomes minutes
3Add a post-deploy check that confirms the scheduler registered its jobs.Half a dayCatches this class of failure before clients are affected
4Replace the wiki checklist with steps that run in the pipeline.Two daysStale steps and manual changes
5Make database migrations reversible, starting with the next schema change.About a week, spread over several releases40-minute rollbacks

5. Where to start

Do recommendations 1 and 2 this week. Together they take about a day and a half, remove the cause of the repeat incident and shorten the time to notice any similar failure from hours to minutes. Recommendations 3 to 5 can follow over the next quarter.

Want a report like this for your application?

Tell me about the problem that keeps coming back. If you want help carrying out the recommendations afterwards, the Implementation Accelerator is scoped separately.

Book a call