Quick Scan report
Missed reminder jobs after releases
- Prepared for
- Example Health Software Ltd., 26 staff
- Application
- Clinic appointment scheduling
- Prepared by
- Emmanuel Comia, Laurentia Systems
1. Summary
Over the last six months, 7 of 28 releases to production were followed by a missed run of the nightly appointment reminder job. Each time, clients missed reminders and the team learned about it the next morning from support tickets.
All seven share one cause. The job only starts when an environment variable is set, and production gets that variable by hand rather than from the pipeline. Releases that rebuild the container remove it. Fixing the cause takes about half a day. Two further changes would make any repeat visible within minutes.
2. What was investigated
- Pipeline configuration for staging and production
- The last 28 deployments and their release notes
- Support tickets and incident notes from the same period
- Alert rules and the monitoring dashboard
- Conversations with two engineers and the support lead
3. Findings
Finding 1: production loses a setting that staging keeps
The reminder job registers itself only when REMINDER_QUEUE is set. Staging sets it in the pipeline. Production sets it by hand on the server. Releases that change the base image or dependencies rebuild the container and drop the setting. Seven of the 28 releases did exactly that, which is why the failures looked random.
02:00:00 scheduler started
02:00:00 job registry: 0 jobs registered (REMINDER_QUEUE not set)
02:00:01 scheduler idleFinding 2: nothing alerts when a job does not run
Monitoring confirms that the web application responds. It does not confirm that scheduled jobs ran. The median time from a missed run to someone noticing was 14 hours, and it was always a support ticket that raised it.
Finding 3: the release checklist is out of date
The checklist is a wiki page last edited 14 months ago. Two of its eleven steps no longer apply, and the manual variable step is not on it.
Finding 4: rollbacks are slow
Database migrations cannot be reversed, so a rollback means restoring from backup. The last one took 40 minutes.
4. Recommendations, in order
| Priority | Change | Effort | What it fixes |
|---|---|---|---|
| 1 | Move REMINDER_QUEUE into the pipeline configuration, the same way staging does it. | Half a day | The repeat incident |
| 2 | Add a heartbeat alert for each scheduled job. Page someone if a run is missed by 15 minutes. | One day | A 14-hour wait to notice becomes minutes |
| 3 | Add a post-deploy check that confirms the scheduler registered its jobs. | Half a day | Catches this class of failure before clients are affected |
| 4 | Replace the wiki checklist with steps that run in the pipeline. | Two days | Stale steps and manual changes |
| 5 | Make database migrations reversible, starting with the next schema change. | About a week, spread over several releases | 40-minute rollbacks |
5. Where to start
Do recommendations 1 and 2 this week. Together they take about a day and a half, remove the cause of the repeat incident and shorten the time to notice any similar failure from hours to minutes. Recommendations 3 to 5 can follow over the next quarter.
