Skip to content
A hands-on workshop · 10 chapters · 3 h 25 min

Notifications that arrive once, in order, on time.

Ten chapters on the engineering behind every email, push and SMS a product sends: the outbox, idempotency, retries, ordering, consent, time zones, batching, priority and webhooks. Each one has a real bug in a small notification service for you to fix.

Start chapter 01Get the lab on GitHub

Every notification takes five steps.

  1. 01

    Event

    A payment clears, someone is invited, a build fails.

    Chapters 01

  2. 02

    Decide

    Who hears about it, on which channel, and is now a good time?

    Chapters 05 to 07

  3. 03

    Render

    Turn the event into a subject, a body, a push payload.

    Chapters 08

  4. 04

    Deliver

    Hand it to an email API, an SMS gateway, a push service.

    Chapters 02 to 04, 09

  5. 05

    Learn

    The provider reports back: delivered, bounced, failed.

    Chapters 10

Each step fails in its own ways. The workshop takes them one at a time.

Ten bugs, ten fixes.

Each chapter explains one idea in depth, then hands you a piece of a notification service that gets it wrong the way real systems do. The dark line on each card is what it prints before you fix it. A checker proves your fix, and rejects the fixes that only hide the symptom.

Part 1Delivering once

4 chapters, 80 minutes

Send only what happened, never twice, and retry only what can succeed, without a storm.

  1. 0120 min

    Send after commit: the outbox

    A notification is a promise about something that happened. Send it only once that thing has really happened, and make sure it is still sent if the process dies a moment later.

    fay@example.com was told they joined ws_orbit. They did not.
  2. 0220 min

    Once, even when you retry

    A timeout does not mean the request failed. It means you do not know. Retries are only safe when the receiver can recognise a repeat, and that takes a key that is the same on every attempt.

    ada@example.com got 4 (should get 1)
  3. 0320 min

    Errors that never heal

    Retrying is right for a provider that is busy or broken, and wrong for an answer that will never change. Read the status code before you retry.

    tok_1042_old requests: 16
  4. 0420 min

    Backoff and jitter

    Exponential backoff spreads one client's retries over time. It does nothing for a thousand clients that failed at the same moment: they back off together and come back together. Jitter breaks them apart.

    busiest second after recovery: t=31 s, 1000 requests (10.0x capacity)

Part 2Delivering right

3 chapters, 60 minutes

The latest state in the right order, only to people who want it, when they are awake.

  1. 0520 min

    Out of order

    Queues reorder messages, and the clocks of different machines disagree. Order state changes by a sequence number from the system that owns the state, and drop anything older than what you have already shown.

    card says: building the truth: live
  2. 0620 min

    Preferences and consent

    A user's choice applies to every message sent after they make it, including messages you queued before. Check preferences at the moment you send, and keep required account notices out of the user's off switch.

    marketing sent after an unsubscribe: 2
  3. 0720 min

    Quiet hours and time zones

    "Not between 22:00 and 07:00" means on the user's clock. A server that checks its own clock in UTC wakes up half the world.

    made 03:05 sent 03:05 Priya commented on your pull request. <- buzzed in quiet hours

Part 3Delivering at scale

3 chapters, 65 minutes

Batch without silence, keep urgent messages fast, and trust what providers tell you.

  1. 0820 min

    Batching without silence

    A digest that waits for a quiet moment never goes out for your most active users. Every batching rule needs a maximum wait that counts from the first update, not the last.

    longest wait: 604 minutes
  2. 0920 min

    Priority lanes

    A sign-in code that waits behind six thousand announcements is useless by the time it arrives. Messages that a person is waiting for need their own lane.

    It arrives 115.0 s later: expired before it arrived
  3. 1025 min

    Webhooks you can trust

    A valid signature proves who sent a request, not when. A captured request stays valid forever unless you check its timestamp and refuse a webhook id you have already handled.

    replays accepted: 200 of 200
Chapter 04, from the lab

The retry storm, second by second.

A thousand sends fail during a 30 second outage, and every sender backs off by the textbook: 1, 2, 4 seconds and up. They all back off together, so they all come back together, ten times over capacity.

Switch on full jitter: the same backoff, with a random wait inside each window.

Retry strategy
outage025050075010000 s30 s60 s90 s120 s150 s180 scapacity: 100 a second
Requests per second, 1,000 senders, seed 7. From the lab's chapter 04 simulation. Bars over capacity are red.
Busiest second after recovery
1000
Requests in all
10,500
Last send through
543 s

Measured, not quoted.

uv run python -m notify try runs the experiments on your machine. These are the numbers from the machine the workshop was written on.

What an outbox row costs, per payment

change only, one commit535.1 µs
change and outbox row, one commit576.7 µs
change, then outbox row, two commits1183.8 µs

SQLite, WAL, synchronous FULL. The commit is the expensive part.

1,000 senders retry through a 30 s outage

strategyrequestsbusiest secondall sent by
retry every second33,0001,00034.0 s
exponential, 1 2 4 8 ... 60 s8,0001,000243.0 s
exponential, full jitter7,5164889.1 s
exponential, equal jitter7,0006860.8 s

Start in two minutes.

You need uv and git. uv installs Python for you, and the lab uses only Python's standard library: the email provider and the webhooks run on localhost.

shell
git clone https://github.com/jars-demo/notifications-demo.git
cd notifications-demo/lab
uv sync
uv run python -m notify run 1