Webhooks in Production: Six Ways They Break, and How to Handle Each One

Receiving a webhook looks like the easiest thing in backend engineering. A provider POSTs JSON to a URL you own, you read the body, you update a row. Twenty lines. Ship it.

Then you run it for six months against real traffic and you learn what you actually signed up for.

A webhook is not an HTTP request. It's a promise made by someone else's infrastructure to eventually tell you something happened. Every part of that sentence is load-bearing. Someone else's — you don't control the retries. Eventually — there is no deadline. Tell you — one-way, no handshake, no confirmation that your understanding matches theirs.

And a promise can be broken. Six ways, specifically:

It will arrive twice → idempotency
It may be forged → verification
It may never arrive → reconciliation
It may arrive out of order → don't trust sequence
It may arrive late → don't assume freshness
Your handler may be slow → acknowledge fast, process after

Almost every webhook tutorial covers #2 and stops. This post takes one section each, with code from three providers I've actually shipped against: Safaricom's Daraja (M-Pesa), Paystack, and Meta's Cloud API for WhatsApp. They solve the same problem three genuinely different ways, and comparing them teaches you more than any one of them alone.

The examples are Node.js and Postgres. The reasoning is language-agnostic — I've written this same handler in PHP, and the shape didn't change.

1. It will arrive twice

Start here, because it's the failure that costs money.

Every serious provider retries. Paystack sends the event every 3 minutes for the first 4 attempts, then hourly for the next 72 hours, until it gets a 200. Meta retries immediately and then "a few more times with decreasing frequency" over 36 hours before dropping the update. Safaricom redelivers STK Push callbacks it doesn't consider acknowledged.

Retries are not a bug in their system. They're the only way a one-way message can be made reliable at all. Which means the duplicate is your problem, by design.

And retries aren't the only source. A network blip after your handler committed but before your 200 reached them produces a duplicate. A load balancer timeout produces a duplicate. Meta explicitly tells you to handle deduplication and warns that batching cannot be guaranteed. You will get the same event twice even when nothing is broken.

So the question that decides whether your integration is correct is not does it work. It's: what happens on the second delivery?

Two layers of defence

Layer one: an event ledger. Every webhook you accept gets a row, and the database enforces uniqueness.

CREATE TABLE webhook_events (
  id           BIGSERIAL PRIMARY KEY,
  provider     TEXT        NOT NULL,
  event_id     TEXT        NOT NULL,
  event_type   TEXT        NOT NULL,
  payload      JSONB       NOT NULL,
  received_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at TIMESTAMPTZ NULL,
  attempts     INT         NOT NULL DEFAULT 0,
  last_error   TEXT        NULL,
  UNIQUE (provider, event_id)
);

Claiming an event is a single statement:

async function claimEvent(db, provider, eventId, eventType, payload) {
  const { rows } = await db.query(
    `INSERT INTO webhook_events (provider, event_id, event_type, payload)
     VALUES ($1, $2, $3, $4)
     ON CONFLICT (provider, event_id) DO NOTHING
     RETURNING id`,
    [provider, eventId, eventType, payload]
  );
  return rows[0]?.id ?? null;   // null means we've seen this one before
}

ON CONFLICT DO NOTHING ... RETURNING is doing something subtle and important: it is a check and a write in one statement. Two workers processing the same retry 50ms apart both try to insert; exactly one gets a row back. The other gets null and stops.

Never write this as a SELECT followed by an INSERT. Between the read and the write there is a gap, and both workers will pass the check. The database is the only component both requests share, so the database has to arbitrate the race. Never check and then act — make the check part of the write.

Layer two: make the effect itself idempotent. The ledger protects you from replays of the same event ID. It does nothing about two different event IDs describing the same real-world fact — a charge.success webhook and your reconciler both discovering the same payment, for example.

So the state transition gets the same treatment:

const { rowCount } = await db.query(
  `UPDATE payments
      SET status = 'completed', receipt = $1, completed_at = now()
    WHERE reference = $2
      AND status = 'pending'`,
  [receipt, reference]
);

if (rowCount === 0) return;   // someone else already completed it — stop
await creditWallet(reference);

AND status = 'pending' is the whole trick. One row changed means you won the transition and you're the one who does the side effect. Zero means someone else got there first, and your job is to do nothing quietly.

I wrote this pattern up in more depth, in PHP, in How to Integrate M-Pesa STK Push with PHP — including the unique constraint on the receipt number that makes double-crediting physically impossible at the storage layer, whatever your application code does.

When the provider gives you no event ID

Stripe gives you evt_.... Paystack gives you an event name and a data.reference. Safaricom gives you nothing explicitly labelled an event identifier, and Meta's WhatsApp payloads don't carry one either.

So derive one, deterministically, from the fields that define the event:

const EVENT_ID_STRATEGIES = {
  paystack: (b) => `${b.event}:${b.data.reference}`,
  mpesa:    (b) => `${b.Body.stkCallback.CheckoutRequestID}:${b.Body.stkCallback.ResultCode}`,
  whatsapp: (b) => {
    const c = b.entry[0].changes[0].value;
    const m = c.messages?.[0], s = c.statuses?.[0];
    return m ? `msg:${m.id}` : `status:${s.id}:${s.status}`;
  },
};

// Used by the handler, the backfill and the tests — one definition, so a change
// to how an ID is derived can never desynchronise two callers.
function deriveEventId(provider, body) {
  const strategy = EVENT_ID_STRATEGIES[provider];
  if (!strategy) throw new Error(`No event-ID strategy for provider: ${provider}`);
  return strategy(body);   // the ledger's UNIQUE (provider, event_id) already scopes it
}

A last-resort fallback is a SHA-256 of the raw body — but only where the provider sends byte-identical retries. It's a weaker key than a real identifier, because any timestamp or trace field in the payload will change between attempts and defeat it. Prefer a derived composite key over a body hash whenever the payload has stable identifying fields.

2. It may be forged

Your webhook URL is public. It has to be — the provider needs to reach it from the open internet. Which means anyone who discovers it can POST whatever they like to it.

This is the failure mode idempotency cannot help you with. A forged callback is the first delivery. It wins the conditional update legitimately. Every safety net from section 1 works perfectly, and marks the attacker's unpaid order as paid.

The three providers solve this three different ways, and it's worth seeing all three side by side.

Paystack: HMAC over the raw body

Paystack signs the payload with your secret key using HMAC-SHA512 and sends it in x-paystack-signature.

const crypto = require("crypto");

// express.raw — NOT express.json. See below; this matters more than it looks.
app.post("/webhooks/paystack",
  express.raw({ type: "application/json" }),
  (req, res, next) => {
    const expected = crypto
      .createHmac("sha512", process.env.PAYSTACK_SECRET_KEY)
      .update(req.body)                    // Buffer: the exact bytes received
      .digest("hex");

    const provided = req.get("x-paystack-signature") || "";

    // timingSafeEqual throws if the lengths differ, so check length first.
    if (provided.length !== expected.length ||
        !crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(provided))) {
      console.warn("Rejected Paystack webhook: bad signature", { ip: req.ip });
      return res.sendStatus(401);
    }

    req.event = JSON.parse(req.body.toString("utf8"));  // parse AFTER verifying
    next();
  },
  handlePaystackEvent
);

Two details that bite people:

Use timingSafeEqual, not ===. String comparison short-circuits on the first differing byte, so response latency leaks how much of the signature you got right. That's enough to recover a valid signature one byte at a time. Constant-time comparison removes the signal. (Same reasoning as hash_equals in PHP.)

Verify the raw bytes, then parse. If you run express.json() first and re-serialise req.body to compute the HMAC, you are signing a different string: key order can shift, whitespace is gone, non-ASCII characters may be escaped differently. Best case, every signature fails and you spend an afternoon confused. Worst case, you get it passing for ASCII payloads and it breaks the first time a customer's name contains an accent. Verify what arrived, act on what you verified.

Meta: a verify token and a signature

Meta splits the problem in two, and people routinely conflate the halves.

The verify token is a one-time subscription handshake. When you register the endpoint, Meta sends a GET and you echo the challenge back:

app.get("/webhooks/whatsapp", (req, res) => {
  const mode      = req.query["hub.mode"];
  const token     = req.query["hub.verify_token"];
  const challenge = req.query["hub.challenge"];

  if (mode === "subscribe" && token === process.env.META_VERIFY_TOKEN) {
    return res.status(200).send(challenge);
  }
  res.sendStatus(403);
});

That proves, once, that whoever configured the subscription controls this URL. It proves nothing about any subsequent POST. The verify token never appears again after the handshake. If your mental model is "I checked the verify token, so my endpoint is authenticated," your endpoint is wide open.

Per-request authenticity comes from X-Hub-Signature-256, an HMAC-SHA256 over the payload keyed with your App Secret, formatted as sha256=<hex>:

function verifyMetaSignature(req) {
  const header = req.get("x-hub-signature-256") || "";
  const expected = "sha256=" + crypto
    .createHmac("sha256", process.env.META_APP_SECRET)
    .update(req.body)                    // raw Buffer again
    .digest("hex");

  return header.length === expected.length &&
         crypto.timingSafeEqual(Buffer.from(header), Buffer.from(expected));
}

Note the App Secret, not the access token. And note that this is a different key from the verify token — two secrets, two jobs. Getting that distinction wrong is the same category of mistake as the one I wrote about in Authentication vs Authorization: building the correct check and putting it at the wrong layer. Verifying the subscription is not verifying the request, exactly as filtering a UI list is not authorising the endpoint underneath it.

M-Pesa: no signature at all

Safaricom's Daraja API doesn't sign STK Push callbacks. There's no HMAC, no shared secret, no signature header. The callback arrives as plain JSON with nothing attaching it to Safaricom.

So you invent the missing mechanism. Register the callback URL with an unguessable path segment and compare it in constant time:

https://api.yourdomain.com/webhooks/mpesa/9f2c1a7e4b8d6f0c3a5e7b9d1f4c6a8e

app.post("/webhooks/mpesa/:token", express.json(), (req, res) => {
  const expected = process.env.MPESA_CALLBACK_SECRET;
  const provided = req.params.token || "";
  if (provided.length !== expected.length ||
      !crypto.timingSafeEqual(Buffer.from(provided), Buffer.from(expected))) {
    console.warn("Rejected M-Pesa callback: bad path secret", { ip: req.ip });
    return res.sendStatus(404);   // 404, not 401 — don't confirm the route exists
  }
  // ...
});

A URL secret is weaker than an HMAC. It's a bearer credential: it travels on every request, it lands in access logs and proxy logs, and it doesn't bind to the payload at all. So you pair it with a second control that doesn't depend on it — never trust the amount in the callback:

if (Number(amountPaid) !== Number(payment.amount)) {
  console.error("Amount mismatch", { reference, expected: payment.amount, got: amountPaid });
  await flagForReview(reference);
  return;                        // record it, do NOT fulfil
}

Your database is the source of truth for what was owed. The callback only tells you what happened. That rule is worth applying to every provider, signed or not.

You can additionally allowlist provider IPs at the edge — Paystack publishes 52.31.139.75, 52.49.173.169 and 52.214.14.220, and Safaricom publishes ranges too. Treat it as defence in depth, never as your primary control. Ranges change without much warning, and the failure mode is a silent outage.

3. It may never arrive

This is the section almost nobody writes, and it's the one that separates a tutorial from a system that's been run.

Everything so far protects you from processing an event twice. None of it protects your customer from the opposite failure: the thing happened, and you never heard about it.

The callback is dropped in transit. Your server is redeploying at that exact moment. Your host has a blip. Your handler throws before it commits, returns a 500 for 36 hours of retries, and then the provider gives up and drops the event permanently. The customer's money moved regardless.

And here's what makes it genuinely dangerous: a webhook that never arrives generates no error. No exception, no failed request, no alert. Your logs look perfect. The only trace is a row that sits in pending forever and a person who is increasingly annoyed.

You cannot fix this inside the request. There is no request. The fix is a second, independent path to the same truth — one that pulls instead of waiting to be pushed.

Layer one: sweep your own pending state

You know what you're waiting for. Anything you initiated that hasn't reached a terminal state after a reasonable delay is a candidate.

// Runs every 5 minutes.
async function sweepPendingPayments(db) {
  const { rows } = await db.query(
    `SELECT id, reference, provider
       FROM payments
      WHERE status = 'pending'
        AND created_at <  now() - interval '5 minutes'   -- give the webhook a chance
        AND created_at >  now() - interval '7 days'      -- still queryable upstream
      ORDER BY created_at
      LIMIT 100`
  );

  for (const p of rows) {
    try {
      const truth = await providers[p.provider].fetchStatus(p.reference);
      // The same completion function the webhook handler calls. Not a copy of it.
      await applyPaymentResult(db, p.reference, truth);
    } catch (err) {
      console.error("Reconcile failed", { reference: p.reference, err: err.message });
      // Leave it pending. Next run tries again.
    }
  }
}

Three rules this encodes:

Reuse the webhook's completion logic — don't reimplement it. Two code paths that can both complete a payment will drift, and the bug will surface in whichever one you test less. One function, two callers.

Because completion is idempotent, the sweep is free. If the webhook lands while the reconciler is mid-flight, one of them wins the conditional update from section 1 and the other does nothing. That is the entire payoff of getting idempotency right first: you can retry as aggressively as you like without fear.

Never guess. A payment you couldn't verify stays pending and gets retried. It does not become failed because your API call timed out.

For Paystack that lookup is GET /transaction/verify/:reference. For M-Pesa it's the STK Push Query endpoint (/mpesa/stkpushquery/v1/query) — I walk through that reconciler in full in the M-Pesa post.

Layer two: pull the event stream you missed

The sweep above only finds gaps in things you started. It won't find events you never had a local row for — a refund raised from the provider's dashboard, a chargeback, a subscription that renewed on their schedule.

For that, walk the provider's event or transaction list from a stored cursor:

async function backfillFromProvider(db, provider) {
  // Upsert, not SELECT: on the very first run for a provider there is no cursor
  // row yet, and a bare SELECT would hand you undefined. Seed it and lock it in
  // one statement so two concurrent backfills cannot both claim the window.
  const { rows: [cur] } = await db.query(
    `INSERT INTO webhook_cursors (provider, last_event_at)
     VALUES ($1, now() - interval '1 hour')
     ON CONFLICT (provider) DO UPDATE SET provider = EXCLUDED.provider
     RETURNING last_event_at`,
    [provider]
  );

  // Overlap the window deliberately — idempotency makes re-seeing events harmless,
  // and a gap at the boundary would be permanent.
  const since = new Date(cur.last_event_at.getTime() - 10 * 60_000);

  for await (const event of providers[provider].listEvents({ since })) {
    const id = deriveEventId(provider, event);
    if (await claimEvent(db, provider, id, event.type, event)) {
      await processEvent(db, provider, event);      // never seen it — handle it now
    }
  }

  await db.query(
    `UPDATE webhook_cursors SET last_event_at = now() WHERE provider = $1`, [provider]
  );
}

The deliberate overlap is the point. Re-processing an event is free — claimEvent returns null and the loop moves on. Missing one at the window boundary is permanent. Given an asymmetry like that, always overlap.

When there is no replay API

Be honest about the ceiling here. Stripe exposes GET /v1/events. Paystack lets you list and verify transactions. Meta gives you nothing — there is no endpoint that replays WhatsApp messages you failed to receive. If your endpoint was down when a customer messaged you, that message is gone, and no reconciler will recover it.

Which changes the engineering, and it's worth saying plainly: when the provider has no replay API, availability of your endpoint is a correctness requirement, not a reliability nice-to-have. That's the strongest argument for section 6's design — store the raw event and acknowledge in milliseconds, so the window in which you can lose one is as small as you can make it.

Layer three: audit for drift

Sweeps catch stuck rows. They don't catch the case where your data is confidently wrong — a status you set from a payload that has since changed, or a webhook you processed with a bug you fixed last week.

Once a day, compare aggregates. Your completed payment total for yesterday against the provider's settlement report. Your active subscription count against theirs. You are not looking for individual rows; you are looking for a number that doesn't match, which tells you a class of event is being handled wrong.

Alert on silence

The last piece is noticing. Every one of these gets a threshold:

Payments in pending for more than 30 minutes — counted and alerted on.
webhook_events rows with processed_at IS NULL older than 10 minutes.
Time since the last webhook of each type. If charge.success volume drops to zero for an hour on a Tuesday afternoon, something is broken — your endpoint, their delivery, or your DNS. Nothing throws an exception when webhooks simply stop, so this has to be a metric, not a log line.

That last one has told me a webhook endpoint was down more often than any error rate ever has. It's the same argument I made in Monitoring and Observability: the absence of an event is a signal, and you only see it if you're measuring for it.

4. It may arrive out of order

Two events, generated two seconds apart, sent by different workers, over different connections, one of which hit a retry. They can reach you in either order. Meta says so directly — batching is not guaranteed. Stripe says it doesn't guarantee delivery in the order events are generated. Nobody guarantees ordering, because over an unordered network with independent retries, nobody can.

So the failure looks like this: a WhatsApp message goes sent → delivered → read. The read status is delivered first time. The delivered status needs a retry and lands ninety seconds later. Your handler writes each status as it arrives, and a message the customer has already read now shows as delivered. Your UI is wrong, and it's wrong permanently, because nothing else will ever come to correct it.

Don't order the events. Make the transitions monotonic. Rank your states and refuse to move backwards:

const STATUS_RANK = { sent: 1, delivered: 2, read: 3, failed: 3 };

await db.query(
  `UPDATE messages
      SET status = $1, status_rank = $2, status_at = $3
    WHERE wa_message_id = $4
      AND status_rank < $2`,     // only ever move forward
  [status, STATUS_RANK[status], new Date(Number(timestamp) * 1000), messageId]
);

The late delivered runs, matches nothing (2 < 3 is false), changes zero rows, and disappears. No branching, no ordering buffer, no timestamps to compare — the same conditional-update shape from section 1, applied to a different problem.

read and failed deliberately share rank 3. They are both terminal and mutually exclusive — a message that failed was never read — so whichever arrives first is the truth, and a strict < makes the other a no-op. Give two states the same rank only when you mean first one wins, permanently; if a later state should ever be able to overwrite an earlier one, it needs a higher rank.

When your states don't form a clean ranking, use the provider's own timestamp as the guard instead:

UPDATE subscriptions
   SET status = $1, provider_updated_at = $2
 WHERE id = $3
   AND provider_updated_at < $2      -- ignore anything older than what we have

Two rules follow from this:

Terminal states are terminal. Once a payment is completed or refunded, no later-arriving event downgrades it. Encode that in the WHERE clause, not in a code review comment.

Never infer sequence from arrival. "We got the cancellation, so the renewal must have come first" is an assumption the network never agreed to. If you need the order, read it from the payload's timestamps or sequence numbers — and if the payload doesn't carry one, that's a design constraint to work around, not something you can wish away.

5. It may arrive late

Late is not the same as out of order. Out of order is about two events relative to each other. Late is about one event relative to now.

Two distinct problems come out of it.

The payload is a snapshot, not the current state

An event describes the world at the moment it was generated. By the time you handle it — after a 36-hour retry window, or after your queue drained a backlog — that description may be historical fiction.

A customer cancels, then re-subscribes eleven minutes later. Both webhooks queue behind an incident. You process subscription.cancelled, faithfully, on a customer who is currently paying you.

The rule: the webhook tells you to look; it doesn't tell you what you'll find. For anything consequential — revoking access, shipping goods, moving money — treat the event as a trigger and re-fetch the object:

async function handleSubscriptionEvent(db, event) {
  // Do not trust event.data.status. Ask what is true right now.
  const live = await paystack.fetchSubscription(event.data.subscription_code);

  await db.query(
    `UPDATE subscriptions
        SET status = $1, synced_at = now()
      WHERE code = $2
        AND synced_at < $3`,
    [live.status, live.subscription_code, new Date(event.receivedAt)]
  );
}

You trade an extra API call for correctness under delay. On low-volume, high-value events that is not a close call. On high-volume, low-stakes events — a message-delivered receipt — trusting the payload is fine, because the cost of being briefly wrong is nearly zero. Choose per event type, deliberately.

A late copy of a real event is a replay attack

An attacker who captures one legitimate webhook — from a log, a proxy, a misconfigured staging endpoint — can POST those exact bytes back at you next week. The signature still verifies. It was, after all, genuinely signed.

Stripe solves this by putting a timestamp inside the signed material: Stripe-Signature: t=1699...,v1=<hmac>, where the HMAC covers timestamp + "." + body. You recompute over both and reject anything outside a tolerance — five minutes is the documented default. Tampering with the timestamp breaks the signature, so age becomes unforgeable.

Paystack, Meta and M-Pesa do not do this. None of their signatures cover a timestamp. Which means your defence against replay is the event ledger from section 1, and that has a consequence people miss when they get around to housekeeping:

If you prune webhook_events faster than an old event can be usefully replayed, you have reopened the replay hole — and quietly, because nothing fails.

Keep event IDs well beyond the provider's retry window. 72 hours for Paystack, 36 for Meta, so 30 days is a comfortable floor and storage is cheap. If you must prune harder than that, archive the IDs to a cheaper table rather than deleting them, and keep the uniqueness check pointed at both.

You can also reject payloads whose own timestamp is implausibly old, where they carry one. It's weaker — the timestamp isn't signed, so a determined attacker edits it — but it's free, and it catches lazy replays:

const ageMs = Date.now() - Number(value.messages[0].timestamp) * 1000;
if (ageMs > 24 * 60 * 60_000) {
  console.warn("Discarding implausibly old WhatsApp payload", { ageMs });
  return res.sendStatus(200);   // 200 — don't make them retry a thing we reject
}

6. Your handler may be slow

The last failure mode is the one you cause yourself.

The provider is holding a connection open waiting for your 200. Paystack documents a 30-second timeout per attempt in test mode; treat that as your budget everywhere. If your handler charges a card, generates a PDF, sends three emails and calls two internal services before responding, you will eventually exceed it.

And the consequence is worse than a slow request. The provider times out, marks the delivery failed, and retries — while your original handler is still running and about to finish successfully. You've built a machine that generates duplicates in proportion to how slow you are. Under load, when you're slowest, it generates the most.

The fix is to separate acknowledgement from processing, and the order of operations is the whole design:

app.post("/webhooks/paystack",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    // 1. Verify FIRST. Never let an unauthenticated request reach your queue.
    if (!verifyPaystackSignature(req)) return res.sendStatus(401);

    const body    = JSON.parse(req.body.toString("utf8"));
    const eventId = deriveEventId("paystack", body);

    // 2. Persist durably. This is the only thing that must happen inline.
    const rowId = await claimEvent(db, "paystack", eventId, body.event, body);

    // 3. Acknowledge. We are now responsible for this event.
    res.sendStatus(200);

    // 4. Process out of band. A crash here is recoverable — the row is on disk.
    if (rowId) await queue.publish("webhook.process", { rowId });
  }
);

Four things about that ordering, all learned the hard way:

Verify before you enqueue. Reverse those and anyone on the internet can flood your job queue with unauthenticated work. The queue is inside your trust boundary; the endpoint is not. Same principle as every other input in API Security Best Practices — validate at the edge, not three layers in.

Persist before you acknowledge. A 200 is a promise that you have the event. If you respond first and then crash before writing anything, the provider believes you succeeded and never retries. That event is gone forever, and with Meta there's no replay API to recover it. The insert is a single indexed write — a millisecond or two — and it is the difference between "we can recover" and "we cannot."

Acknowledge before you process. Everything after res.sendStatus(200) is on your own time. The queue worker can be slow, retry with backoff, and fail loudly without generating a single duplicate delivery.

On serverless, "after the response" is a lie. On Vercel, Lambda, Cloud Functions and friends, the runtime may freeze or terminate your function the moment the response is sent. Work started after res.send() is not guaranteed to run — and this fails intermittently, under exactly the load where you'll least enjoy debugging it. Either await the enqueue before responding (a real queue write, not the work itself), or use your platform's explicit background-work primitive. Never fire-and-forget.

The worker then does the real thing, and marks the row done:

async function processWebhookRow(db, rowId) {
  const { rows: [evt] } = await db.query(
    `SELECT * FROM webhook_events WHERE id = $1 AND processed_at IS NULL`, [rowId]
  );
  if (!evt) return;                                 // already handled

  try {
    await handlers[evt.provider][evt.event_type]?.(evt.payload);
    await db.query(`UPDATE webhook_events SET processed_at = now() WHERE id = $1`, [rowId]);
  } catch (err) {
    await db.query(
      `UPDATE webhook_events SET attempts = attempts + 1, last_error = $2 WHERE id = $1`,
      [rowId, err.message]
    );
    throw err;                                      // let the queue retry it
  }
}

Note the processed_at IS NULL in the SELECT. And note that an unknown event_type is not an error — store it, ignore it, move on. Providers add event types without asking you.

Three providers, one problem

Here's the whole comparison in one place. Same problem, three answers — and the differences tell you exactly how much work each one leaves to you.

	M-Pesa (Daraja)	Paystack	Meta (Cloud API)
Authenticity	None provided — you invent it	HMAC-SHA512	HMAC-SHA256 + one-time handshake
Header / carrier	Secret path segment (your design)	`x-paystack-signature`	`X-Hub-Signature-256: sha256=…`
Key	Your own secret	Your Paystack secret key	Your App Secret (not the access token)
Signed over	Nothing	Raw request body	Raw request body
Subscription proof	—	—	`hub.verify_token` on a `GET`, once
Event ID	None — derive from `CheckoutRequestID`	None — use `event` + `data.reference`	None — use `messages[].id` / `statuses[].id`
Retries	Redelivers unacknowledged callbacks	Every 3 min ×4, then hourly for 72 h	Immediately, then decreasing over 36 h, then dropped
Ordering	Not guaranteed	Not guaranteed	Not guaranteed; batching not guaranteed
Replay protection	None — yours to build	None — yours to build	None — yours to build
Pull / replay API	STK Push Query	`GET /transaction/verify/:reference`	None
Expected response	`200` + `{"ResultCode":0}`	`200 OK` (30 s budget)	`200 OK`

Provider behaviour above was checked against Safaricom, Paystack and Meta's own documentation in August 2026. Retry schedules, timeouts and published IP ranges change without much notice — confirm the specifics against current docs before you depend on them.

Three things fall out of reading that table across:

Nobody protects you from replay. Stripe's signed timestamp is the exception in this group, not the norm. Your dedupe table is load-bearing security, not housekeeping.

Nobody supplies an event ID. Deriving a stable one isn't an optimisation, it's a prerequisite for every other defence in this post.

The column with no replay API is the column where uptime is correctness. M-Pesa and Paystack forgive a dropped webhook because you can go and ask. Meta doesn't forgive it at all.

Testing the six

You can't wait for production to exercise these. Each failure mode has a test that fits in your existing suite:

Duplicate — POST the identical signed payload twice. Assert the side effect ran once. This is the single highest-value test in the file.
Forged — POST with a mangled signature, an empty signature, and no signature header. Assert 401, and assert nothing was written.
Missing — insert a pending row aged past the threshold, stub the provider's status API, run the reconciler, assert it completes exactly like the webhook would.
Out of order — POST read, then delivered. Assert the final status is read.
Late — POST an event whose payload contradicts current state. Assert you re-fetched rather than trusting it.
Slow — assert the handler responds in under a second with the worker stubbed out; separately, assert that an enqueue failure doesn't lose the row.

And for local development, keep a small script that replays captured payloads at your endpoint with a valid signature. Waiting on a real M-Pesa prompt to test a callback change is a slow way to work, and ngrok tunnels expire at the worst moment.

Production checklist

Conclusion

The twenty-line version of a webhook handler isn't wrong, exactly. It's the handler for the world where the promise is always kept.

Everything that makes it trustworthy comes from taking the six broken versions seriously — and they compound in a specific order. Idempotency comes first, because it's what makes every other defence safe to run. Verification comes next, because idempotency will faithfully protect a forgery. Reconciliation is third, and it's the one that separates people who've read the docs from people who've run the system, because a webhook that never arrives raises no error at all. Ordering, freshness and fast acknowledgement are then mostly consequences of the first three, written into WHERE clauses.

Underneath all six is one idea, the same one that runs through the M-Pesa integration: you never learn the truth inside the request that started it. The webhook is a hint that something changed. Your correctness has to live somewhere that doesn't depend on that hint arriving, arriving once, arriving in order, or arriving on time.

Build for the promise being broken, and the kept promise takes care of itself.

Need webhooks built properly?

I've shipped production webhook handlers against Safaricom's Daraja API, Paystack and Meta's WhatsApp Cloud API — for a welfare platform handling member contributions and claims, a job marketplace with automatic commission splitting, and a WhatsApp commerce platform with usage-based billing.

If you're integrating a provider and want idempotency, verification and reconciliation designed in from day one rather than retrofitted after the first double charge — get in touch.