Idempotency in Backend Systems: Preventing Duplicate Operations

1. What Is Idempotency?

Here's a scenario every payments developer eventually meets. A customer taps "Pay", the request reaches your server, the payment is processed, and then the response gets lost on a flaky mobile network. The app sees a timeout, assumes the request failed, and quietly retries. Your server receives the same request a second time, and, having no reason to doubt it, charges the customer again.

Nothing crashed. No error was logged. Every component did exactly what it was told. And yet the customer has been charged twice for one order.

The idea that prevents this is idempotency, and it comes down to one simple rule:

The same request can arrive multiple times, but the intended business effect should happen only once.

In math and HTTP terms, an operation is idempotent if doing it once or ten times leaves the system in the same state. GET, PUT and DELETE are idempotent by design. POST /payments is not: every call creates something new. Idempotency for those operations has to be added deliberately, usually by attaching a unique identifier to the intent (this one payment) so the server can recognise a repeat.

One clarification: idempotency is not the same as "exactly-once delivery", which real networks can't give you. What you build in practice is effectively-once processing: messages are delivered at-least-once, and the receiver deduplicates so the business effect happens once.

The core flow looks like this:

Request
   ↓
Check idempotency key
   ↓
Already processed?
   ├── Yes → Return previous result
   │
   └── No
        ↓
   Process operation
        ↓
   Store result + idempotency key
        ↓
   Return response

The rest of this post shows how to build that flow, from database constraints up to a Node.js/Express implementation, and how it applies to M-Pesa and other payment webhooks.

2. Why Duplicate Requests Happen

Duplicates aren't a sign that something is broken. They're a normal consequence of how distributed systems communicate. There are five common sources.

Payment webhooks

Webhook providers prefer to risk sending an event twice rather than risk losing it. Stripe, GitHub and most other providers use at-least-once delivery, which means the same event may arrive more than once by design. One team documented Stripe sending the same checkout.session.completed event four times in a row, each correctly signed and carrying the same event ID.

The usual trigger is a slow or failed acknowledgement. If your handler does heavy work synchronously and times out, or finishes the work and crashes before returning 200, the provider assumes delivery failed and sends it again.

Network retries

The client sends a request, the server processes it, and the response is lost on the way back. From the client's side, "no response" looks identical to "never arrived", so it retries. This is the classic double-charge scenario from the introduction.

Users clicking twice

Slow "Pay" buttons, double-taps on mobile, and browser refreshes after submitting a form all produce genuine duplicate requests, often milliseconds apart.

Message queues

Most message brokers guarantee at-least-once delivery. If a consumer does the work but crashes before acknowledging the message, the broker redelivers it. A visibility timeout that is shorter than your processing time causes the same effect.

Client retries

HTTP client libraries, mobile SDKs, API gateways and load balancers frequently retry automatically. It's common to have two or three layers of retry logic stacked on top of each other without anyone having designed it that way.

The lesson is the same in every case: you can't prevent duplicates from arriving, so the receiving side has to handle them safely.

3. A Real-World Payment Example

Take a customer paying KES 5,000 for order #42.

The client calls POST /payments with { orderId: 42, amount: 5000 }.
The server charges the customer and inserts a payment row.
The response is lost because of a network blip.
The client's retry logic fires and sends the same request again.
Without idempotency, the server charges the customer a second time.

The result is two deductions and one order. That means a support ticket, a refund, a reconciliation headache and, worst of all, a customer who is less willing to pay you next time.

The fix is to make step 4 detectable. If the server can tell that the second request is a repeat of the first, it can return the original result instead of doing the work again. That is what idempotency keys are for.

4. Idempotency Keys

An idempotency key is a unique token that the client generates once per logical operation and reuses on every retry of that operation.

Rules for good keys

The client generates the key before the first attempt and sends the same key on every retry. A UUIDv4 is the usual choice, and the IETF draft for the Idempotency-Key header also recommends a UUID or a similar random identifier.
A new operation gets a new key. Don't derive keys by hashing the request body. Two genuine, identical purchases would collapse into one.
Keys are scoped. Uniqueness should be per user, merchant or API key, not global, so one customer's key can never collide with another's.

What the server should do

Stripe's model is the best-known reference. It saves the resulting status code and body of the first request made with a given key, whether it succeeded or failed, and returns that same result for every later request with the same key. That even includes 500 errors. Stripe deliberately does not save a result if the request failed validation or conflicted with another request still executing, because in those cases no operation actually started, so the client can safely retry.

Expiry

Keys don't live forever. In Stripe's v1 API, keys can be pruned once they're at least 24 hours old, and reusing a pruned key creates a brand-new request. Stripe's newer v2 API extends the replay window to 30 days. Other providers use different windows, so don't treat 24 hours as a universal constant. Choose a retention period longer than the longest realistic retry window for your system.

The standard

There is an IETF draft for an Idempotency-Key HTTP header, aimed at making POST and PATCH fault-tolerant. It is still a draft rather than a finished RFC, but its suggested error handling is a useful template:

400 Bad Request when a required key is missing.
422 Unprocessable Content when a key is reused with a different request payload.
409 Conflict when a request with the same key is still being processed.

5. Using Database Constraints

Application-level checks such as "does this payment exist? No? Then insert it" contain a race condition. Two concurrent requests can both see "no" and both insert. The database is the only component that can reliably arbitrate between them, so use unique constraints as your last line of defence.

Unique payment/reference IDs

CREATE TABLE payments (
  id                  BIGSERIAL PRIMARY KEY,
  order_id            BIGINT      NOT NULL,
  amount_cents        BIGINT      NOT NULL,
  status              TEXT        NOT NULL DEFAULT 'pending',
  provider_reference  TEXT,        -- M-Pesa receipt, Stripe payment id, etc.
  checkout_request_id TEXT,
  created_at          TIMESTAMPTZ NOT NULL DEFAULT now(),
  CONSTRAINT uq_payments_provider_ref UNIQUE (provider_reference),
  CONSTRAINT uq_payments_checkout     UNIQUE (checkout_request_id)
);

Good candidates for unique constraints include the provider's transaction or receipt number, the webhook event ID, the checkout request ID, and your own natural key such as (user_id, order_id) when an order can only have one payment.

With a constraint in place, a duplicate insert fails loudly with a unique-violation error (PostgreSQL error code 23505) instead of silently succeeding. Your code can catch that error and treat it as "already done".

6. Implementing Idempotency in Node.js/Express

The design uses a dedicated table that stores the key, a fingerprint of the request, a status, and the saved response.

CREATE TABLE idempotency_keys (
  scope           TEXT        NOT NULL,   -- user id / merchant id / API key
  key             TEXT        NOT NULL,
  request_hash    TEXT        NOT NULL,
  status          TEXT        NOT NULL DEFAULT 'in_progress', -- in_progress | completed
  response_status INT,
  response_body   JSONB,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (scope, key)
);

And the Express middleware:

// idempotency.js
const crypto = require('crypto');
const pool = require('./db'); // pg Pool

const hashBody = (body) =>
  crypto.createHash('sha256').update(JSON.stringify(body ?? {})).digest('hex');

function idempotent() {
  return async (req, res, next) => {
    const key = req.get('Idempotency-Key');
    if (!key) {
      return res.status(400).json({ error: 'Idempotency-Key header is required' });
    }

    const scope = req.user.id;              // never make keys global
    const requestHash = hashBody(req.body);

    // 1. Try to claim the key. Only one concurrent request can win this insert.
    const claim = await pool.query(
      `INSERT INTO idempotency_keys (scope, key, request_hash)
       VALUES ($1, $2, $3)
       ON CONFLICT (scope, key) DO NOTHING
       RETURNING key`,
      [scope, key, requestHash]
    );

    if (claim.rowCount === 0) {
      // 2. The key already exists. Look at what happened before.
      const { rows } = await pool.query(
        `SELECT request_hash, status, response_status, response_body
           FROM idempotency_keys WHERE scope = $1 AND key = $2`,
        [scope, key]
      );
      const prev = rows[0];

      if (prev.request_hash !== requestHash) {
        return res.status(422).json({ error: 'Idempotency-Key reused with a different request' });
      }
      if (prev.status === 'in_progress') {
        return res.status(409).json({ error: 'A request with this key is still being processed' });
      }
      // Completed: replay the stored result.
      res.set('Idempotent-Replayed', 'true');
      return res.status(prev.response_status).json(prev.response_body);
    }

    // 3. We own the key. Capture the response so we can store it.
    const originalJson = res.json.bind(res);
    res.json = (body) => {
      pool.query(
        `UPDATE idempotency_keys
            SET status = 'completed', response_status = $3, response_body = $4
          WHERE scope = $1 AND key = $2`,
        [scope, key, res.statusCode, JSON.stringify(body)]
      ).catch(console.error);
      return originalJson(body);
    };

    next();
  };
}

module.exports = idempotent;

Using it is one line:

app.post('/payments', idempotent(), async (req, res) => {
  const { orderId, amountCents } = req.body;
  const payment = await createPayment(orderId, amountCents);
  res.status(201).json(payment);
});

This version is a good starting point, but it has known weaknesses worth understanding:

The result is saved after the handler runs. If the process crashes between the business work and the final UPDATE, the key stays in_progress forever. Add a lock timeout (treat in_progress keys older than a few minutes as retryable) and a recovery job.
JSON.stringify is not canonical, because key order can change the hash. Use a stable-stringify library for the fingerprint.
The final UPDATE is fire-and-forget. The next section removes that gap.

7. Implementing It with PostgreSQL

The strongest pattern puts the key claim and the business writes in one transaction. PostgreSQL does the arbitration for you: a second INSERT ... ON CONFLICT DO NOTHING on the same key waits for the first transaction to commit or roll back, and then proceeds knowing the outcome.

async function createPaymentIdempotent({ scope, key, requestHash, orderId, amountCents }) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    const claim = await client.query(
      `INSERT INTO idempotency_keys (scope, key, request_hash, status)
       VALUES ($1, $2, $3, 'in_progress')
       ON CONFLICT (scope, key) DO NOTHING
       RETURNING key`,
      [scope, key, requestHash]
    );

    if (claim.rowCount === 0) {
      const { rows } = await client.query(
        `SELECT request_hash, response_status, response_body
           FROM idempotency_keys WHERE scope = $1 AND key = $2`,
        [scope, key]
      );
      await client.query('COMMIT');
      if (rows[0].request_hash !== requestHash) {
        return { status: 422, body: { error: 'Key reused with a different payload' } };
      }
      return { status: rows[0].response_status, body: rows[0].response_body, replayed: true };
    }

    // Business work and the idempotency record commit atomically.
    const { rows: [payment] } = await client.query(
      `INSERT INTO payments (order_id, amount_cents, status)
       VALUES ($1, $2, 'pending') RETURNING *`,
      [orderId, amountCents]
    );

    const result = { status: 201, body: payment };
    await client.query(
      `UPDATE idempotency_keys
          SET status = 'completed', response_status = $3, response_body = $4
        WHERE scope = $1 AND key = $2`,
      [scope, key, result.status, JSON.stringify(result.body)]
    );

    await client.query('COMMIT');
    return result;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

If the process crashes midway, the whole transaction rolls back: no payment row and no key. The retry starts from a clean slate.

There's one important limit. This only protects what lives inside your database. If the handler also calls an external API such as a card processor or M-Pesa, that call is not rolled back. For those flows, commit the key as in_progress first, make the external call using a downstream idempotency key or a reference the provider deduplicates on, and then record the result. Never hold a database transaction open across a slow network call.

8. Idempotency and M-Pesa/Payment Webhooks

M-Pesa (Daraja) STK Push is a good real-world case because everything is asynchronous. You call the API, get back a CheckoutRequestID, and the outcome arrives later on your callback URL.

What identifies a transaction

CheckoutRequestID identifies the STK request. Store it when you initiate the push, mark the order as pending, and match the callback against it later.
MpesaReceiptNumber, present in the callback metadata on success, is the anchor that matches M-Pesa's own transaction history.
ResultCode tells you the outcome. 0 means success. Other codes cover cases such as 1032 (the customer cancelled the prompt) and 1 (insufficient balance).

Two failure modes to handle

Duplicate callbacks. Anything delivered over the internet can arrive twice, and callback handlers should be written on that assumption.
Missing callbacks. Community guides for Daraja note that failed callbacks are not repeated, so if Safaricom can't reach your server you may never hear the result. That is what the STK Push Query endpoint is for. A background job should periodically query any payment that has been pending for too long.

Sources differ on exactly how Safaricom retries callbacks, so check the current Daraja documentation and test on your own shortcode. Either way, a correct design tolerates both duplicates and drops.

An idempotent callback handler

app.post('/mpesa/callback', async (req, res) => {
  // Acknowledge quickly; do the real work safely.
  res.json({ ResultCode: 0, ResultDesc: 'Accepted' });

  const cb = req.body?.Body?.stkCallback;
  if (!cb) return;

  const { CheckoutRequestID, ResultCode } = cb;
  const items = cb.CallbackMetadata?.Item ?? [];
  const receipt = items.find((i) => i.Name === 'MpesaReceiptNumber')?.Value;

  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // Conditional update: only the FIRST callback can flip pending -> paid.
    const { rowCount } = await client.query(
      `UPDATE payments
          SET status = $2, provider_reference = $3
        WHERE checkout_request_id = $1 AND status = 'pending'`,
      [CheckoutRequestID, ResultCode === 0 ? 'paid' : 'failed', receipt ?? null]
    );

    if (rowCount === 1 && ResultCode === 0) {
      await fulfillOrder(client, CheckoutRequestID); // same transaction
    }
    // rowCount === 0 means a duplicate callback (or unknown ID): safe no-op.

    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    if (err.code !== '23505') console.error(err); // 23505 = receipt already recorded
  } finally {
    client.release();
  }
});

Three things make this safe:

The WHERE status = 'pending' guard means a second callback updates zero rows and fulfils nothing.
The UNIQUE (provider_reference) constraint stops one M-Pesa receipt from being credited to two payments.
Fulfilment (crediting a wallet, sending a receipt, marking the order paid) runs inside the same transaction as the status change, so it can't run twice.

The same pattern for other webhooks

For Stripe and similar providers, record every processed event ID in a table with a unique constraint, and skip events you've already recorded. Add a state guard as well, because a unique delivery ID stops one delivery from running twice, but a provider can emit two different events describing the same business transition.

Also don't rely on event order. Providers such as Stripe explicitly do not guarantee that events arrive in the order they were generated, so handlers should be built around current state, and should re-fetch from the provider's API when in doubt.

One more practical note: Safaricom has reportedly begun masking the phone number in STK callbacks, so don't include the phone number in a deduplication key. Use CheckoutRequestID and the receipt number instead.

9. Common Mistakes

Check-then-insert without a constraint. A SELECT followed by an INSERT is a race. Use a unique index or ON CONFLICT.
Not comparing the payload. A reused key with different data should be rejected with a 422, not answered with the old response.
Global keys. Two users' keys can collide. Scope by user, merchant or API key.
Storing the key but not the result. Then a retry has nothing to replay, and you either re-run the work or return an unhelpful error.
Marking work "done" before it is durable. If you record success and then crash, the retry is skipped and the work never actually happened.
Doing slow work before acknowledging a webhook. This triggers the very retries you're defending against. Persist the event, return 2xx, then process it in a queue or worker.
Hashing the body to create the key. Two genuine identical purchases get merged into one. Keys represent intent, not content.
Keys that expire before retries stop. Match the retention period to your longest retry window, and clean up old keys with a scheduled job.
Trusting the callback alone. Reconcile against the provider (STK Push Query, transaction status APIs), because callbacks can be lost.
Forgetting non-database side effects. Emails, SMS messages and wallet credits need their own protection, or must be triggered from inside the same atomic unit, for example through a transactional outbox.

10. Idempotency vs Database Transactions

These two are often confused, but they solve different problems, and production systems need both.

A database transaction answers the question: "If this operation fails halfway, is my data left consistent?" It gives you atomicity within a single execution. Either every write happens or none does.

Idempotency answers a different question: "If this request is repeated, does the effect happen twice?" It protects you across multiple executions, whether they come from retries, double-clicks or redelivered messages.

Here is why neither is enough alone. A transaction can make "deduct wallet and create order" all-or-nothing, but it does nothing to stop the client from sending that request twice. Each copy is a perfectly valid transaction, so you end up with two orders. An idempotency key makes the second request a no-op. In the other direction, an idempotency key does not stop a half-finished operation from leaving partial state behind. That is the transaction's job.

The strongest designs write the idempotency record and the business change in the same transaction, as shown in section 7, so the two can never drift apart.

11. Practical Checklist

Before shipping any endpoint or handler that changes state and can be retried, work through this list:

Identify retryable operations. Payments, orders, transfers, signups and anything triggered by a webhook or queue.
Accept an idempotency key (or rely on a natural unique reference), generated by the client once per intent and reused on every retry.
Scope keys per user or merchant, and store a request fingerprint, a status and the saved response.
Return the right errors. 400 for a missing key, 422 for a reused key with a different payload, 409 for a request still in flight.
Add unique constraints on provider references, receipt numbers, webhook event IDs and checkout request IDs.
Guard state transitions with conditions such as WHERE status = 'pending', rather than just "insert if not exists".
Commit the key and the business writes together in one transaction wherever possible.
Handle external calls carefully. Use a downstream idempotency key or a recoverable in_progress state, and keep slow calls out of open transactions.
Acknowledge webhooks fast. Persist first, return 2xx, process asynchronously.
Assume queue redelivery. Make consumers idempotent.
Set key retention longer than your retry window, and clean up expired keys.
Run a reconciliation job for lost callbacks, such as polling STK Push Query for stale pending payments.
Test duplicates on purpose. Fire the same request several times, including concurrently, and assert that exactly one business effect occurred.
Log replays so duplicates are visible in your monitoring rather than silently absorbed.

12. Conclusion

Retries, at-least-once webhooks, impatient users and redelivered queue messages aren't edge cases. They are how networked systems behave, so handling them is part of the design, not an afterthought.

The idea is simple. Give each intent a stable identity, let the database enforce it, and when a repeat arrives, return the result of the first attempt instead of doing the work again. For payments, that means an idempotency key at the API edge, unique constraints on provider references, guarded state transitions, atomic writes, and a reconciliation job for the callbacks that never arrive.

Get those pieces right and a double-tap, a timeout or a duplicate M-Pesa callback stops being a support ticket and becomes a non-event. Which is exactly what your users should experience.