Never charge twice: designing idempotent APIs for a trading backend

A user taps Buy on a train, the tunnel swallows the response, and the app retries. Your backend now has to answer one question correctly: is this a second order, or the same order asked twice?

Every trading and payments backend meets this on the way to scale. As the app I worked on at Raseed Invest grew to 100K+ active users, retries stopped being an edge case and became a daily traffic pattern. This is the design I reach for now.

The retry nobody planned for

Retries come from everywhere: flaky mobile networks, load balancers, client SDKs, queue consumers, and impatient users. You cannot turn them off, and exactly once delivery does not exist over a network. What you can build is at-least-once delivery with idempotent handlers, which gives you exactly-once effects.

Don’t try to stop duplicates from arriving. Make duplicates harmless.

The contract

Start with the API, not the code. Every mutating endpoint that moves money or places an order follows four rules:

1. The client generates an Idempotency-Key (a UUID) once per user intent, and reuses it on every retry of that intent.
2. The same key with the same request returns the original response, including the original order id.
3. The same key with a different request is an error (422), never a silent overwrite.
4. A key that is still executing returns 409 with Retry-After, so a fast double-tap cannot run twice in parallel.

One intent. One order.Request → retry → replay

Claim, execute, replay

The whole trick is an atomic claim. Redis SET NX gives one winner among concurrent requests in a single round trip. The fingerprint stops a client bug from reusing a key for a different order.

export async function idempotent(req, res, next) {
  const key = req.header('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'Idempotency-Key required' });

  const scope = `idem:${req.user.id}:${key}`;
  const fingerprint = sha256(req.method + req.path + JSON.stringify(req.body));

  // 1. Claim the key. Exactly one concurrent request wins.
  const won = await redis.set(scope, JSON.stringify({ state: 'in_flight', fingerprint }),
    { NX: true, EX: 120 });
  if (won) { res.locals.idemScope = scope; return next(); }

  // 2. Someone already used this key.
  const prior = JSON.parse(await redis.get(scope) ?? 'null')
    ?? await db.idempotency.findOne({ userId: req.user.id, key });
  if (!prior) return res.status(409).json({ error: 'retry' });
  if (prior.fingerprint !== fingerprint)
    return res.status(422).json({ error: 'Key reused with a different request' });
  if (prior.state === 'in_flight')
    return res.status(409).set('Retry-After', '1').json({ error: 'in progress' });

  return res.status(prior.status).json(prior.body);   // replay, never re-execute
}

Redis is the fast path, not the source of truth. Keys expire and caches get flushed. The durable record lives in the database, under a unique index on (userId, key), so the guarantee survives a Redis restart.

Make the side effects safe too

Idempotent HTTP handlers are not enough if the order is saved but the event that triggers settlement is lost, or sent twice. Write the business change, the idempotency record and an outbox row in one transaction, then publish from the outbox.

await session.withTransaction(async () => {
  const order = await orders.insertOne({ userId, symbol, qty, side, status: 'accepted' }, { session });
  await outbox.insertOne({ topic: 'order.accepted', payload: { orderId: order.insertedId }, sentAt: null }, { session });
  await idempotency.insertOne({ userId, key, fingerprint, status: 201, body: { id: order.insertedId } }, { session });
});
// a separate worker polls the outbox and publishes; consumers dedupe by orderId

The outbox gives you at-least-once publishing, so consumers must dedupe, usually by a natural id like orderId. That is the same idea one layer down, which is why the pattern composes well.

Failure modes worth testing

I write a test for each of these before I trust the endpoint:

Failure	Expected behaviour
Retry after a timeout, original succeeded	Same 201 and same order id
Two identical requests at the same instant	One executes, the other gets 409 then a replay
Process crashes after commit, before responding	Retry replays from the durable record
Process crashes before commit	Nothing persisted; retry executes once
Redis flushed mid-flight	Unique index rejects the duplicate
Key reused with a different body	422, no side effects

What I measure

Correctness you cannot observe is correctness you are guessing about. I track three numbers per endpoint:

• Replay rate: the share of requests served from an existing key. A sudden jump usually points to a network or client problem, not a backend one.
• Conflict rate: 409 responses, which show how often users double-submit.
• Reconciliation drift: a scheduled job comparing orders, ledger entries and provider records. The target is zero, and every non-zero result is an incident review.

The trade-offs

This design costs an extra write per request, a retention policy for keys (24 hours is a common window), and discipline on every client. In return, the worst case for a flaky network becomes a boring one: the user sees the same confirmation twice. For anything that touches money, that is the right trade.

If you remember one thing: decide what “the same request” means before you write the handler. Everything else follows from that definition.