We learned that the hard way last year, and the fix was not a new service. It was one header and one table.

The failure

Our client retried a POST after a 504. The server had already committed the charge. Two charges, one order.

The retry logic was correct for reads. For writes it was a coin flip, because the endpoint had no way to tell "this is the same request" from "this is a new one".

The fix

Every write now carries a client-generated key:

POST /charges
Idempotency-Key: 6f1c2a9e-8b3d-4f71-9c2a-...

The server stores the key together with the response body, in the same transaction that does the work:

INSERT INTO idempotency_keys (key, request_hash, response_body, created_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (key) DO NOTHING;

If the insert conflicts, we return the stored response instead of charging again. Same key, same answer, no second write.

What surprised me

The hard part was not the storage. It was expiry. Keys need to live longer than any client retry window, and longer than most people expect. We picked 24 hours after watching a mobile client retry a request eleven times across a subway ride.

We also had to hash the request body. The same key with a different body is a client bug, and silently returning a cached success hides exactly the bug you want to see.

The takeaway

Idempotency is a contract, not a feature. Once clients know it exists, they retry more freely, which is precisely what you want during an incident: the retry stops being a gamble.