Idempotency: the concept every senior dev needs to master
How to keep the same payment from being processed twice: idempotency keys, queues, external APIs, and the trade-offs.
Idempotency is a concept I've used at every company I've worked at. And it's by far one of the topics that comes up most in back-end interviews.
In this article I want to explain idempotency in the most practical way possible: what it is, when you don't need it, how to implement it, which failures each piece covers, and what it costs. At the end, I'll leave the short answer I'd give if someone asked me about it in an interview.
1. What idempotency is#
Picture a payment system. On one side, a client. On the other, a server. The client makes an HTTP call asking to add $20 to an account, and the server writes that to the database.
But the client, like any well-built system, is resilient: if something goes wrong, it does a retry, meaning it sends the same call again. Let's say it tried four times: three failed and only the fourth succeeded.
What happens on the server? It adds $20, then another $20, another $20, and another $20. The user's intent was one $20 transaction, and the result was an $80 balance.
When a system behaves like this, it's not idempotent: the same intent was processed several times and the result ended up inconsistent.
In an idempotent system, the scenario is the same (the client can fail and repeat the call four times), but there's a structure between client and server that guarantees the final result is just $20.
2. The worst case: the timeout#
The client sends "add $20" and the response never comes back. It timed out. What could have happened? Three things:
The request left the client and never reached the server.
It arrived and failed inside the server.
It arrived, was processed successfully, but the response got lost on the way back.
The third case is the worst. The $20 has already been added, but the client doesn't know that. So it sends it again, the server processes it again, and in the end the account got $40.
3. Before implementing: do you really need it?#
Before you start implementing, the first question is whether your operation is already naturally idempotent. You can tell in two ways: by the business rule and by the HTTP method.
By the business rule#
Set the balance to $100: idempotent. No matter how many times you send it, the final balance is $100.
Add $10 to the balance: not idempotent. Every execution adds another 10.
Delete a Pix key: idempotent. The final state is the same (the key doesn't exist), even if the response changes from the first time to the second.
Register a Pix key: idempotent with one condition: the target database needs a uniqueness constraint on that key. If you try to register it again, you get a uniqueness error and nothing gets duplicated.
Send a $50 Pix transfer: not idempotent. Sent it twice, $100 went out.
The rule I use: whenever the operation is relative to a previous state (add, increment, send one more), it's not idempotent by nature.
By the HTTP method#
The HTTP specification itself (RFC 9110) defines which methods are idempotent:
GET: fetches data. Idempotent.POST: creates or executes something. No guarantee. It can be, but not by definition.PUT: replaces the whole resource (or creates it, if it doesn't exist). Idempotent: sending the same PUT again leaves the resource unchanged.PATCH: updates part of the resource. It depends on the change: "set the name to Yudi" is idempotent, "add 10 dollars" isn't.DELETE: deletes. Idempotent: delete it once or ten times, the resource doesn't exist at the end.
One important detail: this is the standard. In practice, every company does it its own way, and lots of APIs don't follow the definition 100%. So if you've ever struggled to understand the difference between these methods by looking at the systems you've worked on, that's normal.
A real case where the answer was "you don't need it"#
At one of the companies I worked at, when an account was closed, we had to delete that account's Pix keys at the Central Bank and also in our internal database. For that there was a whole async structure, with an account-closing job, several states in a state machine, and a reconciliation job comparing the source of truth with our database.
But look at the operation: delete. And deleting is already naturally idempotent. That whole structure was unnecessary.
We removed it. The system became much simpler to understand and, as a bonus, inconsistencies became much less common, because there was way less stuff that could go wrong.
4. Implementing it: the idempotency key#
Now, when the operation isn't naturally idempotent (like "add $20"), the central piece is the idempotency key: a unique key that identifies that transaction intent.
The client generates the key (for example,
ID10) and saves it in its own database before sending.It sends the payload along with the key to the server.
The server also saves the key in its database.
If another request comes in with
ID10, the server realizes it has already processed that intent and doesn't add the $20 again. It just replies: "already done".
Notice that the key resolves the ambiguity: two $20 calls can be legitimate (the user really wanted to make two transfers) or a retry. With different keys, they're two transactions. With the same key, it's the same one.
Why does the client save the key in the database?#
It could just generate the key in memory and send it. But if the client process dies between one attempt and the next, the key dies with it. On the next retry it generates a new key, and the server will think it's a different transaction. Saving it to the database before sending guarantees every retry uses the same key.
Who creates the key?#
The key has to be born at the moment the operation's intent is created, in the first system that records that transaction. From there, it travels through every service in the flow, and they all know it's a single operation.
Bonus: that's also why the idempotency key is great for log tracing. You can follow the same business intent as it passes through several systems.
5. What the server stores (and why)#
The more complexity you add, the more points of failure show up. The server can also die halfway through, and that's why it stores the key in the database: when it comes back, the key is still there. But it's not just the key. The idempotency table usually stores:
Idempotency key (unique): identifies the intent.
Request hash: detects when someone sends the same key with a different body. That points to a bug, an inconsistency, or even a fraud attempt, and the request should fail.
Status (processing or completed): tells you what to do with a retry that arrives midway.
Response body: the original response, to return exactly as it was on the retry.
Time to live (TTL): how long the key is kept.
The retry arrived: what do you respond?#
Still processing: usually respond with
409 Conflict.Already completed: respond as if it were the first call, with the same status (for example,
201) and the same response body.
Why return the original body? Think of an app making a transfer. If it had to retry, it wants exactly the response that would show the "transfer completed successfully" screen. A bare "already done" won't cut it.
Watch out for the TTL#
The key doesn't need to be stored forever. It can last, say, a day. But the TTL has to be in sync with whoever does the retry: if the client can retry two days from now and the key only lives for one, the retry hits a server that already forgot the key, and the duplication is back. When you integrate with a system, find out how long its idempotency key lasts.
6. Two requests in the same millisecond#
What if two requests with the same key arrive at the same time? It could be parallel processing, a call that got delayed on the network... Both ask the database "does ID10 exist yet?", both hear "no", and both start processing.
Can it happen? Of course it can. What saves you here is the structure itself: the idempotency key is a unique key in the database. When the server tries to insert the same key a second time, the database throws a uniqueness error. Only one of the two manages to write, and only that one processes.
7. What if instead of a client it were a queue?#
It works the same way, with the same idempotency table. The difference is that, instead of responding with success to the client, you send the ack (acknowledgement) to the queue, and only after all the processing is done.
And with queues this matters even more. If you read the RabbitMQ, Kafka, or SQS docs, you'll find the term at-least-once: the message will be delivered at least once. In other words, it can arrive again, and making sure that doesn't cause problems is your responsibility.
They don't do this out of spite. Only your system knows whether the message was processed successfully. So the queue keeps redelivering until you confirm with the ack. If you don't confirm, it sends it again, no hesitation.
8. What if there's an external API in the way?#
Now your server is also someone's client: it calls an external payment system, which is the source of truth (it could be the Central Bank, for example).
One more piece, one more point of failure. The server saves the key, calls the external API and, before marking the status as completed, dies. The record stays as "processing". When something picks that record back up, it'll call the external API again, and the transaction may get duplicated on the other side.
From there, you have three paths:
The external API accepts an idempotency key: great. You send the same key in both calls and it doesn't duplicate. That's why the key has to be born at the start of the intent and travel to the end of the flow: if you sent a new key to the external system, it wouldn't help at all.
It doesn't accept one: you can check first (a GET to see if the record already exists) and only create it if it doesn't. It's more complex, but it avoids inconsistency.
None of that works: what's left is reconciliation. A job that runs, say, every day at 3 a.m., compares what's in the source of truth (transactions and balance) with your internal database and fixes the differences. The external system is the source of truth; your database is there to serve the data faster.
I've had cases like that. Reconciliation is the last line of defense against inconsistency.
9. Senior Mode: what does all this cost?#
In an interview for a senior role, knowing how to implement it isn't enough. You need to talk about the trade-offs. In general, the cost is always complexity and resource usage, but it's worth answering specifically:
Storage: a table (or database) for the key, the hash, the response... on the server and on the client.
Cleanup: if the TTL is one day, something has to delete those keys. One more job, more complexity.
Latency: every request gets extra database read and write steps.
"So I'll put the idempotency table in Redis"#
Careful. That's usually not a good idea, because the key needs to live in the same database as the effect. You need to write the key and execute the +$20 in the same transaction, otherwise you lose atomicity: one of the two things can happen without the other.
Redis makes sense when there's no effect in your internal database to protect. For example, if your service just forwards a message to the payment system and doesn't write anything locally.
10. The 30-second interview answer#
"How would you make sure a payment system doesn't process the same payment twice?"
Here's how I'd answer:
I'd use idempotency. Assuming retries exist, the client generates an idempotency key at the start of the transaction intent and saves it to the database before sending. The server saves that key as unique, along with the request hash and the response body. If the client retries, the server returns the same response; if it's still processing, it responds with 409 Conflict. If there's an external system, I pass the same key to it. And if there's still duplication on that side, a reconciliation job sorts it out at the end.
You don't need to memorize all of this#
It's a lot, but the idea isn't to know it by heart. Day to day you'll look things up: "how does a Transactional Outbox work again?", "can I store this in Redis or does it have to be in SQL?". What matters is having the reasoning. With it, even in an interview you can talk it through with the interviewer and reach the solution.
And this topic shows up in every kind of interview: system design, behavioral (when you talk about architecture decisions you've made), and even coding exercises.
If you'd rather see all of this being drawn out, the full video is right at the top of the article.
