Retry Requests Fearlessly with Idempotence

Reading time: 5 min
Cover generated by using a tool created by Charles Berlin


Every single day of our lives is unique — most of the things that happen are likely not going to happen again in the same manner. They occur exactly once. Unfortunately, that isn’t the case with software systems and requests between them. Why should that matter? Imagine purchasing a new laptop and because of some duplicate service requests being charged twice. That doesn’t sound fun, does it?

In theory, exactly-once request delivery is impossible — the classic Two Generals Problem illustrates the pitfalls and design challenges. However, in practice, we rarely care about delivering our requests exactly once. We usually go for the second-best thing — at least once delivery, where we keep sending the same service request until we are confident that it is received (it might be received multiple times though). But hey, we don’t want to get charged twice, right? We can avoid that by ensuring that handling the same request multiple times doesn’t change the result beyond the initial application. In fact, that’s the essence of idempotence.

Now that you have received a duplicate service request, how do you handle idempotence? Let’s explore your options with an example of a bank account. The code examples are written in C# because its syntax is relatively simple and should not be challenging to understand for most developers.

Maybe you don’t need to do anything after all!

Some requests are idempotent naturally, and you don’t have to worry about retrying them at all. Take a simple bank account aggregate, for instance:

To open a new bank account, you can send an OpenAccountRequest like this:

OpenAccountRequest is idempotent as it is. Whenever you receive a request, you can check if an account with the same ID exists in our database. If the account already exists — you know that you received a duplicate request. Therefore you can ignore it. Moreover, if your database guarantees account IDs to be unique, you can check for the duplicate key exception instead:

It’s all fun and games until you try to withdraw or deposit money. If you send the same deposit request twice because of a network error — you deposit twice the money even though you initially intended to deposit once. Twice the money — twice the fun!

Do you know the current state?

One way to handle duplicate requests is to ask for the current state of the aggregate. Strictly speaking, this solution is typically considered for a bit broader problem of optimistic concurrency than idempotence, and it doesn’t identify duplicate requests. However, if you ask to provide the account balance in your DepositToAccountRequest, you will be capable of protecting yourself against duplicates:

Since you can’t identify duplicate requests, you can’t be sure whether you received a duplicate request, or the client wasn’t aware that the balance had been changed. Therefore you would most likely reject all invalid state requests and ask the client to get the latest state and try again (hence no try-catch block ignoring InvalidStateException).

This solution is relatively easy to implement, and it doesn’t introduce technical details into our account aggregate. However, it is not very flexible since for other requests to be protected in the same way providing account balance may not be enough. Not to mention it could fail in highly concurrent scenarios, i.e., while you retry a successful withdrawal request, someone else deposits the same amount of money, making you withdraw twice the amount. To mitigate those issues, you can artificially generate a state by versioning your account. Whenever you change the account aggregate, you must increase its version:

Unlike with account balance, you can include the account version into any of your requests. Highly concurrent scenarios are covered as well since you are only allowed to increase the version. And if you are using the Event Sourcing pattern, most likely, you already have a version in your aggregates.

It is crucial to consider the limitations of this approach. To send a request, you need to know the current state of the aggregate. And if the aggregate is changed rather frequently, using the system becomes quite tricky and inefficient since many requests get rejected because of outdated state. But hey, at least you are not that scared of duplicates anymore, right?

Just put a database transaction all over this!

Another widespread solution requires an ACID-compliant database. First, you need to add unique IDs to your requests, which shouldn’t be changed when retrying:

Then you have to create another database table that is going to store the history of request IDs. The table should guarantee IDs are unique. When you receive a request, you need to insert the request ID into the new database table right before persisting your aggregate state within a database transaction. If the request ID is already stored, you know that it’s a duplicate request. Unlike with account balance or version, now you are actually making requests idempotent.

Can we optimize this? You probably don’t need to store every request ID since the dawn of time. Can you assume that you never receive duplicate requests after a certain amount of time? If that’s the case — you can have a process that periodically removes IDs from the history table, making it very lightweight.

What if a transaction isn’t an option?

Sometimes you don’t have an ACID-compliant database, making transactions not an option anymore. What you can do instead is store request IDdirectly into your aggregate:

This doesn’t protect you against duplicates in highly concurrent scenarios. However, if you don’t send your request retries concurrently (in most cases, you shouldn’t), you don’t have to worry about that.

If the aggregate is changed frequently, storing every request ID might increase its size by quite a lot. But if you only care about the most recent requests (like with the database table before), you can optimize this by storing only a certain number of the most recent operations.

Once again, if you are one of the Event Sourcing folks, life sometimes is just a tiny bit better for you. Since you are already storing your aggregates as event streams, you can add a request ID to each of the resulting events. Whenever you rebuild your aggregate, you can reconstruct the request history and validate a new request against it:

There’s one more option if you are using Event Sourcing. It is based upon being able to generate deterministic UUIDs defined in RFC 4122. If your event store can guarantee unique event IDs, you can use a request ID to generate resulting event IDs deterministicallyThe only thing left is to catch a duplicate key exception since your event store is handling the rest:

Wow, you are still reading? It’s time to summarize!

There’s no right way to deal with idempotence — it depends solely on your business and technical requirements. Sometimes you might receive service requests that are idempotent naturally. At other times, if your aggregates don’t change very often, it could make sense to provide the current aggregate state in your requests to deal with duplicate requests. Maybe you have an ACID-compliant database? If that’s the case, you could potentially go with database transactions since it is reasonably simple to implement, can be used with all of your requests, and doesn’t leak technical details into the domain. Do you use the Event Sourcing pattern with an event store that guarantees unique event IDs (i.e., SQL Server)Then you could avoid transactions by generating deterministic event IDs. What if your event store doesn’t ensure unique event IDs (i.e., Greg Young’s Event Store)? You could store request IDs in your aggregates instead. Not to mention that likely there are other ways used in specific scenarios that I have not covered.

Thank you for reading. I’d love to hear how you handle idempotent requests.