Idemfu*kingwhat?
Every single day of our lives is unique — most of the things that happen are likely not going to happen again in the same manner. They occur exactly once. Unfortunately, that isn’t the case with software systems and requests between them. Why should that matter? Imagine purchasing a new laptop and, because of some duplicate service requests, being charged twice. That doesn’t sound fun, does it?
In theory, exactly-once request delivery is impossible — the classic Two Generals Problem illustrates the pitfalls and design challenges. However, in practice, we rarely care about delivering our requests exactly once. We usually go for the second-best thing — at least once delivery, where we keep sending the same service request until we are confident that it is received (it might be received multiple times though). But hey, we don’t want to get charged twice, right? We can avoid that by ensuring that handling the same request multiple times doesn’t change the result beyond the initial application. In fact, that’s the essence of idempotence.
Now that you have received a duplicate service request, how do you handle idempotence? Let’s explore your options with an example of a bank account. The code examples are written in C# because its syntax is relatively simple and should not be challenging to understand for most developers.
Maybe you don’t need to do anything after all!
Some requests are idempotent naturally, and you don’t have to worry about retrying them at all. Take a simple bank account aggregate, for instance:
public sealed class Account
{
public Guid Id { get; }
public decimal Balance { get; private set; }
public Account(Guid id, decimal balance)
{
if (id == default)
throw new InvalidOperationException("Account id must be provided");
if (balance < 0)
throw new InvalidOperationException("Balance cannot be negative");
Id = id;
Balance = balance;
}
public void Withdraw(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot withdraw negative amount");
if (amount > Balance)
throw new InvalidOperationException("Cannot withdraw more than existing balance");
Balance -= amount;
}
public void Deposit(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot deposit negative amount");
Balance += amount;
}
}
To open a new bank account, you can send an OpenAccountRequest like this:
public sealed class OpenAccountRequest
{
public Guid AccountId { get; }
public decimal Balance { get; }
public OpenAccountRequest(Guid accountId, decimal balance)
{
AccountId = accountId;
Balance = balance;
}
}OpenAccountRequest is idempotent as it is. Whenever you receive a request, you can check if an account with the same ID exists in our database. If the account already exists, you know you received a duplicate request. Therefore you can ignore it. Moreover, if your database guarantees account IDs to be unique, you can check for the duplicate key exception instead:
public async Task HandleAsync(OpenAccountRequest request, CancellationToken token = default)
{
var account = new Account(request.AccountId, request.Balance);
try
{
await _repository.InsertAsync(account, token);
}
catch (DuplicateKeyException)
{
//Ignore
}
}It’s all fun and games until you try to withdraw or deposit money. If you send the same deposit request twice because of a network error — you deposit twice the money even though you initially intended to deposit it once. Twice the money — twice the fun!
Do you know the current state?
One way to handle duplicate requests is to ask for the current state of the aggregate. Strictly speaking, this solution is typically considered for a bit broader problem of optimistic concurrency than idempotence, and it doesn’t identify duplicate requests. However, if you ask to provide the account balance in your DepositToAccountRequest, you will be capable of protecting yourself against duplicates:
public sealed class DepositToAccountRequest
{
public Guid AccountId { get; }
public decimal Amount { get; }
public decimal AccountBalance { get; }
public DepositToAccountRequest(Guid accountId, decimal amount, decimal accountBalance)
{
AccountId = accountId;
Amount = amount;
AccountBalance = accountBalance;
}
}public async Task HandleAsync(DepositToAccountRequest request, CancellationToken token = default)
{
var account = await _repository.GetAsync(request.AccountId, token) ??
throw new EntityNotFoundException();
account.Deposit(request.Amount);
await _repository.UpdateAsync(account, request.AccountBalance, token);
}public sealed class AccountRepository : IAccountRepository
{
//....
public async Task UpdateAsync(Account account, decimal expectedBalance, CancellationToken token = default)
{
var sql = "UPDATE Accounts SET Balance = @Balance WHERE Id = @Id AND Balance = @ExpectedBalance";
var sqlParams = new
{
Id = account.Id,
Balance = account.Balance,
ExpectedBalance = expectedBalance
};
await using var connection = new SqlConnection(_connectionString);
await connection.OpenAsync(token);
var rowsAffected = await connection.ExecuteAsync(sql, sqlParams);
if (rowsAffected == 0)
throw new InvalidStateException();
}
//....
}Since you can’t identify duplicate requests, you can’t be sure whether you received a duplicate request or the client wasn’t aware that the balance had been changed. Therefore you would most likely reject all invalid state requests and ask the client to get the latest state and try again (hence no try-catch block ignoring InvalidStateException).
This solution is relatively easy to implement and doesn’t introduce technical details into our account aggregate. However, it is not very flexible since for other requests to be protected in the same way providing account balance may not be enough. Not to mention it could fail in highly concurrent scenarios, i.e., while you retry a successful withdrawal request, someone else deposits the same amount of money, making you withdraw twice the amount. To mitigate those issues, you can artificially generate a state by versioning your account. Whenever you change the account aggregate, you must increase its version:
public sealed class Account
{
public Guid Id { get; }
public int Version { get; private set; }
public decimal Balance { get; private set; }
public Account(Guid id, decimal balance)
: this(id, 0, balance) { }
//Constructor to restore the state
private Account(Guid id, int version, decimal balance)
{
if (id == default)
throw new InvalidOperationException("Account id must be provided");
if (version < 0)
throw new InvalidOperationException("Version cannot be negative");
if (balance < 0)
throw new InvalidOperationException("Balance cannot be negative");
Id = id;
Version = version;
Balance = balance;
}
public void Withdraw(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot withdraw negative amount");
if (amount > Balance)
throw new InvalidOperationException("Cannot withdraw more than existing balance");
Balance -= amount;
Version++;
}
public void Deposit(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot deposit negative amount");
Balance += amount;
Version++;
}
}public sealed class DepositToAccountRequest
{
public Guid AccountId { get; }
public int AccountVersion { get; }
public decimal Amount { get; }
public DepositToAccountRequest(Guid accountId, int accountVersion, decimal amount)
{
AccountId = accountId;
AccountVersion = accountVersion;
Amount = amount;
}
}public async Task HandleAsync(DepositToAccountRequest request, CancellationToken token = default)
{
var account = await _repository.GetAsync(request.AccountId, token) ??
throw new EntityNotFoundException();
account.Deposit(request.Amount);
await _repository.UpdateAsync(account, request.Version, token);
}public sealed class AccountRepository : IAccountRepository
{
//....
public async Task UpdateAsync(Account account, int expectedVersion, CancellationToken token = default)
{
var sql = "UPDATE Accounts SET Balance = @Balance, Version = @Version WHERE Id = @Id AND Version = @ExpectedVersion";
var sqlParams = new
{
Id = account.Id,
Balance = account.Balance,
Version = account.Version,
ExpectedVersion = expectedVersion
};
await using var connection = new SqlConnection(_connectionString);
await connection.OpenAsync(token);
var rowsAffected = await connection.ExecuteAsync(sql, sqlParams);
if (rowsAffected == 0)
throw new InvalidStateException();
}
//....
}Unlike with account balance, you can include the account version in any of your requests. Highly concurrent scenarios are also covered since you can only increase the version. And if you are using the Event Sourcing pattern, you likely already have a version in your aggregates.
It is crucial to consider the limitations of this approach. To send a request, you need to know the current state of the aggregate. And if the aggregate is changed rather frequently, using the system becomes quite tricky and inefficient since many requests get rejected because of an outdated state. But hey, at least you are not that scared of duplicates anymore, right?
Just put a database transaction all over this!
Another widespread solution requires an ACID-compliant database. First, you need to add unique IDs to your requests, which shouldn’t be changed when retrying:
public sealed class DepositToAccountRequest
{
public Guid Id { get; }
public Guid AccountId { get; }
public decimal Amount { get; }
public DepositToAccountRequest(Guid id, Guid accountId, decimal amount)
{
Id = id;
AccountId = accountId;
Amount = amount;
}
}Then you have to create another database table that will store the history of request IDs. The table should guarantee IDs are unique. When you receive a request, you need to insert the request ID into the new database table before persisting your aggregate state within a database transaction. If the request ID is already stored, you know that it’s a duplicate request. Unlike with account balance or version, now you are actually making requests idempotent.
public sealed class AccountRepository : IAccountRepository
{
//....
public async Task UpdateAsync(Account account, Guid requestId, CancellationToken token = default)
{
var requestSql = "INSERT INTO RequestIds VALUES (@Id)";
var requestSqlParams = new
{
Id = requestId.ToString()
};
var accountSql = "UPDATE Accounts SET Balance = @Balance WHERE Id = @Id";
var accountSqlParams = new
{
Id = account.Id,
Balance = account.Balance
};
await using var connection = new SqlConnection(_connectionString);
await connection.OpenAsync(token);
await using var transaction = await connection.BeginTransactionAsync(token);
try
{
await connection.ExecuteAsync(requestSql, requestSqlParams);
}
catch (Exception e) when (IsDuplicateKeyException(e))
{
throw new DuplicateKeyException();
}
await connection.ExecuteAsync(accountSql, accountSqlParams);
await transaction.CommitAsync(token);
}
//....
}public async Task HandleAsync(DepositToAccountRequest request, CancellationToken token = default)
{
var account = await _repository.GetAsync(request.AccountId, token) ??
throw new EntityNotFoundException();
account.Deposit(request.Amount);
try
{
await _repository.UpdateAsync(account, request.Id, token);
}
catch (DuplicateRequestException)
{
//Ignore
}
}Can we optimize this? You probably don’t need to store every request ID since the dawn of time. Can you assume you never receive duplicate requests after some time? If that’s the case — you can have a process that periodically removes IDs from the history table, making it very lightweight.
What if a transaction isn’t an option?
Sometimes you don’t have an ACID-compliant database, making transactions not an option anymore. What you can do instead is store request IDs directly into your aggregate:
public sealed class Account
{
private readonly HashSet<Guid> _requestIds;
public Guid Id { get; }
public decimal Balance { get; private set; }
public Account(Guid id, decimal balance)
: this(id, balance, new HashSet<Guid>()) { }
//Constructor to restore the state
private Account(Guid id, decimal balance, HashSet<Guid> requestIds)
{
if (id == default)
throw new InvalidOperationException("Account id must be provided");
if (balance < 0)
throw new InvalidOperationException("Balance cannot be negative");
if (requestIds == null)
throw new ArgumentNullException(nameof(requestIds));
Id = id;
Balance = balance;
_requestIds = requestIds;
}
public void SetRequestId(Guid id)
{
if (_requestIds.Contains(id))
throw new InvalidOperationException("Duplicate request");
_requestIds.Add(id);
}
public void Withdraw(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot withdraw negative amount");
if (amount > Balance)
throw new InvalidOperationException("Cannot withdraw more than existing balance");
Balance -= amount;
}
public void Deposit(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot deposit negative amount");
Balance += amount;
}
}public async Task HandleAsync(DepositToAccountRequest request, CancellationToken token = default)
{
var account = await _repository.GetAsync(request.AccountId, token) ??
throw new EntityNotFoundException();
try
{
account.SetRequestId(request.Id);
}
catch (InvalidOperationException)
{
//Ignore
return;
}
account.Deposit(request.Amount);
await _repository.UpdateAsync(account, token);
}This doesn’t protect you against duplicates in highly concurrent scenarios. However, if you don’t send your request retries concurrently (in most cases, you shouldn’t), you don’t have to worry about that.
If the aggregate is changed frequently, storing every request ID might increase its size by quite a lot. But if you only care about the most recent requests (like with the database table before), you can optimize this by storing only a certain number of the most recent operations.
Once again, if you are one of the Event Sourcing folks, life sometimes is just a tiny bit better for you. Since you are already storing your aggregates as event streams, you can add a request ID to each event. Whenever you rebuild your aggregate, you can reconstruct the request history and validate a new request against it:
public abstract class EventSourcedAggregate : Entity
{
private readonly HashSet<Guid> _requestIds = new HashSet<Guid>();
private readonly List<IVersionedEvent> _events = new List<IVersionedEvent>();
private Guid _requestId = Guid.NewGuid();
protected EventSourcedAggregate(Guid id)
: base(id) { }
protected EventSourcedAggregate(Guid id, IEnumerable<IVersionedEvent> events)
: this(id)
{
foreach (var @event in events)
ApplyEvent(@event);
}
public int Version { get; private set; }
public IReadOnlyList<IVersionedEvent> GetUncommittedEvents()
{
return _events;
}
public void MarkEventsAsCommitted()
{
_events.Clear();
}
public void SetRequestId(Guid id)
{
if (_requestIds.Contains(id))
throw new InvalidOperationException("Duplicate request");
_requestId = id;
}
protected void Raise(VersionedEvent @event)
{
@event.RequestId = _requestId;
@event.SourceId = Id;
@event.Version = Version + 1;
_events.Add(@event);
ApplyEvent(@event);
}
private void ApplyEvent(IVersionedEvent @event)
{
this.AsDynamic().Apply(@event);
Version = @event.Version;
if (!_requestIds.Contains(@event.RequestId))
_requestIds.Add(@event.RequestId);
}
}public sealed class Account : EventSourcedAggregate
{
public decimal Balance { get; }
public Account(Guid id, IEnumerable<IVersionedEvent> events)
: base(id, events) { }
public Account(Guid id, decimal balance)
: base(id)
{
if (id == default)
throw new InvalidOperationException("Account id must be provided");
if (balance < 0)
throw new InvalidOperationException("Balance cannot be negative");
Raise(new AccountOpened(balance));
}
public void Withdraw(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot withdraw negative amount");
if (amount > _balance)
throw new InvalidOperationException("Cannot withdraw more than existing balance");
Raise(new WithdrawnFromAccount(amount));
}
public void Deposit(decimal amount)
{
if (amount < 0)
throw new InvalidOperationException("Cannot deposit negative amount");
Raise(new DepositedToAccount(amount));
}
private void Apply(AccountOpened evt)
{
_balance = evt.Balance;
}
private void Apply(WithdrawnFromAccount evt)
{
_balance -= evt.Amount;
}
private void Apply(DepositedToAccount evt)
{
_balance += evt.Amount;
}
}public async Task HandleAsync(DepositToAccountRequest request, CancellationToken token = default)
{
var account = await _repository.GetAsync(request, token);
try
{
account.SetRequestId(request.Id);
}
catch (InvalidOperationException)
{
//Ignore duplicate request
return;
}
account.Deposit(request.Amount);
await _repository.SaveAsync(account, token);
}There’s one more option if you are using Event Sourcing. It is based on generating deterministic UUIDs defined in RFC 4122. If your event store can guarantee unique event IDs, you can use a request ID to generate resulting event IDs deterministically. The only thing left is to catch a duplicate key exception since your event store is handling the rest:
public static class DeterministicGuid
{
public static Guid Create(Guid val)
{
return Create(val.ToString());
}
public static Guid Create(string val)
{
using var generator = new NameBasedGenerator(HashType.SHA1);
return generator.GenerateGuid(UUIDNameSpace.URL, val);
}
}public sealed class Repository<TEventSourcedAggregate> : IEventSourcedRepository<TEventSourcedAggregate> where TEventSourcedAggregate : EventSourcedAggregate
{
//....
public async Task SaveAsync(TEventSourcedAggregate aggregate, Guid requestId, CancellationToken token = default)
{
//....
var nextIdGenerationSource = requestId;
var eventsToSave = aggregate.GetUncommittedEvents().Select(evt =>
{
var id = DeterministicGuid.Create(nextIdGenerationSource);
nextIdGenerationSource = id;
var type = evt.GetType().Name;
var data = Serialize(evt);
return new EventData(id, type, data);
});
//....
}
//....
}Wow, you are still reading? It’s time to summarize!
There’s no right way to deal with idempotence — it depends solely on your business and technical requirements. Sometimes you might receive service requests that are idempotent naturally. At other times, if your aggregates don’t change very often, it could make sense to provide the current aggregate state in your requests to deal with duplicate requests. Maybe you have an ACID-compliant database? If that’s the case, you could potentially go with database transactions since it is reasonably simple to implement, can be used with all of your requests, and doesn’t leak technical details into the domain. Do you use the Event Sourcing pattern with an event store that guarantees unique event IDs (i.e., SQL Server)? Then you could avoid transactions by generating deterministic event IDs. What if your event store doesn’t ensure unique event IDs (i.e., Greg Young’s Event Store)? You could store request IDs in your aggregates instead. Not to mention that there are likely other ways used in specific scenarios that I have not covered.
Thank you for reading. I’d love to hear how you handle idempotent requests.
