The Promised Land of Event Sourcing

If you have ever developed a software system in an inherently complex business domain, I bet you know how complicated it could get. As a software engineer, I had an epiphany about dealing with complex business domains when I heard about DDD (domain-driven design) for the first time. A few years later, working in the financial field, I met some brilliant people using a DDD pattern called Event Sourcing. It took me quite a few days to get a grasp of it. It was so different yet so elegant. My only regret now is that I didn't hear about Event Sourcing earlier.

I spent quite some time reading books and articles, having discussions with experts in this field, and implementing the pattern myself. As a result, I wrote this guide, which, I believe, is one of the quickest ways to start with Event Sourcing.

What is Event Sourcing?

We're used to the "classic" way of storing all application state changes as the latest state of our business model aggregates, such as an Account or an Order. It's not unusual to find those kinds of systems to be CRUD-based and built around relational databases. Unlike the "classic" approach, when using the Event Sourcing pattern, we persist an application state changes as a sequence of state-changing events — immutable business-related information about things that happened in the past. In a nutshell, Event Sourcing is just a different way to store data. Let's take a hypothetical payments related application, for example. Whenever we make a new payment, a new PaymentMade event is appended to the Account aggregate events list and persisted within a single atomic operation. Later, the Account balance can be reconstructed by replaying persisted PaymentMade events.

Event Sourcing allows tracking all business-related things that happen within an application, which is often extremely important in specific domains. I can't imagine a bank that would not allow clients to view their transactions. At the same time, the ability to check what happened to a brand-new smartphone that was supposed to arrive on your doorstep one week ago is something that most people would certainly appreciate. Moreover, we can use events to reconstruct past aggregate states to solve complicated issues or roll back unwanted changes. 

If you're like me before I knew anything about Event Sourcing, you should have more questions than answers by now. 

Why would one use Event Sourcing?

Event Sourcing is by no means the silver bullet for software systems. Yet it can be an incredible tool if you know all the whens and the hows.

Event Sourcing provides a complete audit trail of business transactions. You can't delete a transaction or change it once it has happened. You can use those stored transactions to determine the system's state at any previous point in time by querying the events up to that point in time. It enables you to answer historical questions from the business about the system.

Troubleshooting is another fantastic Event Sourcing advantage. You can troubleshoot problems in a production system by copying the production event store and replaying it in a test environment. By troubleshooting, you might discover source code errors. Rather than performing risky manual data adjustments, you can fix the code and replay the event stream letting the system recalculate values correctly based on the new code version.

Event Sourcing often leads to better performance and scalability than complex relational storage models since you only use append-only operations for small immutable event objects. Having append-only operations allows you to reap the benefits of databases such as Cassandra. Moreover, Event Sourcing is usually combined with CQRS, which could be a crazy improvement compared to the "classic" relational storage models.

Event Sourcing is an excellent fit in distributed systems (i.e., microservices architecture). You can publish events to notify other subsystems of changes to the application's state in a considerably decoupled manner. Even though, strictly speaking, you can do that without Event Sourcing, having an event-based system makes it a lot easier.

A single one of those benefits could be handy on its own, but having them all combined often makes Event Sourcing a no-brainer.

Yes, yes, yes, but where is the code?

I wouldn't be surprised if, at this point, you weren't completely sold on the idea. The concept might seem difficult to grasp and evaluate whether all of these benefits are real or I am just a snake oil salesman trying to sell you my goods. Hopefully, the code will provide you with some answers.

The code examples are written in C# because its syntax is relatively simple and should not be challenging to understand for most developers. However, you could implement the same thing in many different programming languages, even the functional ones (see resources section).

Let's define the Account aggregate model having some basic operations:

public sealed class Account
{
    private Guid Id;
    private decimal _balance;

    public static Account Open(Guid id, decimal balance)
    {

    }

    public void Withdraw(decimal amount)
    {

    }

    public void Deposit(decimal amount)
    {
  
    }
}

Instead of going for the "classic" way, we need to split each of our methods into two parts : validating the data and raising events and applying events. This separation is crucial since we will restore the Account from our event store by applying events.

public abstract class EventSourcedAggregate
{
    private readonly List<Event> _events = new List<Event>();

    public Guid Id { get; protected set; }
    public int Version { get; private set; }

    public void LoadFromHistory(IEnumerable<Event> events)
    {
        foreach (var @event in events)
            ApplyEvent(@event);
    }

    public IReadOnlyList<Event> GetUncommittedEvents()
    {
        return _events;
    }

    public void MarkEventsAsCommitted()
    {
        _events.Clear();
    }

    protected void Raise(Event @event)
    {
        @event.Version = Version + 1;
        _events.Add(@event);
        ApplyEvent(@event);
    }

    public void ApplyEvent(Event @event)
    {
        this.AsDynamic().Apply(@event);
        Version++;
    }
}

public sealed class Account : EventSourcedAggregate
{
    private decimal _balance;

    public static Account Open(Guid id, decimal balance)
    {
        if (balance < 0)
            throw new InvalidOperationException("Balance cannot be negative.");

        var account = new Account();
        account.Raise(new AccountOpened(id, balance));
        return account;
    }

    public void Withdraw(decimal amount)
    {
        if (amount <= 0)
            throw new InvalidOperationException("Amount must be positive.");

        if (amount > _balance)
            throw new InvalidOperationException("Cannot withdraw more than balance.");

        Raise(new WithdrawnFromAccount(Id, amount));
    }

    public void Deposit(decimal amount)
    {
        if (amount <= 0)
            throw new InvalidOperationException("Amount must be positive.");

        Raise(new DepositedToAccount(Id, amount));
    }

    private void Apply(AccountOpened @event)
    {
        Id = @event.AccountId;
        _balance = @event.Balance;
    }

    private void Apply(WithdrawnFromAccount @event)
    {
        _balance -= @event.Amount;
    }

    private void Apply(DepositedToAccount @event)
    {
        _balance += @event.Amount;
    }
}

public abstract class Event
{
    public int Version { get; set; }
}

public sealed class AccountOpened : Event
{
    public Guid AccountId { get; }
    public decimal Balance { get; }

    public AccountOpened(Guid accountId, decimal balance)
    {
        AccountId = accountId;
        Balance = balance;
    }
}

public sealed class DepositedToAccount : Event
{
    public Guid AccountId { get; }
    public decimal Amount { get; }

    public DepositedToAccount(Guid accountId, decimal amount)
    {
        AccountId = accountId;
        Amount = amount;
    }
}

public sealed class WithdrawnFromAccount : Event
{
    public Guid AccountId { get; }
    public decimal Amount { get; }

    public WithdrawnFromAccount(Guid accountId, decimal amount)
    {
        AccountId = accountId;
        Amount = amount;
    }
}

Let's add some command handlers to orchestrate our application:

public sealed class AccountCommandHandlers
{
    private readonly IEventSourcedRepository<Account> _repository;

    public AccountCommandHandlers(IEventSourcedRepository<Account> repository)
    {
        _repository = repository;
    }

    public async Task HandleAsync(OpenAccount command)
    {
        var account = Account.Open(command.AccountId, command.Balance);
        await _repository.SaveAsync(account);
    }

    public async Task HandleAsync(DepositToAccount command)
    {
        var account = await _repository.GetAsync(command.AccountId) ??
            throw new EntityNotFoundException(nameof(Account), command.AccountId);

        account.Deposit(command.Amount);
        await _repository.SaveAsync(account);
    }

    public async Task HandleAsync(WithdrawFromAccount command)
    {
        var account = await _repository.GetAsync(command.AccountId) ??
            throw new EntityNotFoundException(nameof(Account), command.AccountId);

        account.Withdraw(command.Amount);
        await _repository.SaveAsync(account);
    }
}

public sealed class OpenAccount
{
    public Guid AccountId { get; }
    public decimal Balance { get; }

    public OpenAccount(Guid accountId, decimal balance)
    {
        AccountId = accountId;
        Balance = balance;
    }
}

public sealed class DepositToAccount
{
    public Guid AccountId { get; }
    public decimal Amount { get; }

    public DepositToAccount(Guid accountId, decimal amount)
    {
        AccountId = accountId;
        Amount = amount;
    }
}

public sealed class WithdrawFromAccount
{
    public Guid AccountId { get; }
    public decimal Amount { get; }

    public WithdrawFromAccount(Guid accountId, decimal amount)
    {
        AccountId = accountId;
        Amount = amount;
    }
}

This is how we could implement the repository if we were to use Greg Young's Event Store:

public sealed class EventStore : IEventStore
{
    private readonly IEventStoreConnection _connection;

    public EventStore(IEventStoreConnection connection)
    {
        _connection = connection;
    }

    public async IAsyncEnumerable<Event> GetAggregateEventsAsync<TEventSourcedAggregate>(Guid aggregateId)
        where TEventSourcedAggregate : EventSourcedAggregate
    {
        var streamName = GetAggregateStreamName<TEventSourcedAggregate>(aggregateId);
        StreamEventsSlice currentSlice;
        var nextSliceStart = (long)StreamPosition.Start;

        do
        {
            currentSlice = await _connection.ReadStreamEventsForwardAsync(streamName, nextSliceStart, 10, false);
            nextSliceStart = currentSlice.NextEventNumber;

            foreach (var resolvedEvent in currentSlice.Events)
                yield return EventStoreSerializer.Deserialize(resolvedEvent);

        } while (!currentSlice.IsEndOfStream);
    }

    public async Task SaveAggregateEventsAsync<TEventSourcedAggregate>(Guid aggregateId, IEnumerable<Event> events, int expectedVersion)
        where TEventSourcedAggregate : EventSourcedAggregate
    {
        var streamName = GetAggregateStreamName<TEventSourcedAggregate>(aggregateId);
        var eventsToSave = events.Select(@event => EventStoreSerializer.Serialize(@event));
        await _connection.AppendToStreamAsync(streamName, expectedVersion, eventsToSave);
    }

    private static string GetAggregateStreamName<TEventSourcedAggregate>(Guid aggregateId)
        where TEventSourcedAggregate : EventSourcedAggregate
    {
        return $"{typeof(TEventSourcedAggregate).Name}-{aggregateId}";
    }
}

public sealed class EventSourcedRepository<TEventSourcedAggregate> : IEventSourcedRepository<TEventSourcedAggregate>
       where TEventSourcedAggregate : EventSourcedAggregate, new()
{
    private readonly IEventStore _eventStore;

    public EventSourcedRepository(IEventStore eventStore)
    {
        _eventStore = eventStore;
    }

    public async Task SaveAsync(TEventSourcedAggregate aggregate)
    {
        var uncommittedEvents = aggregate.GetUncommittedEvents();
        if (!uncommittedEvents.Any())
            return;

        var originalVersion = aggregate.Version - uncommittedEvents.Count;
        var expectedVersion = originalVersion == 0
            ? ExpectedVersion.NoStream
            : originalVersion - 1;

        try
        {
            await _eventStore.SaveAggregateEventsAsync<TEventSourcedAggregate>(aggregate.Id, uncommittedEvents, expectedVersion);
        }
        catch (WrongExpectedVersionException e)
            when (e.ExpectedVersion == ExpectedVersion.NoStream)
        {
            throw new DuplicateKeyException(aggregate.Id);
        }

        aggregate.MarkEventsAsCommitted();
    }

    public async Task<TEventSourcedAggregate?> GetAsync(Guid id)
    {
        TEventSourcedAggregate? aggregate = null;

        await foreach (var @event in _eventStore.GetAggregateEventsAsync<TEventSourcedAggregate>(id))
            (aggregate ??= new TEventSourcedAggregate()).ApplyEvent(@event);

        return aggregate;
    }
}

This is what the stored data looks like in the Event Store:

In a nutshell, the data is a complete log of past events. Now, whenever the business throws you a tricky "what happened" question, you can give a very detailed answer, feel like the smartest person in the room, and as a bonus, hopefully, get a raise.

Have you noticed the Version? The Version enables us to time-travel in the realm of Event Sourcing. To get a specific Account version, we can query related events from the database up to that version and use them to restore the Account. If using a version to query the data is not your cup of tea, you could use a Timestamp from your event store instead.

That's it! That's all there is! Well… not really… but by now, you should have a decent understanding and know where to start!

Event Sourcing is not all fun and games

The Event Sourcing pattern has a lot of good parts. However, there are some bad ones as well. If you think you need Event Sourcing — please read on.

Most of the developers are not familiar enough with the Event Sourcing concept. Having to explain Event Sourcing to less experienced colleagues takes much time. Moreover, explaining only Event Sourcing is usually not enough. DDD, CQRS, and the eventual consistency model often go hand-in-hand with Event Sourcing. It's a very steep learning curve!

Most of the domains are not that complex. There, I said it. Event sourcing, on the other hand, is very complex. You have to consider snapshotting algorithms, not being able to change the data in the database in case there's an urgent production issue, handling event schema changes, and that's just the tip of the iceberg. Using Event Sourcing for a simple CRUD-based application is like using a chainsaw instead of a butter knife to make a sandwich — they both cut things, but one of them is a bit of an overkill. I will let you decide which one. 

Even though the Event Sourcing pattern is excellent, you must thoroughly consider whether it's worth using it for your specific problem.

Frequently asked questions

As it turns out, Event Sourcing isn't as straightforward as it might seem at first sight. However, I find that some of the questions about Event Sourcing are asked more frequently than others.

Does Event Sourcing hurt the performance?

If done well, Event Sourcing doesn't hurt the performance. It's often quite the opposite since Event Sourcing usually comes together with the CQRS pattern, which solves many performance issues. Of course, there are cases where aggregates tend to have long and complex lifetimes (having many events), making the querying part very slow, but that's solvable with snapshots.

What's the difference between domain and integrational events?

People often split events into two categories — integration and domain, especially when working with microservices-based architecture. These categories are very similar since both events are state change notifications. However, the reasoning behind them is different. Integration events notify other services about the state changes inside your service. Therefore, they rarely change and might contain additional information that domain events don't. Domain events notify aggregates inside your domain boundaries (and restore the state in Event Sourcing). Therefore, they can be changed easier and have only a minimal amount of relevant information.

What data stores can I use to implement an event store?

We can implement an event store using SQL Server, MySQL, Cassandra, Event Store, and many other data stores. We should be able to query events in the same order as we produced them for a specified aggregate ID, save events atomically, and utilize snapshotting. Other event store capabilities are mostly related to functional and non-functional application requirements. It's worth mentioning that many debates on the internet discuss whether we could use streaming platforms such as Kafka to implement an event store. I've seen it work for specific business problems where there wasn't much data stored and heavy in-memory caching was used. However, in most cases, you're better off with a different data store since you would likely need to query events for a specified aggregate ID in a reasonable amount of time.

Can I update stored events?

One day our event schemas are inevitably going to change. Therefore, it's crucial to think about dealing with it ahead of time. It becomes very tempting to change the stored data when the inevitable happens. However, most of the time, we shouldn't. By doing that, we're messing with historical data, which eliminates any guarantees of the data being authentic and makes it less accurate. What we should do instead is to version our event schemas. Event schema versioning allows us to adjust the code according to new business requirements while keeping the stored data unchanged. We could hide the mapping between old and new schemas somewhere in the infrastructure layer of our application.

Can I use Event Sourcing without CQRS?

While it's possible to use Event Sourcing without CQRS, it's often not very practical. We usually need to query data from multiple aggregates combined. Moreover, it is relatively simple to build custom query models by subscribing to produced events. Event Sourcing and CQRS combination is like a pair of socks — you could go with one sock instead of two, but why on earth would you do that?

Resources

If you're interested in learning more about Event Sourcing, here are some of the best resources you should start with.

Exploring CQRS and Event Sourcing: A journey into high scalability, availability, and maintainability with Windows Azure

I can't stress enough how good this book is! This free e-book is hands down the single best Event Sourcing resource available right now. It covers a team's journey to CQRS and Event Sourcing in a very detailed fashion with plenty of code examples and explanations of every decision. All of the code used in the book is available on GitHub.

Event Storming

Event Storming and Event Sourcing are entirely different things. However, they work exceptionally well together. Using the Event Storming method, we model our business domain on a whiteboard in aggregates, commands, and events. We can easily transform the whiteboard model into the source code if done right.

Simple CQRS

Simple CQRS is a sample project created by Greg Young to illustrate CQRS and Event Sourcing patterns. While the code is not quite production-ready, the complexity is extremely low, making it an excellent example for beginners. 

CQRS & EventSourcing

CQRS & EventSourcing is a sample project I created to illustrate CQRS and Event Sourcing patterns. The implementation is slightly more complex than Simple CQRS. However, it uses EventSoreDB and MySQL databases and is somewhat closer to what you might see in the production environment. I recommend running the project and playing around to get the feel of Event Sourcing.

Domain-Driven Design, Event Sourcing and CQRS with F# and EventStore

If you're not a big fan of OOP, you might enjoy trying the pattern in a functional language such as F#.

Final words

To use or not to use Event Sourcing, that is the question. The Event Sourcing pattern brings more complexity than the classic "store the current state" approach. On the other hand, it tends to bring many fantastic benefits such as performance gain, restoring the past states of your application, doing state rollbacks, and, most importantly, telling exactly what happened if things were to go south. If used in the right place at the right time, Event Sourcing will be your best friend that you wish you had met sooner.

Thank you for reading. I'd love to hear about your experiences with Event Sourcing.