Event sourcing and CQRS: storing what happened, not just the current state

Key points

• An event records a business fact in the past tense, such as Withdrawn, and is never updated; an entity's state is a fold over its stream of events.
• Writes append with an expected version, so two writers that loaded the same state cannot both succeed: optimistic concurrency without locks.
• Projections read the log in order and build read models for queries. They lag behind the log, so they must be idempotent and the UI must tolerate stale reads.
• Snapshots shorten replay for long streams; they are a cache of state, and the events stay the source of truth.
• Old events never change, so new code must read old shapes: tolerant readers, version numbers and upcasters that convert on read.
• Martin Fowler and Microsoft both warn that these patterns add real complexity. Use them in the parts of a system that need history and audit, not everywhere.

Event sourcing: the log is the source of truth

A typical application stores the current state. When a customer withdraws money, a row's balance goes from 100 to 70 and the 100 is gone; any history has to be written separately, to an audit table that may or may not agree with the data. Martin Fowler's 2005 description of event sourcing turns this around: "Capture all changes to an application state as a sequence of events." The events are stored, and the state is derived from them.

Each entity, an account or an order, has its own stream: the ordered events that changed it, such as AccountOpened, Deposited and Withdrawn. Events are named in the past tense because they are facts that already happened, and they are only ever appended, never updated or deleted. To load an entity, you read its stream and apply each event in order. Microsoft's Azure Architecture Center calls this rehydration.

Fowler lists what the log gives you. You can throw away the current state and rebuild it completely by replaying events. You can answer temporal queries, the state as of any past moment, by replaying up to that point. And when a past event turns out to be wrong, you can recompute everything after it. Microsoft adds a design rule that decides whether the log is worth keeping: record the business intent, not just the result. Its example contrasts an event saying two seats were reserved with one saying the remaining seats changed to 42; the first tells you what happened, the second is a change log with no meaning.

An event-sourced aggregate, with replay and optimistic concurrency

In domain-driven design terms, the unit that enforces rules and owns a stream is an aggregate. A minimal account aggregate and an in-memory event store, in plain Python:

class ConcurrencyError(Exception): pass

class EventStore:
    def __init__(self):
        self.log = []  # (position, stream_id, event), in commit order
    def read(self, stream_id):
        return [e for _, s, e in self.log if s == stream_id]
    def append(self, stream_id, events, expected_version):
        current = len(self.read(stream_id))
        if current != expected_version:  # optimistic concurrency: someone wrote first
            raise ConcurrencyError(f"{stream_id} is at {current}, expected {expected_version}")
        for e in events:
            self.log.append((len(self.log) + 1, stream_id, e))

class Account:
    def __init__(self, history):
        self.balance, self.is_open, self.version = 0, False, 0
        for e in history:  # replay: current state is a fold over past events
            self.apply(e)
    def apply(self, e):
        if e["type"] == "AccountOpened": self.is_open = True
        elif e["type"] == "Deposited": self.balance += e["amount"]
        elif e["type"] == "Withdrawn": self.balance -= e["amount"]
        self.version += 1
    def withdraw(self, amount):  # a command: check the rules, return new events
        if not self.is_open or amount > self.balance:
            raise ValueError("rejected: account not open or balance too low")
        return [{"type": "Withdrawn", "amount": amount}]

A command such as withdraw loads the current state, checks the business rule, and returns new events rather than changing anything. The store appends them only if the stream is still at the version the command saw. Greg Young's CQRS Documents describe the same check on a relational events table: each event carries a version number that is sequential within its aggregate, and an insert succeeds only if the expected version matches the current one. KurrentDB, formerly EventStoreDB, exposes it as the expected state of an append: any, no stream yet, stream exists, or a specific revision.

store = EventStore()
store.append("acct-42", [{"type": "AccountOpened"}, {"type": "Deposited", "amount": 100}], 0)

acct = Account(store.read("acct-42"))  # load by replaying the stream
store.append("acct-42", acct.withdraw(30), expected_version=acct.version)

try:  # a second writer still holding the account at version 2
    store.append("acct-42", acct.withdraw(50), expected_version=acct.version)
except ConcurrencyError as err:
    print(err)  # acct-42 is at 3, expected 2

print(Account(store.read("acct-42")).balance)      # 70
print(Account(store.read("acct-42")[:2]).balance)  # 100: the state as of event 2

The second append fails because the stream moved on since the command read it. The caller reloads, re-runs the rule against the new balance and retries, or reports the conflict. Without that check, two concurrent withdrawals could each see a balance of 100 and both succeed. The last line is a temporal query: replaying the first two events gives the state before the withdrawal.

Projections and read models

Replaying a stream answers "what is the state of account 42" well and "which accounts are overdrawn" not at all. Microsoft notes that an event store offers no general query mechanism; you can read the stream for one entity, and that is it. Queries go to read models instead: tables, documents or search indexes shaped for one screen or report, kept up to date by projections that read the events in commit order.

A projection must survive restarts and redelivery. Event delivery to consumers is typically at least once, so a projection will see some events twice; Microsoft's guidance is to track the last processed position for each consumer and skip duplicates, or make every update safe to repeat. The simplest reliable form writes the read-model change and the checkpoint in one database transaction. Continuing the example, with SQLite standing in for the read database:

import sqlite3
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE balances (account_id TEXT PRIMARY KEY, balance INTEGER, moves INTEGER);
CREATE TABLE checkpoints (projection TEXT PRIMARY KEY, position INTEGER);
INSERT INTO checkpoints VALUES ('balances', 0);
""")
SIGN = {"Deposited": 1, "Withdrawn": -1}
def project(log):
    with db:  # the read-model rows and the checkpoint commit together, or not at all
        (done,) = db.execute("SELECT position FROM checkpoints WHERE projection = 'balances'").fetchone()
        for position, account_id, e in log:
            if position <= done:
                continue  # already applied: a redelivery, a restart or a replay
            if e["type"] == "AccountOpened":
                db.execute("INSERT INTO balances VALUES (?, 0, 0)", (account_id,))
            elif e["type"] in SIGN:
                db.execute("UPDATE balances SET balance = balance + ?, moves = moves + 1 "
                           "WHERE account_id = ?", (SIGN[e["type"]] * e["amount"], account_id))
            done = position
        db.execute("UPDATE checkpoints SET position = ? WHERE projection = 'balances'", (done,))

project(store.log)
project(store.log)  # delivered twice: no double counting
print(db.execute("SELECT * FROM balances").fetchall())  # [('acct-42', 70, 2)]

Because the checkpoint and the rows commit together, a crash leaves either both or neither, and a rerun skips what is already applied. The same idea, with dedupe records and unique constraints for consumers that cannot share a transaction, is in idempotency and job queues. The payoff of projections is that read models are disposable: drop one, replay the log from position zero, and you have it back, or build a new one for a report nobody had thought of when the events were written.

The cost is that read models lag the log. A user who withdraws money and immediately opens the statement page may not see the withdrawal yet. Common answers are to return the new version or log position from the command and have the read side wait until its checkpoint passes it, to read that user's own data from the write model, or to update the UI optimistically. Which guarantee you need, and what it costs, is the subject of consistency models and the CAP theorem.

Snapshots

An account with ten events replays instantly. A stream with a hundred thousand does not. Greg Young's documents describe rolling snapshots: a serialized copy of an aggregate's state as of a given event, stored beside the stream, so loading means reading the latest snapshot and replaying only the events after it. Microsoft suggests taking one every N events and weighing the storage against the replay time saved.

• A snapshot is a cache. The stream stays the source of truth, and any snapshot can be thrown away and rebuilt from it.
• Store the stream version with the snapshot, and replay strictly from the version after it.
• Version the snapshot format too. When the aggregate's fields change, discard old snapshots and rebuild them rather than upcasting them.
• Measure before adding them. Many aggregates never grow long enough to need one, and a stream that keeps growing for years may instead be a sign the aggregate is too broad, for example an account that could close a stream per statement period.

CQRS: separate models for commands and queries

Command Query Responsibility Segregation grew out of Bertrand Meyer's command-query separation principle, which says a method should either change state or return data, not both. Greg Young's CQRS Documents explain the step up: in CQRS, the object that served both is split in two, one holding the commands and one holding the queries. Fowler, who first heard the pattern from Young, describes it as using one model to update information and a different model to read it.

The write model validates commands and enforces invariants. The read models hold whatever shape the screens need, with no business logic. Microsoft's CQRS page describes two levels: separate models over one shared database, and separate databases for reads and writes, which lets each scale and use its own storage technology but has to be kept in sync. When the write side publishes events for that, Microsoft recommends the transactional outbox, so the state change and the event commit atomically, and an idempotent consumer on the read side.

CQRS does not require event sourcing, and event sourcing does not strictly require CQRS, but they fit together. With event sourcing, the event store is the write model and the single source of truth, and projections are the read models. Without CQRS, an event-sourced system can answer only the queries it can answer by replaying one stream at a time.

Event schema evolution and upcasting

Events are kept forever and never edited, but the code that reads them keeps changing. Fields get renamed, an event type is split, a unit changes. Microsoft's Event Sourcing page lists four strategies, to be used alone or together:

• Tolerant deserialization: readers ignore fields they do not know and use defaults for missing ones. It covers additive changes, such as a new optional field, with no transformation at all.
• Event versioning: each event carries a version, in its metadata or its type name, and readers pick the handling for that version.
• Upcasting: functions convert old shapes to the current one as events are read, chained version by version, so the domain code only ever sees the latest shape and the stored events stay untouched.
• In-place migration: rewriting stored events to the new schema. It breaks immutability and the audit trail, so treat it as a last resort.

An upcaster for a renamed event, applied on read to the aggregate from earlier:

def upcast(e):  # runs on read; the stored event is never rewritten
    if e["type"] == "MoneyAdded" and e.get("v") == 1:  # v1 name and field, before a rename
        return {"type": "Deposited", "v": 2, "amount": e["sum"]}
    return e

stored = [{"type": "AccountOpened"}, {"type": "MoneyAdded", "v": 1, "sum": 25}]
print(Account([upcast(e) for e in stored]).balance)  # 25

Upcasting fixes shapes, not facts. If a bug wrote events with wrong values, they stay in the log; the usual correction is a compensating event that records the reversal, after which projections are rebuilt. Greg Young's documents make the same point about deletion in general: an event cannot be made not to have happened, so a removal is modeled as a new event. Keep a catalog of every event type and version, and treat a change to an event's shape with the care you would give a public API.

CRUD vs event sourcing, and when not to use it

The warnings come from the pattern's own advocates. Fowler writes that for most systems CQRS adds risky complexity, that a domain which suits it is the minority case, and that it belongs only in specific portions of a system, a bounded context in domain-driven design terms, not across the whole. For a domain that does not fit but has heavy queries, he suggests a separate reporting database as the simpler option. Microsoft's Event Sourcing page opens with a similar caution: the pattern changes how you store data, handle concurrency, evolve schemas and query state, is costly to migrate to or away from, and for most systems traditional data management is sufficient.

Microsoft lists where it is a poor fit: plain CRUD with no need for audit, replay or history; prototypes, MVPs and short-lived systems; anything that needs views that are consistent in real time; mostly static reference data; and teams with no experience of event-driven systems. Fowler's original article adds the hard part people discover late: replaying events must not send the same messages to external systems again, and external systems queried during the original run may answer differently during a replay. His remedy is to put every external call behind a gateway that can be switched off or fed recorded responses during replay.

The strongest cases are domains that already think in events and need their history: ledgers and balances, orders and fulfillment, bookings, inventory movements, workflows with an audit requirement. Even there, Microsoft suggests applying it selectively, to an order pipeline say, while user profiles and settings stay CRUD. If you are weighing service boundaries at the same time, see monolith vs microservices.

Deleting personal data from an immutable log

The GDPR's right to erasure, and similar rights in other privacy laws, collide with a store whose first rule is that nothing is deleted. Microsoft names the tension directly and gives two designs. This is not legal advice; what counts as erasure for your data is a question for your counsel, and GDPR for apps covers the obligations themselves.

• Keep personal data out of events. Store names, emails and addresses in an ordinary table that can be deleted, and put only an identifier in the events. The log stays intact and holds nothing personal.
• Crypto-shredding, when personal data must sit inside events: encrypt those fields with a key per data subject, kept in a separate key store. To erase a person, delete their key; the events remain, structurally intact, but the encrypted fields can no longer be read.

Crypto-shredding has edges. Every read and write pays for encryption, and the key store becomes critical infrastructure: lose it and you have shredded everyone. Projections, snapshots, search indexes, caches, analytics copies and downstream consumers may hold the plaintext, so erasure has to reach them too, typically by rebuilding them after the key is gone. Backups of the key store need the same deletion policy. Design this before the first event is written; retrofitting it means rewriting the log, which is exactly what the pattern is meant to avoid.

Running it: event stores, replays and the tooling you need

An event store can be a purpose-built database or a table in a general-purpose one. Microsoft notes that purpose-built stores give you stream reads, optimistic concurrency and often snapshots out of the box, while a relational database needs you to build them: an append-only table with a unique constraint on stream id and version, plus a global position for projections to follow. It also warns against using a message broker such as Apache Kafka as the event store, since brokers typically lack per-entity stream reads and optimistic concurrency; they work well for fanning events out to consumers. KurrentDB, as one example of a purpose-built store, also skips an event appended again with the same id shortly after the first, so a retried write does not usually store it twice; its documentation describes the limits.

• Projection lag: monitor the distance between the log's head position and each projection's checkpoint, in events and in seconds, and alert when it grows.
• Rebuilds: build a new version of a read model alongside the old one from position zero, switch reads when it catches up, then drop the old one. Measure how long a full replay takes before you depend on it.
• Poison events: when a projection cannot handle an event, stop and alert rather than skip it silently, or the read model drifts from the log for good.
• Side effects: keep projections, which only write read models, apart from handlers that send emails or call other systems, and make sure replays reach only the first kind. See notification system design and webhook delivery system design for the second.
• Testing: Microsoft recommends given-when-then tests: given past events, when a command runs, then assert the new events. Add integration tests for projections, idempotency and every upcaster.
• Inspection: you will need a way to browse a stream, see an entity at a past version, and replay a single stream into a scratch read model while debugging.

Read models are usually ordinary tables, so the usual tuning applies; see database indexing explained. Projections that run as workers have the same failure modes as any queue consumer; see background jobs and queues.

If you are building a web app that needs a ledger, an audit trail or history you can query, see our web app development service: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.