K
Klivvr Blog

Eventual Consistency: Embracing Distributed Data

A deep dive into eventual consistency in distributed event-driven systems, covering consistency models, conflict resolution strategies, and TypeScript implementations for building reliable systems that embrace asynchronous data propagation.

technical11 min readBy Klivvr Engineering
Share:

Eventual consistency is one of the most misunderstood concepts in distributed systems. Critics dismiss it as "inconsistency by another name," while proponents treat it as a silver bullet that solves all scalability problems. The truth, as always, lies somewhere in between. Eventual consistency is a deliberate trade-off --- you accept temporary inconsistencies in exchange for higher availability, lower latency, and better partition tolerance. Understanding when this trade-off is appropriate, and how to manage its consequences, is essential for building reliable event-driven systems.

In this article, we examine what eventual consistency actually means, explore the patterns for managing it effectively, and provide TypeScript implementations that you can apply in real-time event processing platforms like Starburst.

Understanding Consistency Models

Before diving into eventual consistency, it helps to understand the spectrum of consistency models available.

Strong consistency guarantees that after a write completes, every subsequent read returns the updated value. This is the default expectation in single-node databases and is what most developers assume when they write application code. The cost is latency and availability --- achieving strong consistency across multiple nodes requires coordination, which takes time and fails if nodes are unreachable.

Eventual consistency guarantees that if no new updates are made, all replicas will eventually converge to the same value. There is no guarantee about how long this takes, and during the convergence window, different consumers may see different states.

Causal consistency sits between the two. It guarantees that if operation A causally precedes operation B, then every node that sees B will also see A. Operations that are not causally related may be seen in different orders.

// Modeling different consistency levels in TypeScript
enum ConsistencyLevel {
  Strong = "strong",
  Causal = "causal",
  Eventual = "eventual",
}
 
interface ReadOptions {
  consistency: ConsistencyLevel;
  maxStalenessMs?: number; // For eventual reads
  causalToken?: string;    // For causal reads
}
 
interface ConsistencyAwareStore<T> {
  write(key: string, value: T): Promise<WriteResult>;
  read(key: string, options: ReadOptions): Promise<ReadResult<T>>;
}
 
interface WriteResult {
  version: number;
  causalToken: string;
  timestamp: Date;
}
 
interface ReadResult<T> {
  value: T | null;
  version: number;
  staleness: number; // Milliseconds behind the latest write
  causalToken: string;
}
 
// Implementation that routes to different read strategies
class MultiConsistencyStore<T> implements ConsistencyAwareStore<T> {
  constructor(
    private primary: DataStore<T>,
    private replicas: DataStore<T>[],
    private causalTracker: CausalTracker
  ) {}
 
  async write(key: string, value: T): Promise<WriteResult> {
    const result = await this.primary.write(key, value);
 
    // Asynchronously replicate to other nodes
    this.replicateAsync(key, value, result.version);
 
    return result;
  }
 
  async read(key: string, options: ReadOptions): Promise<ReadResult<T>> {
    switch (options.consistency) {
      case ConsistencyLevel.Strong:
        return this.primary.read(key);
 
      case ConsistencyLevel.Causal:
        return this.readCausal(key, options.causalToken);
 
      case ConsistencyLevel.Eventual:
        return this.readEventual(key, options.maxStalenessMs);
    }
  }
 
  private async readCausal(
    key: string,
    causalToken?: string
  ): Promise<ReadResult<T>> {
    if (!causalToken) {
      // No causal dependency, read from any replica
      return this.readFromReplica(key);
    }
 
    const requiredVersion = this.causalTracker.getVersion(causalToken);
 
    // Try replicas first, fall back to primary if needed
    for (const replica of this.replicas) {
      const result = await replica.read(key);
      if (result.version >= requiredVersion) {
        return result;
      }
    }
 
    // Replicas are behind; read from primary
    return this.primary.read(key);
  }
 
  private async readEventual(
    key: string,
    maxStalenessMs?: number
  ): Promise<ReadResult<T>> {
    const result = await this.readFromReplica(key);
 
    if (maxStalenessMs && result.staleness > maxStalenessMs) {
      // Too stale, escalate to primary
      return this.primary.read(key);
    }
 
    return result;
  }
 
  private async readFromReplica(key: string): Promise<ReadResult<T>> {
    // Simple round-robin selection
    const index = Math.floor(Math.random() * this.replicas.length);
    return this.replicas[index].read(key);
  }
 
  private async replicateAsync(
    key: string,
    value: T,
    version: number
  ): Promise<void> {
    const promises = this.replicas.map((replica) =>
      replica.write(key, value).catch((error) => {
        console.error(`Replication to replica failed:`, error);
      })
    );
 
    // Fire and forget
    Promise.allSettled(promises);
  }
}
 
interface DataStore<T> {
  write(key: string, value: T): Promise<WriteResult>;
  read(key: string): Promise<ReadResult<T>>;
}
 
interface CausalTracker {
  getVersion(causalToken: string): number;
  createToken(version: number): string;
}

Conflict Resolution Strategies

In an eventually consistent system, conflicts are inevitable. Two services may update the same entity concurrently, and the system must resolve the conflict. There are several well-established strategies.

Last-writer-wins (LWW) is the simplest approach. Each update carries a timestamp, and the update with the latest timestamp wins. This is easy to implement but can lead to data loss if timestamps are not synchronized precisely.

// Last-Writer-Wins conflict resolution
interface Timestamped<T> {
  value: T;
  timestamp: number;
  nodeId: string;
}
 
function resolveLastWriterWins<T>(
  existing: Timestamped<T>,
  incoming: Timestamped<T>
): Timestamped<T> {
  if (incoming.timestamp > existing.timestamp) {
    return incoming;
  }
 
  if (
    incoming.timestamp === existing.timestamp &&
    incoming.nodeId > existing.nodeId
  ) {
    // Tie-break using node ID for deterministic resolution
    return incoming;
  }
 
  return existing;
}

Merge functions are domain-specific resolution strategies that combine conflicting updates rather than choosing one. This works well for data structures like counters, sets, and maps.

// Conflict-free Replicated Data Types (CRDTs)
 
// G-Counter: A grow-only counter
class GCounter {
  private counts: Map<string, number>;
 
  constructor(private readonly nodeId: string) {
    this.counts = new Map();
  }
 
  increment(amount: number = 1): void {
    const current = this.counts.get(this.nodeId) ?? 0;
    this.counts.set(this.nodeId, current + amount);
  }
 
  value(): number {
    let total = 0;
    for (const count of this.counts.values()) {
      total += count;
    }
    return total;
  }
 
  merge(other: GCounter): GCounter {
    const merged = new GCounter(this.nodeId);
 
    // Take the maximum count for each node
    const allNodes = new Set([
      ...this.counts.keys(),
      ...other.counts.keys(),
    ]);
 
    for (const node of allNodes) {
      const thisCount = this.counts.get(node) ?? 0;
      const otherCount = other.counts.get(node) ?? 0;
      merged.counts.set(node, Math.max(thisCount, otherCount));
    }
 
    return merged;
  }
 
  toJSON(): Record<string, number> {
    return Object.fromEntries(this.counts);
  }
 
  static fromJSON(nodeId: string, data: Record<string, number>): GCounter {
    const counter = new GCounter(nodeId);
    counter.counts = new Map(Object.entries(data));
    return counter;
  }
}
 
// PN-Counter: A counter that supports both increment and decrement
class PNCounter {
  private increments: GCounter;
  private decrements: GCounter;
 
  constructor(nodeId: string) {
    this.increments = new GCounter(nodeId);
    this.decrements = new GCounter(nodeId);
  }
 
  increment(amount: number = 1): void {
    this.increments.increment(amount);
  }
 
  decrement(amount: number = 1): void {
    this.decrements.increment(amount);
  }
 
  value(): number {
    return this.increments.value() - this.decrements.value();
  }
 
  merge(other: PNCounter): PNCounter {
    const merged = new PNCounter("");
    merged.increments = this.increments.merge(other.increments);
    merged.decrements = this.decrements.merge(other.decrements);
    return merged;
  }
}
 
// OR-Set: An observed-remove set that supports add and remove
class ORSet<T> {
  private elements: Map<string, Set<string>> = new Map();
  private tombstones: Map<string, Set<string>> = new Map();
 
  constructor(private readonly nodeId: string) {}
 
  add(element: T): void {
    const key = JSON.stringify(element);
    const tag = `${this.nodeId}-${Date.now()}-${Math.random()}`;
 
    if (!this.elements.has(key)) {
      this.elements.set(key, new Set());
    }
 
    this.elements.get(key)!.add(tag);
  }
 
  remove(element: T): void {
    const key = JSON.stringify(element);
    const tags = this.elements.get(key);
 
    if (tags) {
      if (!this.tombstones.has(key)) {
        this.tombstones.set(key, new Set());
      }
 
      for (const tag of tags) {
        this.tombstones.get(key)!.add(tag);
      }
 
      this.elements.delete(key);
    }
  }
 
  has(element: T): boolean {
    const key = JSON.stringify(element);
    const tags = this.elements.get(key);
    const deadTags = this.tombstones.get(key) ?? new Set();
 
    if (!tags) return false;
 
    for (const tag of tags) {
      if (!deadTags.has(tag)) return true;
    }
 
    return false;
  }
 
  values(): T[] {
    const result: T[] = [];
 
    for (const [key, tags] of this.elements) {
      const deadTags = this.tombstones.get(key) ?? new Set();
      const alive = Array.from(tags).some((tag) => !deadTags.has(tag));
 
      if (alive) {
        result.push(JSON.parse(key));
      }
    }
 
    return result;
  }
 
  merge(other: ORSet<T>): ORSet<T> {
    const merged = new ORSet<T>(this.nodeId);
 
    // Union all elements
    for (const [key, tags] of this.elements) {
      merged.elements.set(key, new Set(tags));
    }
    for (const [key, tags] of other.elements) {
      if (!merged.elements.has(key)) {
        merged.elements.set(key, new Set());
      }
      for (const tag of tags) {
        merged.elements.get(key)!.add(tag);
      }
    }
 
    // Union all tombstones
    for (const [key, tags] of this.tombstones) {
      merged.tombstones.set(key, new Set(tags));
    }
    for (const [key, tags] of other.tombstones) {
      if (!merged.tombstones.has(key)) {
        merged.tombstones.set(key, new Set());
      }
      for (const tag of tags) {
        merged.tombstones.get(key)!.add(tag);
      }
    }
 
    return merged;
  }
}

CRDTs (Conflict-free Replicated Data Types) are mathematically guaranteed to converge without coordination. They are the gold standard for conflict resolution in eventually consistent systems, though they come with constraints on what operations are supported.

Managing Consistency in Event-Driven Workflows

In event-driven systems, eventual consistency manifests as events that take time to propagate. A service publishes an event, but consumers may not process it for milliseconds, seconds, or even minutes. During this window, different parts of the system have different views of the world.

// Consistency-aware event processor
class ConsistencyManager {
  private versionVectors: Map<string, Map<string, number>> = new Map();
 
  updateVersion(
    entityId: string,
    source: string,
    version: number
  ): void {
    if (!this.versionVectors.has(entityId)) {
      this.versionVectors.set(entityId, new Map());
    }
    this.versionVectors.get(entityId)!.set(source, version);
  }
 
  isConsistent(
    entityId: string,
    requiredVersions: Map<string, number>
  ): boolean {
    const currentVersions = this.versionVectors.get(entityId);
 
    if (!currentVersions) return false;
 
    for (const [source, requiredVersion] of requiredVersions) {
      const currentVersion = currentVersions.get(source) ?? 0;
      if (currentVersion < requiredVersion) {
        return false;
      }
    }
 
    return true;
  }
}
 
// Saga that handles eventual consistency across services
class OrderConsistencySaga {
  private expectedStates = new Map<string, Set<string>>();
 
  constructor(
    private eventBus: EventBus,
    private consistencyManager: ConsistencyManager
  ) {
    this.eventBus.subscribe("OrderPlaced", this.onOrderPlaced.bind(this));
    this.eventBus.subscribe("InventoryReserved", this.onInventoryReserved.bind(this));
    this.eventBus.subscribe("PaymentAuthorized", this.onPaymentAuthorized.bind(this));
  }
 
  private async onOrderPlaced(event: DomainEvent): Promise<void> {
    const orderId = event.aggregateId;
 
    // Track what we are waiting for
    this.expectedStates.set(
      orderId,
      new Set(["inventory_reserved", "payment_authorized"])
    );
 
    // Set a timeout for consistency convergence
    setTimeout(() => {
      this.checkConsistencyTimeout(orderId);
    }, 30000); // 30-second deadline
  }
 
  private async onInventoryReserved(event: DomainEvent): Promise<void> {
    const orderId = event.payload.orderId as string;
    this.markCompleted(orderId, "inventory_reserved");
  }
 
  private async onPaymentAuthorized(event: DomainEvent): Promise<void> {
    const orderId = event.payload.orderId as string;
    this.markCompleted(orderId, "payment_authorized");
  }
 
  private markCompleted(orderId: string, step: string): void {
    const remaining = this.expectedStates.get(orderId);
    if (!remaining) return;
 
    remaining.delete(step);
 
    if (remaining.size === 0) {
      // All steps completed, the order is fully consistent
      this.expectedStates.delete(orderId);
      this.eventBus.publish({
        eventId: crypto.randomUUID(),
        eventType: "OrderFullyProcessed",
        timestamp: new Date(),
        aggregateId: orderId,
        version: 1,
        payload: { orderId },
        metadata: {
          correlationId: orderId,
          causationId: orderId,
          source: "consistency-saga",
        },
      });
    }
  }
 
  private checkConsistencyTimeout(orderId: string): void {
    const remaining = this.expectedStates.get(orderId);
 
    if (remaining && remaining.size > 0) {
      console.error(
        `Order ${orderId} did not reach consistency within deadline. ` +
        `Missing steps: ${Array.from(remaining).join(", ")}`
      );
 
      this.eventBus.publish({
        eventId: crypto.randomUUID(),
        eventType: "OrderConsistencyTimeout",
        timestamp: new Date(),
        aggregateId: orderId,
        version: 1,
        payload: {
          orderId,
          missingSteps: Array.from(remaining),
        },
        metadata: {
          correlationId: orderId,
          causationId: orderId,
          source: "consistency-saga",
        },
      });
    }
  }
}

Communicating Consistency to Users

Perhaps the most practical challenge of eventual consistency is the user experience. Users expect immediate feedback, and telling them "your data will be available eventually" is not acceptable. Several patterns help bridge this gap.

// Optimistic UI pattern: Return predicted state immediately
interface OptimisticResponse<T> {
  data: T;
  confirmed: boolean;
  confirmationToken: string;
  estimatedConfirmationMs: number;
}
 
class OptimisticOrderService {
  constructor(
    private commandBus: CommandBus,
    private readModel: OrderReadModel,
    private confirmationTracker: ConfirmationTracker
  ) {}
 
  async placeOrder(
    command: CreateOrderCommand
  ): Promise<OptimisticResponse<OrderSummary>> {
    const result = await this.commandBus.dispatch(command);
 
    if (!result.success) {
      throw new Error(result.error);
    }
 
    // Create a confirmation token for the client to poll
    const token = await this.confirmationTracker.create(
      result.aggregateId!,
      result.events ?? []
    );
 
    // Return an optimistic response built from command data
    return {
      data: {
        orderId: result.aggregateId!,
        status: "pending",
        items: command.payload.items,
        totalAmount: command.payload.items.reduce(
          (sum, item) => sum + item.unitPrice * item.quantity,
          0
        ),
        createdAt: new Date(),
      },
      confirmed: false,
      confirmationToken: token,
      estimatedConfirmationMs: 2000,
    };
  }
 
  async checkConfirmation(
    token: string
  ): Promise<{ confirmed: boolean; data?: OrderSummary }> {
    const status = await this.confirmationTracker.check(token);
 
    if (status.confirmed) {
      const order = await this.readModel.getOrder(status.aggregateId);
      return { confirmed: true, data: order ?? undefined };
    }
 
    return { confirmed: false };
  }
}
 
interface ConfirmationTracker {
  create(aggregateId: string, events: string[]): Promise<string>;
  check(token: string): Promise<{
    confirmed: boolean;
    aggregateId: string;
  }>;
}
 
interface OrderSummary {
  orderId: string;
  status: string;
  items: any[];
  totalAmount: number;
  createdAt: Date;
}
 
interface OrderReadModel {
  getOrder(orderId: string): Promise<OrderSummary | null>;
}

Practical Tips

Be explicit about your consistency guarantees. Document which operations are strongly consistent and which are eventually consistent. Different parts of your system can have different consistency requirements.

Measure convergence time. Track the time between when an event is published and when all consumers have processed it. This metric tells you whether "eventual" is milliseconds or minutes, and helps you set appropriate expectations.

Use compensating actions for eventual failures. If a step in a distributed workflow fails after other steps have succeeded, use compensating events to undo the completed steps rather than trying to achieve distributed transactions.

Design for idempotency everywhere. In an eventually consistent system, the same event may be processed multiple times. Every consumer must handle this gracefully.

Consider the business context. Some operations genuinely require strong consistency --- financial transactions, inventory counts near zero, and anything where human safety is involved. Do not force eventual consistency where it does not belong.

Conclusion

Eventual consistency is not a compromise; it is a deliberate architectural choice that enables distributed systems to be more available, more responsive, and more resilient to failure. The key is understanding the trade-offs and applying the right patterns to manage them.

From CRDTs that guarantee convergence without coordination, to sagas that track consistency across services, to optimistic UI patterns that maintain a smooth user experience, the toolkit for managing eventual consistency is rich and well-established. By combining these patterns with TypeScript's type safety and the real-time processing capabilities of Starburst, you can build systems that embrace eventual consistency while delivering a reliable and predictable experience.

Related Articles

business

Monitoring Event-Driven Systems at Scale

A practical guide to building comprehensive monitoring and observability for event-driven systems, covering metrics, distributed tracing, alerting strategies, and operational dashboards for maintaining healthy event processing pipelines.

12 min read
business

Migrating to Event-Driven Architecture

A practical guide for planning and executing a migration from traditional request-response systems to event-driven architecture, covering assessment frameworks, migration strategies, risk management, and organizational change.

12 min read
business

Real-Time Data Processing: Business Impact and ROI

An exploration of the business value of real-time data processing, covering measurable ROI, competitive advantages, and practical frameworks for justifying investment in event-driven infrastructure.

11 min read