K
Klivvr Blog

Designing Idempotent APIs and Workflows

A comprehensive guide to designing idempotent operations in distributed workflows, covering idempotency keys, deduplication strategies, and practical patterns for ensuring safe retries.

technical11 min readBy Klivvr Engineering
Share:

An operation is idempotent if performing it multiple times produces the same result as performing it once. This property is the bedrock of reliable distributed workflows. In a system where any network call can fail after the request is sent but before the response is received, you can never be certain whether an operation actually completed. The only safe option is to retry it, and retrying is only safe if the operation is idempotent.

Consider a payment processing step in a workflow. The workflow sends a charge request to the payment service. The payment service processes the charge successfully and sends back a response, but the response is lost due to a network partition. From the workflow's perspective, the step failed. It retries, sending the same charge request again. Without idempotency, the customer gets charged twice. With idempotency, the payment service recognizes the duplicate request and returns the result of the original charge without processing it again.

This article explores the patterns and techniques that Alfred uses to make workflows and their constituent steps idempotent.

Idempotency Keys

The most widely used approach to idempotency is the idempotency key: a unique identifier attached to each request that allows the receiver to detect duplicates. If a request arrives with an idempotency key that has already been processed, the receiver returns the stored result instead of processing the request again.

import { v5 as uuidv5 } from 'uuid';
 
// Deterministic idempotency key generation
// Same inputs always produce the same key
function generateIdempotencyKey(
  workflowId: string,
  stepName: string,
  attempt: number
): string {
  const namespace = '6ba7b810-9dad-11d1-80b4-00c04fd430c8';
  const input = `${workflowId}:${stepName}:${attempt}`;
  return uuidv5(input, namespace);
}
 
// Using idempotency keys in a workflow step
const chargePayment = async (ctx: OrderContext): Promise<StepResult<OrderContext>> => {
  const idempotencyKey = generateIdempotencyKey(
    ctx.workflowId,
    'charge-payment',
    ctx.currentAttempt
  );
 
  const result = await paymentService.charge({
    orderId: ctx.orderId,
    amount: ctx.totalAmount,
    currency: 'USD',
    idempotencyKey,
  });
 
  return StepResult.success({ ...ctx, paymentId: result.id });
};

The idempotency key is derived deterministically from the workflow ID, step name, and attempt number. This ensures that if the same step is retried with the same attempt number, it produces the same idempotency key, allowing the downstream service to deduplicate the request.

A critical detail: the attempt number is included in the key. This is intentional. If a step fails and its compensating action runs, the next execution of the same step is semantically a new operation and should not be deduplicated against the previous one. Including the attempt number ensures that only true retries of the same operation are deduplicated.

Server-Side Idempotency Implementation

When you build services that Alfred workflows call, implementing server-side idempotency correctly requires attention to several details.

import { Request, Response, NextFunction } from 'express';
import { Redis } from 'ioredis';
 
interface IdempotencyRecord {
  status: 'processing' | 'completed' | 'failed';
  statusCode?: number;
  body?: unknown;
  createdAt: number;
  expiresAt: number;
}
 
class IdempotencyMiddleware {
  private redis: Redis;
  private ttl: number;
 
  constructor(redis: Redis, ttlSeconds: number = 86400) {
    this.redis = redis;
    this.ttl = ttlSeconds;
  }
 
  middleware() {
    return async (req: Request, res: Response, next: NextFunction) => {
      const idempotencyKey = req.headers['idempotency-key'] as string;
      if (!idempotencyKey) {
        return next(); // No idempotency key, process normally
      }
 
      const recordKey = `idempotency:${idempotencyKey}`;
 
      // Try to acquire the processing lock
      const acquired = await this.redis.set(
        recordKey,
        JSON.stringify({
          status: 'processing',
          createdAt: Date.now(),
          expiresAt: Date.now() + this.ttl * 1000,
        } as IdempotencyRecord),
        'EX',
        this.ttl,
        'NX' // Only set if key doesn't exist
      );
 
      if (acquired) {
        // First time seeing this key, process the request
        // Intercept the response to store the result
        const originalJson = res.json.bind(res);
        res.json = (body: unknown) => {
          const record: IdempotencyRecord = {
            status: 'completed',
            statusCode: res.statusCode,
            body,
            createdAt: Date.now(),
            expiresAt: Date.now() + this.ttl * 1000,
          };
          this.redis.set(recordKey, JSON.stringify(record), 'EX', this.ttl);
          return originalJson(body);
        };
        return next();
      }
 
      // Key already exists, check the status
      const existingRecord = await this.redis.get(recordKey);
      if (!existingRecord) {
        // Race condition: key expired between our SET and GET
        return next();
      }
 
      const record: IdempotencyRecord = JSON.parse(existingRecord);
 
      if (record.status === 'processing') {
        // Another request with the same key is still being processed
        return res.status(409).json({
          error: 'CONFLICT',
          message: 'A request with this idempotency key is currently being processed',
        });
      }
 
      if (record.status === 'completed') {
        // Return the stored result
        return res.status(record.statusCode ?? 200).json(record.body);
      }
 
      // Previous attempt failed, allow retry
      return next();
    };
  }
}

Several things are worth noting in this implementation. The NX flag on the Redis SET command ensures atomicity. If two duplicate requests arrive simultaneously, only one of them acquires the lock. The other receives a 409 Conflict, telling the caller to wait and retry.

The TTL on idempotency records prevents the storage from growing unbounded. Choose a TTL that exceeds the maximum time a client might retry a request. For Alfred workflows, 24 hours is a reasonable default since workflows that take longer than that typically have explicit wait states rather than retries.

Failed requests are stored with a failed status, allowing the client to retry with the same idempotency key. Only successful responses are treated as final.

Making Workflow Steps Naturally Idempotent

While idempotency keys are a powerful tool, the best approach is to design operations that are naturally idempotent, operations where the nature of the operation itself ensures that repetition is harmless.

// BAD: Not naturally idempotent
// If called twice, the balance is decremented twice
async function decrementBalance(accountId: string, amount: number): Promise<void> {
  const account = await db.query('SELECT balance FROM accounts WHERE id = $1', [accountId]);
  const newBalance = account.rows[0].balance - amount;
  await db.query('UPDATE accounts SET balance = $1 WHERE id = $2', [newBalance, accountId]);
}
 
// GOOD: Naturally idempotent using a unique transaction reference
// If called twice with the same transactionId, the second call is a no-op
async function decrementBalance(
  accountId: string,
  amount: number,
  transactionId: string
): Promise<void> {
  await db.query(
    `INSERT INTO transactions (id, account_id, amount, type)
     VALUES ($1, $2, $3, 'debit')
     ON CONFLICT (id) DO NOTHING`,
    [transactionId, accountId, amount]
  );
 
  await db.query(
    `UPDATE accounts SET balance = (
       SELECT COALESCE(SUM(CASE WHEN type = 'credit' THEN amount ELSE -amount END), 0)
       FROM transactions WHERE account_id = $1
     ) WHERE id = $1`,
    [accountId]
  );
}

The second implementation is naturally idempotent because the balance is always derived from the complete set of transactions. Inserting the same transaction twice has no effect thanks to the ON CONFLICT DO NOTHING clause. The balance update always computes the correct value regardless of how many times it runs.

Here are common patterns for making operations naturally idempotent:

// Pattern 1: Upsert instead of insert
// Idempotent: inserting the same record twice results in one record
async function createUserProfile(userId: string, profile: UserProfile): Promise<void> {
  await db.query(
    `INSERT INTO user_profiles (user_id, name, email, created_at)
     VALUES ($1, $2, $3, NOW())
     ON CONFLICT (user_id) DO UPDATE
     SET name = EXCLUDED.name, email = EXCLUDED.email`,
    [userId, profile.name, profile.email]
  );
}
 
// Pattern 2: Conditional state transitions
// Idempotent: transitioning from 'pending' to 'confirmed' twice is safe
async function confirmOrder(orderId: string): Promise<boolean> {
  const result = await db.query(
    `UPDATE orders SET status = 'confirmed', confirmed_at = NOW()
     WHERE id = $1 AND status = 'pending'
     RETURNING id`,
    [orderId]
  );
  return result.rowCount > 0;
}
 
// Pattern 3: Set-based operations instead of delta operations
// Idempotent: setting the quantity to 5 is the same whether done once or twice
async function setCartItemQuantity(
  cartId: string,
  itemId: string,
  quantity: number
): Promise<void> {
  await db.query(
    `INSERT INTO cart_items (cart_id, item_id, quantity)
     VALUES ($1, $2, $3)
     ON CONFLICT (cart_id, item_id) DO UPDATE SET quantity = EXCLUDED.quantity`,
    [cartId, itemId, quantity]
  );
}

Deduplication at the Workflow Level

Beyond individual step idempotency, Alfred provides workflow-level deduplication to prevent the same workflow from being started multiple times.

import { WorkflowEngine, DeduplicationStrategy } from '@alfred/core';
 
const engine = new WorkflowEngine({
  deduplication: {
    strategy: DeduplicationStrategy.BY_CORRELATION_ID,
    ttl: '24h',
    onDuplicate: 'return-existing', // or 'reject'
  },
});
 
// Starting a workflow with a correlation ID
const result = await engine.start('order-fulfillment', {
  correlationId: `order-${orderId}`, // Deterministic: same order always gets same ID
  context: {
    orderId,
    customerId,
    items,
  },
});
 
if (result.deduplicated) {
  // This workflow was already started, 'result' contains the existing instance
  console.log(`Workflow already exists: ${result.workflowInstanceId}`);
} else {
  console.log(`New workflow started: ${result.workflowInstanceId}`);
}

The correlation ID serves as a natural deduplication key. When a message is delivered twice by a message broker or when a user double-clicks a submit button, the second workflow start request is detected as a duplicate and handled according to the configured strategy: either returning the existing workflow instance or rejecting the request outright.

Alfred stores correlation IDs in a fast lookup store, typically Redis, with a configurable TTL. The TTL should be long enough to cover the maximum time between potential duplicate requests but short enough to allow legitimate re-submissions of the same logical operation after the original has completed.

// Advanced deduplication: composite keys for more precise matching
const result = await engine.start('invoice-generation', {
  correlationId: DeduplicationKey.composite(
    customerId,
    billingPeriod,
    invoiceType
  ),
  context: {
    customerId,
    billingPeriod,
    invoiceType,
  },
});

Composite deduplication keys allow you to define exactly what constitutes a "duplicate" for each workflow type. For invoice generation, two requests for the same customer, billing period, and invoice type are duplicates, even if they arrive minutes apart.

Testing Idempotency

Idempotency is one of those properties that is easy to claim and hard to verify. Alfred provides testing utilities specifically for idempotency validation.

import { IdempotencyTester } from '@alfred/testing';
 
describe('Payment Processing Step', () => {
  it('should be idempotent: charging twice produces the same result', async () => {
    const tester = new IdempotencyTester(chargePaymentStep);
 
    const context: OrderContext = {
      workflowId: 'test-workflow-1',
      orderId: 'order-123',
      totalAmount: 99.99,
      currentAttempt: 1,
    };
 
    // Execute the step twice with the same context
    const result1 = await tester.execute(context);
    const result2 = await tester.execute(context);
 
    // Both executions should produce the same result
    expect(result1.context.paymentId).toBe(result2.context.paymentId);
 
    // The payment should only be charged once
    const charges = await paymentService.getCharges('order-123');
    expect(charges).toHaveLength(1);
    expect(charges[0].amount).toBe(99.99);
  });
 
  it('should handle concurrent duplicate requests', async () => {
    const tester = new IdempotencyTester(chargePaymentStep);
 
    const context: OrderContext = {
      workflowId: 'test-workflow-2',
      orderId: 'order-456',
      totalAmount: 49.99,
      currentAttempt: 1,
    };
 
    // Execute the step concurrently
    const [result1, result2] = await Promise.all([
      tester.execute(context),
      tester.execute(context),
    ]);
 
    // One should succeed, the other should either succeed with the same result
    // or receive a conflict that resolves to the same result
    expect(result1.context.paymentId).toBe(result2.context.paymentId);
  });
 
  it('should allow re-execution after compensation', async () => {
    const tester = new IdempotencyTester(chargePaymentStep);
 
    const context1: OrderContext = {
      workflowId: 'test-workflow-3',
      orderId: 'order-789',
      totalAmount: 29.99,
      currentAttempt: 1,
    };
 
    // First execution
    const result1 = await tester.execute(context1);
    expect(result1.context.paymentId).toBeDefined();
 
    // Compensate (refund)
    await chargePaymentStep.compensate(result1.context);
 
    // Second execution with a new attempt number
    const context2 = { ...context1, currentAttempt: 2 };
    const result2 = await tester.execute(context2);
 
    // Should be a new charge, not deduplicated against the first
    expect(result2.context.paymentId).not.toBe(result1.context.paymentId);
  });
});

These tests validate three critical scenarios: basic idempotency under sequential retries, idempotency under concurrent access, and correct behavior after compensation where a new attempt should not be deduplicated against a compensated previous attempt.

Practical Tips

Keep these guidelines in mind when designing for idempotency.

Generate idempotency keys deterministically from the request content, not randomly. If you use random UUIDs, a retry will have a different key and bypass deduplication entirely. The key must be the same for every retry of the same logical operation.

Store idempotency records in a system that survives restarts. In-memory deduplication is useless if the process crashes between the original request and the retry. Use Redis, a database, or another durable store.

Set appropriate TTLs on idempotency records. Too short, and legitimate retries will not be deduplicated. Too long, and your storage grows unbounded. Align the TTL with your retry policy's maximum duration.

Handle the "processing" state carefully. If a request is marked as "processing" and the server crashes before completing it, the idempotency record will block retries. Implement a timeout mechanism that clears stale "processing" records.

Prefer naturally idempotent designs over idempotency keys when possible. A set-based operation that overwrites state is inherently simpler than a delta operation that requires deduplication logic.

Conclusion

Idempotency is not an optional nice-to-have in distributed workflows. It is a fundamental requirement for safe retries, reliable compensations, and correct behavior in the presence of network failures. Alfred builds idempotency into its workflow engine at every level: deterministic idempotency key generation for outgoing requests, workflow-level deduplication to prevent duplicate workflow instances, and testing utilities to verify idempotent behavior.

By combining idempotency keys for external service calls, naturally idempotent designs for your own services, and workflow-level deduplication, you create a system where retries are always safe and failures are always recoverable. That confidence is what allows you to build complex multi-step business processes without living in fear of the next network hiccup.

Related Articles

technical

Testing Complex Workflows: Strategies and Tools

A comprehensive guide to testing multi-step distributed workflows, covering unit testing individual steps, integration testing complete flows, chaos testing, and time-travel debugging.

13 min read
business

Error Recovery Patterns in Workflow Engines

Explore the error recovery patterns used in production workflow engines, from simple retries to complex human-in-the-loop escalation strategies, with a focus on business continuity.

13 min read