Solving the Dual-Write Problem with Spring Boot Starter Outbox

# The Problem

Publishing messages to a message broker inside a database transaction is unreliable. If the transaction rolls back after publishing, you have a phantom message. If publishing fails after the transaction commits, you have a lost message.

Consider this flow in a typical microservice:

1. Your database — source of truth for business state (orders, users, shipments)
2. Your message broker — communication channel to other services (Kafka topics, RabbitMQ exchanges)

When an order is placed, you naturally want to save it to the database AND publish an event to notify other services. But this naive approach has a critical race condition:

• Broker publish succeeds, then DB transaction rolls back → consumers see an order that never existed (phantom message)
• DB transaction commits, then broker publish fails → consumers never hear about the order (lost message)
• Process crashes between the two writes → either phantom or lost message, depending on when

Traditional fixes like two-phase commit, distributed transactions, or ad-hoc retry logic are costly, slow, and still vulnerable to failures. Event sourcing is powerful but requires an architectural commitment that many teams can’t justify.

The Transactional Outbox pattern elegantly sidesteps the problem.

# Why the Outbox Pattern

The pattern works in three phases:

Phase 1 - Atomic Transaction (Synchronous): Save both the business entity AND the outbox event row in the same database transaction. Both writes succeed together or fail together. No partial states.

Phase 2 - Relay and Publisher (Asynchronous): A separate scheduler polls the outbox table for PENDING events and publishes them to the message broker. If the broker is down, events remain PENDING in the database. No loss.

Phase 3 - Message Broker (Asynchronous): Events are published to topics/exchanges. Downstream consumers subscribe and process them idempotently using the event ID (UUID) as a deduplication key.

Key insight: If the database commit succeeds, the outbox row definitely exists. If it fails, neither the business data nor the event row persist. No inconsistency.

This creates a strong consistency boundary between the database and the message broker, allowing them to maintain ACID guarantees internally while being eventually consistent externally.

# Architectural Benefits

• Exactly-once semantics (achieved with consumer deduplication): Combined with consumer idempotency via event ID, you get at-least-once from the pattern plus exactly-once from the consumer.
• No broker coupling: The service is resilient to broker downtime. Events queue in the database until the broker recovers.
• Failure isolation: Broker issues don’t cascade to the business transaction.
• Observability: Failed events are recorded in the database with retry counts and error messages.
• Partition key preservation: The aggregateId becomes the Kafka partition key, preserving per-entity ordering.
• Separation of concerns: Database writes are decoupled from message broker concerns via the relay mechanism.
• Operational simplicity: No distributed transactions or two-phase commit needed.

# The Outbox Table

All events live in a single table, outbox_events:

CREATE TABLE outbox_events (
    id             UUID          PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(100)  NOT NULL,
    aggregate_id   VARCHAR(255)  NOT NULL,
    event_type     VARCHAR(100)  NOT NULL,
    payload        TEXT          NOT NULL,
    status         VARCHAR(20)   NOT NULL DEFAULT 'PENDING',
    retry_count    INTEGER       NOT NULL DEFAULT 0,
    error_message  TEXT,
    created_at     TIMESTAMPTZ   NOT NULL DEFAULT now(),
    processed_at   TIMESTAMPTZ
);

CREATE INDEX idx_outbox_status_created ON outbox_events(status, created_at);

Column meanings:

• id - Deduplication key for consumers (required for idempotent processing)
• aggregate_type / aggregate_id - Business routing (which entity owns this event)
• event_type - Event classification (maps to Kafka topic or RabbitMQ routing key)
• payload - JSON serialization of the domain event
• status - PENDING (awaiting relay), PROCESSED (successfully delivered), FAILED (exhausted retries)
• retry_count - Number of attempts made
• error_message - Last failure reason for debugging
• created_at - When the event was persisted (DB commit time)
• processed_at - When the relay successfully delivered it

# Data Flow Diagram

The flow moves through three logical layers:

graph TD
    subgraph Service["Service Layer (Business Logic)"]
        OS["OrderService.createOrder()"]
        TXN["@Transactional Boundary"]
        DBW["Save Order to DB"]
        OUTBOX["Publish Event to Outbox"]
    end

    subgraph Database["PostgreSQL Database"]
        ORDERS["orders table"]
        OUTBOXTBL["outbox_events table<br/>(status=PENDING)"]
    end

    subgraph Relay["OutboxEventRelay (@Scheduled)"]
        POLL["Poll: SELECT WHERE status=PENDING"]
        SEND["Broker Adapter.send()"]
        UPDATE["UPDATE status=PROCESSED"]
    end

    subgraph Broker["Kafka or RabbitMQ"]
        TOPIC["Topic/Exchange"]
    end

    subgraph Consumer["Downstream Services"]
        CS["Consumer Service"]
        DEDUP["Deduplicate by event.id"]
        PROCESS["Process Event"]
    end

    OS --> TXN
    TXN --> DBW
    DBW --> ORDERS
    TXN --> OUTBOX
    OUTBOX --> OUTBOXTBL
    OUTBOXTBL -->|COMMIT| RELAY
    RELAY --> POLL
    POLL -->|Every 5s| SEND
    SEND --> TOPIC
    TOPIC --> CS
    CS --> DEDUP
    DEDUP --> PROCESS

# Failure Modes and Recovery

Monitor the outbox table for events stuck in PENDING state (age > 5× relay interval) or accumulating in FAILED state. These are early warning signs of broker issues or payload corruption.

# Implementation Guide

Prerequisites:

• Spring Boot 3.x or 4.x
• Java 17+
• Spring Data JPA with PostgreSQL (or compatible database with UUID support)
• Apache Kafka or RabbitMQ
• Gradle or Maven

Add the starter to build.gradle.kts:

dependencies {
    implementation("io.github.saumilp.starters:spring-boot-starter-outbox:1.0.0")
    implementation("org.springframework.kafka:spring-kafka:3.3.0")
    // or:
    // implementation("org.springframework.boot:spring-boot-starter-amqp:3.2.0")
}

Step 1: Create the Outbox Table

Run the SQL DDL above against your PostgreSQL database. The table should be created during initial schema setup or via Flyway/Liquibase migrations.

Step 2: Configure the Starter

In application.yml:

spring:
  outbox:
    enabled: true
    relay-interval-ms: 5000
    max-retries: 3
    backoff-multiplier: 2.0
    kafka:
      bootstrap-servers: localhost:9092
      topic-prefix: outbox.

Step 3: Implement Your Service

@Service
public class OrderService {
    private final OrderRepository orderRepository;
    private final OutboxPublisher outboxPublisher;

    @Transactional
    public Order createOrder(CreateOrderRequest req) {
        Order order = orderRepository.save(new Order(req));
        
        outboxPublisher.publish(
            new OrderCreatedEvent(
                order.getId(),
                order.getCustomerId(),
                order.getTotal()
            )
        );
        
        return order;
    }
}

The @Transactional annotation ensures both the order AND the outbox event are persisted atomically.

Step 4: Consuming Events

Consumers must deduplicate by event ID to handle at-least-once delivery:

@Service
public class OrderEventConsumer {
    private final OrderEventRepository eventRepository;

    @KafkaListener(topics = "outbox.ORDER_CREATED")
    public void handleOrderCreated(OrderCreatedEvent event) {
        // Check if already processed
        if (eventRepository.existsByEventId(event.getId())) {
            return;
        }
        
        // Process the event
        processOrder(event);
        
        // Record that we processed it
        eventRepository.save(new ProcessedEvent(event.getId()));
    }
}

Consumer Idempotency Patterns:

1. In-memory deduplication (low-traffic): Store event IDs in a HashSet. Simple but lost on restart.

2. Database deduplication (recommended): Store processed event IDs in a database table with unique constraint. Survives restarts.

3. Idempotent business operations (best): Make the operation itself idempotent. Use event ID as a key. If the operation runs twice, it’s safe.

4. Database constraints (clever): Use UNIQUE constraint on (event_id, business_key). The second insert fails safely.

# Infrastructure and Deployment

Docker Compose for local development:

version: '3.8'
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: outbox_demo
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

volumes:
  postgres_data:

Testing the flow:

1. Start services: docker-compose up
2. Create an order: curl -X POST http://localhost:8080/orders -H "Content-Type: application/json" -d '{"customerId":"cust-123", "amount":99.99}'
3. Check outbox table: SELECT * FROM outbox_events;
4. Verify Kafka topic has the event: kafka-console-consumer.sh --topic outbox.ORDER_CREATED --bootstrap-servers localhost:9092
5. Verify consumer processed it: SELECT * FROM processed_events;

CI/CD Pipeline (GitHub Actions):

name: Test Outbox Starter
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v3
        with:
          java-version: 21
          distribution: temurin
      - run: ./gradlew test --info

# Key Takeaways

For Architects: The Outbox pattern is NOT a substitute for event sourcing. Use it for reliable event publishing in transaction-based services. For temporal queries, event replay, or audit trails, event sourcing is the better choice.

For Developers: Remember: the pattern provides at-least-once delivery, so consumers MUST be idempotent. Use the event ID as your deduplication key. Test both success and failure paths (relay down, broker down, transaction rollback).

For DevOps: Monitor the outbox table cardinality and relay latency. Set up alerts for FAILED events accumulating. The relay is stateless, so you can run multiple instances for redundancy.

Best Practices:

• Always use event IDs for consumer deduplication
• Make business operations idempotent, not just event deduplication
• Set reasonable max-retries (usually 3-5)
• Use exponential backoff for retries
• Log failed events with full error messages
• Partition by aggregateId for ordering guarantees
• Never delete from outbox table; archive to cold storage after 30 days

Kafka vs RabbitMQ Trade-offs:

For the Outbox pattern, Kafka’s partition-based ordering is ideal. For RabbitMQ, use topic exchanges with routing keys bound to dedicated queues.

# References and Further Reading

• Transactional Outbox Pattern: https://microservices.io/patterns/data/transactional-outbox.html
• Spring Boot Starter Outbox: https://github.com/saumilpatel/spring-boot-starter-outbox
• Event-Driven Architecture: https://www.oreilly.com/library/view/building-event-driven-microservices/
• Kafka at-least-once semantics: https://kafka.apache.org/documentation/#consumerconfigs_enable.auto.commit
• Spring Data JPA: https://spring.io/projects/spring-data-jpa
• PostgreSQL UUID type: https://www.postgresql.org/docs/current/datatype-uuid.html

# Troubleshooting

Q: Events are stuck in PENDING status A: Check if the relay is running (logs should show “Polling outbox…”). Verify broker connectivity. Check error_message column for details.

Q: “Transaction already rolled back” error when calling publish() A: OutboxPublisher must be called within a @Transactional method. The transaction must be active and not already marked for rollback.

Q: Duplicate messages in the broker A: The pattern guarantees at-least-once. Consumers MUST deduplicate by event ID. This is not a bug; it’s the contract.

Q: Relay is taking too long (> relay-interval-ms) A: Increase relay-interval-ms or add database indexes. If outbox_events table is huge, partition it by created_at or status.

Q: How do I handle event schema changes? A: Version your event payloads (add a “version” field). Consumers should support multiple versions. Use JSON Schema or Protocol Buffers for validation.

Q: Can I use this with RabbitMQ? A: Yes. Configure rabbitmq section instead of kafka in application.yml. The pattern is broker-agnostic.