Solving the Dual-Write Problem with Transactional Outbox & Apache Kafka

The Dual-Write Pitfall

A common pattern in modern microservices is handling an incoming HTTP request by writing state to a relational database and publishing an event to Apache Kafka:

@Transactional
public void processOrder(OrderRequest request) {
    Order order = orderRepository.save(new Order(request));
    kafkaTemplate.send("order-events", new OrderCreatedEvent(order));
}

While clean on the surface, this code has a critical flaw known as the Dual-Write Problem:

1. Broker Failure After DB Commit: If Kafka is momentarily down or the network times out during kafkaTemplate.send(), the database transaction commits, but downstream consumers never receive the event.
2. Crash Before Commit: If the Kafka message sends successfully but the database commit subsequently fails due to constraint violations, downstream services process an event for an entity that does not exist in the database.

The Solution: Transactional Outbox Pattern

The Transactional Outbox pattern guarantees that database state updates and event publishing are bound by the same local ACID transaction.

flowchart LR
    Service[Spring Boot Service] -->|1. Single ACID Transaction| DB[(PostgreSQL)]
    DB -.->|Outbox Table| Relay[Outbox Relay / Debezium]
    Relay -->|2. Publish Events| Kafka((Apache Kafka))
    Kafka -->|3. Consume| Consumer[Downstream Services]

1. Database Schema

We introduce an outbox_events table in PostgreSQL:

CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(64) NOT NULL,
    aggregate_id VARCHAR(64) NOT NULL,
    event_type VARCHAR(64) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    published BOOLEAN DEFAULT FALSE
);

CREATE INDEX idx_outbox_unpublished ON outbox_events (created_at) WHERE published = FALSE;

2. Atomic Writing in Spring Boot

When business logic executes, we append the event record in the exact same transaction:

@Transactional
public Order createOrder(OrderRequest request) {
    Order order = orderRepository.save(new Order(request));
    
    OutboxEvent event = OutboxEvent.builder()
        .aggregateType("ORDER")
        .aggregateId(order.getId().toString())
        .eventType("ORDER_CREATED")
        .payload(objectMapper.writeValueAsString(order))
        .build();
        
    outboxRepository.save(event);
    return order;
}

If the transaction rolls back, both the order and the outbox event are rolled back cleanly. If it succeeds, the event is guaranteed to persist.

Conclusion

By isolating event persistence to the local transaction and decoupling broker transmission to an asynchronous poller or Change Data Capture (CDC) pipeline, we achieve true At-Least-Once delivery semantics with zero data loss.