Implement Outbox Pattern in C# Microservices

This article provides a practical guide to implementing the transactional outbox pattern in C# microservices to ensure reliable message publishing. You will learn how this pattern resolves the distributed "dual-write" problem, how to capture domain events within an Entity Framework Core database transaction, and how to reliably process and dispatch these messages to a message broker using a background worker.

The Dual-Write Problem

In a microservice architecture, a service often needs to update its local database and notify other services via a message broker (such as RabbitMQ or Azure Service Bus). If the database write succeeds but the network call to the broker fails, the system enters an inconsistent state. Conversely, if the message publishes first and the database transaction rolls back, external services act on phantom data.

The Outbox Pattern resolves this by storing outgoing messages in a dedicated database table within the same ACID transaction as the business operation. A separate background process then reads unprocessed records from the outbox table and dispatches them to the message broker, guaranteeing at-least-once delivery.

Step 1: Define the Outbox Message Entity

The outbox table needs enough metadata to track the message payload, event type, creation time, and processing status.

public sealed class OutboxMessage
{
    public Guid Id { get; set; }
    public string Type { get; set; } = string.Empty;
    public string Content { get; set; } = string.Empty;
    public DateTime OccurredOnUtc { get; set; }
    public DateTime? ProcessedOnUtc { get; set; }
    public string? Error { get; set; }
}

Configure the entity in your EF Core DbContext configuration, adding an index on ProcessedOnUtc to optimize polling queries:

public void Configure(EntityTypeBuilder<OutboxMessage> builder)
{
    builder.ToTable("OutboxMessages");
    builder.HasKey(m => m.Id);
    builder.HasIndex(m => m.ProcessedOnUtc);
}

Step 2: Persist Events Atomically

Domain events should be serialized into JSON and saved to the outbox table inside the same transaction as the domain aggregate. You can achieve this automatically by intercepting SaveChangesAsync in EF Core:

public sealed class OutboxInterceptor : SaveChangesInterceptor
{
    public override ValueTask<InterceptionResult<int>> SavingChangesAsync(
        DbContextEventData eventData,
        InterceptionResult<int> result,
        CancellationToken cancellationToken = default)
    {
        var dbContext = eventData.Context;
        if (dbContext is null)
        {
            return base.SavingChangesAsync(eventData, result, cancellationToken);
        }

        var outboxMessages = dbContext.ChangeTracker
            .Entries<IHasDomainEvents>()
            .Select(entry => entry.Entity)
            .SelectMany(entity =>
            {
                var events = entity.GetDomainEvents();
                entity.ClearDomainEvents();
                return events;
            })
            .Select(domainEvent => new OutboxMessage
            {
                Id = Guid.NewGuid(),
                OccurredOnUtc = DateTime.UtcNow,
                Type = domainEvent.GetType().Name,
                Content = JsonSerializer.Serialize(domainEvent, domainEvent.GetType())
            })
            .ToList();

        dbContext.Set<OutboxMessage>().AddRange(outboxMessages);

        return base.SavingChangesAsync(eventData, result, cancellationToken);
    }
}

Register this interceptor in your dependency injection container when configuring the DbContext.

Step 3: Publish Messages Using a Background Worker

A background processor regularly queries the outbox table for unprocessed messages, publishes them to the broker, and updates the timestamp upon success.

public sealed class OutboxPublisherBackgroundService : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<OutboxPublisherBackgroundService> _logger;

    public OutboxPublisherBackgroundService(
        IServiceScopeFactory scopeFactory,
        ILogger<OutboxPublisherBackgroundService> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessOutboxMessagesAsync(stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "An error occurred while processing outbox messages.");
            }

            await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
        }
    }

    private async Task ProcessOutboxMessagesAsync(CancellationToken cancellationToken)
    {
        using var scope = _scopeFactory.CreateScope();
        var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();
        var messagePublisher = scope.ServiceProvider.GetRequiredService<IMessagePublisher>();

        var messages = await dbContext.Set<OutboxMessage>()
            .Where(m => m.ProcessedOnUtc == null)
            .OrderBy(m => m.OccurredOnUtc)
            .Take(20)
            .ToListAsync(cancellationToken);

        foreach (var message in messages)
        {
            try
            {
                await messagePublisher.PublishAsync(message.Type, message.Content, cancellationToken);
                message.ProcessedOnUtc = DateTime.UtcNow;
            }
            catch (Exception ex)
            {
                message.Error = ex.ToString();
                _logger.LogError(ex, "Failed to publish message {MessageId}", message.Id);
                break;
            }
        }

        await dbContext.SaveChangesAsync(cancellationToken);
    }
}

Register the worker in Program.cs:

builder.Services.AddHostedService<OutboxPublisherBackgroundService>();

Key Considerations for Production

  • At-Least-Once Delivery: The outbox pattern guarantees delivery, not deduplication. If the publisher sends a message but crashes before writing ProcessedOnUtc, the message will be republished. Consumers must implement the Idempotent Consumer pattern.
  • Concurrency and Scaling: If multiple instances of your microservice run concurrently, use row-locking mechanisms (such as FOR UPDATE SKIP LOCKED in PostgreSQL or WITH (UPDLOCK, READPAST) in SQL Server) to prevent duplicate processing by competing workers.
  • Table Maintenance: Regularly archive or purge processed records to keep table sizes manageable and prevent index fragmentation.
  • Production Libraries: For advanced features like automated retries, partition leasing, and native broker transports, consider utilizing established libraries such as MassTransit or Wolverine, both of which provide built-in, production-ready outbox implementations.