Service-to-Service Communication

Real-world analogy

What is it?

When a system is split into services, they must talk over a network, and networks fail. Synchronous calls such as REST or gRPC are simple but couple the caller to the callee being up. Asynchronous messaging through a broker decouples them in time, but brings duplicates, ordering questions and delayed results. Good designs add timeouts, retries and circuit breakers, make consumers idempotent, and use an outbox so that saving data and announcing it cannot get out of step.

Real-world relevance

A common failure is an order that is saved in the database while the "order placed" message is lost because the broker was down for a second. The customer paid, but no email, no stock update and no invoice follow. Another is a message delivered twice and a customer charged twice. Teams fix these with an outbox table and idempotent consumers, and they accept that other services see the order slightly later.

Key points

• Synchronous: ask and wait — With REST or gRPC, the caller waits for the other service to answer. You get the result right away, but your request fails whenever that service is down or slow.
• Asynchronous: send a message — The sender puts a message on a broker such as RabbitMQ or Azure Service Bus and moves on. Services are less tied together, but results arrive later.
• Timeouts, retries, breakers — Set a timeout so a slow service cannot keep your requests open. Retry only calls that are safe to repeat, with growing delays. A circuit breaker stops calling a service that keeps failing.
• At-least-once means duplicates — Brokers usually promise the message arrives at least once, so it can arrive twice. The consumer must be idempotent: handling it twice gives the same result as once.
• Idempotent consumer — Store the id of each processed message, and skip ids already seen. Saving that id in the same transaction as the work keeps the check reliable.
• The outbox pattern — Save the business change and an "outbox" row in one database transaction. A separate publisher reads the outbox and sends the messages, retrying until done. It can send a message twice after a crash, so consumers still need to be idempotent.
• Eventual consistency — After an order is saved, the email service learns about it a moment later. For a short time the systems disagree, and the design must accept that.
• Saga and API gateway — A saga is a chain of local transactions, one per service. If a later step fails, compensating steps undo the earlier ones. An API gateway is one front door that routes, authenticates and limits calls to the services behind it.

Code example

public record OrderPlaced(Guid MessageId, Guid OrderId);
public class OutboxMessage { public Guid Id { get; set; } public string Type { get; set; } = ""; public string Payload { get; set; } = ""; public DateTime? SentAt { get; set; } }
public class ProcessedMessage { public Guid Id { get; set; } }   // primary key blocks duplicates

public class PlaceOrderService(ShopContext db)
{
    public async Task PlaceAsync(Order order, CancellationToken ct)   // Order.Id is a Guid set at creation
    {
        var messageId = Guid.NewGuid();
        db.Orders.Add(order);
        db.Outbox.Add(new OutboxMessage
        {
            Id = messageId,
            Type = nameof(OrderPlaced),
            Payload = JsonSerializer.Serialize(new OrderPlaced(messageId, order.Id))
        });
        await db.SaveChangesAsync(ct);   // order and outbox row commit together
    }
}

public class OrderPlacedConsumer(ReceiptsContext db)
{
    public async Task HandleAsync(OrderPlaced msg, CancellationToken ct)
    {
        if (await db.Processed.AnyAsync(p => p.Id == msg.MessageId, ct))
            return;                                          // duplicate: already handled

        db.Receipts.Add(new Receipt { OrderId = msg.OrderId });
        db.Processed.Add(new ProcessedMessage { Id = msg.MessageId });
        await db.SaveChangesAsync(ct);                       // work and marker commit together
    }
}

Line-by-line walkthrough

1. 1. OrderPlaced carries a MessageId, so any consumer can recognise the same message if it arrives again.
2. 2. OutboxMessage is a row in the same database as the orders, holding the message until a publisher sends it. ShopContext exposes it as an Outbox DbSet, not shown here.
3. 3. ProcessedMessage has the message id as its primary key. If two copies run at the same moment, the second insert fails and its receipt rolls back with it.
4. 4. PlaceAsync adds the order and the outbox row, then calls SaveChangesAsync once, so both are in one transaction.
5. 5. A separate publisher (not shown) reads unsent outbox rows, sends them to the broker, and sets SentAt.
6. 6. The consumer first checks for the message id and returns early on a duplicate. ReceiptsContext, with its Receipts and Processed DbSets and a simple Receipt entity, is not shown.
7. 7. Receipts and Processed are saved in one SaveChangesAsync, so a crash cannot leave the work done but unmarked.

Spot the bug

public async Task<Guid> PlaceAsync(Order order, CancellationToken ct)
{
    db.Orders.Add(order);
    await db.SaveChangesAsync(ct);

    // broker is an IMessageBroker
    await broker.PublishAsync(new OrderPlaced(Guid.NewGuid(), order.Id), ct);   // broker down or app crashes here
    return order.Id;
}

Explain like I'm 5

Fun fact

Hands-on challenge

More resources

• Asynchronous message-based communication (Microsoft Learn)
• Implement resilient applications (Microsoft Learn)
• Saga design pattern (Microsoft Learn)