Background Jobs, Performance and Scale

Real-world analogy

What is it?

Background processing means doing work outside the HTTP request, so the user gets a fast response. In ASP.NET Core you can run a BackgroundService that reads jobs from a Channel, or use a library such as Hangfire or Quartz when jobs must survive restarts, run on a schedule or be retried. Performance work is a loop of measuring, finding the real bottleneck and fixing it. Scaling out is possible when the app keeps no per-user state in its own memory.

Real-world relevance

A checkout page that sends a receipt email inside the request is slow, and it fails if the mail server is down. Moving the email to a queue makes the page fast and lets the worker retry. When a report endpoint is slow, teams that profile often find one N+1 query rather than a CPU problem. And an app that stores sessions in server memory often breaks when a second server is added, because the next request may reach a server that does not know the user.

Key points

• Move slow work off the request — Sending email, building a report or importing a file should not make the user wait. Save the request, queue the work and return quickly.
• BackgroundService runs with the app — Inherit BackgroundService and register it with AddHostedService. It starts with the app and stops with it, and can loop over a queue.
• Create a scope per job — A hosted service is a singleton, so a DbContext in its constructor would live as long as the app. As in lesson 9, inject IServiceScopeFactory and create a scope for each job.
• Channel<T> as an in-memory queue — A controller writes a job, and a worker reads it asynchronously. A bounded channel pushes back when full. Queued jobs are lost if the process stops.
• Hangfire and Quartz for more — These libraries can store jobs in a database, so they survive restarts. They add scheduling, retries and a dashboard.
• Jobs should be idempotent — A job may run twice after a retry or a crash. Design it so running twice has the same effect as once, for example by checking a status before sending.
• Measure before you optimise — Guessing wastes days. Use logs, metrics and a profiler to find where the time goes, then change one thing and measure again.
• Usual bottlenecks, in order — In many apps the database is first: slow queries, N+1 and missing indexes. Chatty calls to other services and missing caches follow, and plain CPU work is less common.
• Scaling out needs statelessness — Vertical scaling gives one server more CPU and memory, and has a ceiling. Horizontal scaling adds servers, which works when no request depends on one server's memory.

Code example

public record EmailJob(int OrderId);
public class EmailQueue
{
    private readonly Channel<EmailJob> _channel = Channel.CreateBounded<EmailJob>(100);

    public ValueTask EnqueueAsync(EmailJob job, CancellationToken ct) =>
        _channel.Writer.WriteAsync(job, ct);

    public IAsyncEnumerable<EmailJob> ReadAllAsync(CancellationToken ct) =>
        _channel.Reader.ReadAllAsync(ct);
}

public class EmailWorker(EmailQueue queue, IServiceScopeFactory scopes, ILogger<EmailWorker> log)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var job in queue.ReadAllAsync(stoppingToken))
        {
            try
            {
                using var scope = scopes.CreateScope();            // fresh scope per job
                var sender = scope.ServiceProvider.GetRequiredService<IReceiptSender>();
                await sender.SendAsync(job.OrderId, stoppingToken);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                log.LogError(ex, "Email job for order {OrderId} failed", job.OrderId);
            }
        }
    }
}
// Program.cs
builder.Services.AddSingleton<EmailQueue>();
builder.Services.AddHostedService<EmailWorker>();

Line-by-line walkthrough

1. 1. EmailJob is a small record that holds only the order id, not the whole order.
2. 2. EmailQueue wraps a bounded Channel of 100 jobs, so a flood of requests cannot fill memory.
3. 3. EnqueueAsync writes a job to the channel, and ReadAllAsync gives the worker a stream of jobs.
4. 4. EmailWorker inherits BackgroundService, and ExecuteAsync loops with await foreach until the app stops.
5. 5. For each job it creates a scope and resolves IReceiptSender from it, so scoped services live and die with that job. IReceiptSender is a scoped service, not shown, that uses the DbContext and sends the email.
6. 6. The catch block logs a failed job and lets the loop continue, but it lets cancellation through.
7. 7. In Program.cs, the last two lines register the queue as a singleton, so the controller and the worker share one channel, and add the worker as a hosted service.

Spot the bug

[HttpPost]
public IActionResult Create(CreateReport req)
{
    _ = Task.Run(async () =>
    {
        var rows = await db.Orders.Where(o => o.CreatedAt >= req.From).ToListAsync();   // db is the request's scoped DbContext
        await reports.SaveAsync(rows);
    });
    return Accepted();
}

Explain like I'm 5

Fun fact

Hands-on challenge

More resources

• Background tasks with hosted services in ASP.NET Core (Microsoft Learn)
• Channels in .NET (Microsoft Learn)
• ASP.NET Core best practices (Microsoft Learn)