Almost every API I have worked on eventually grows one endpoint that takes minutes to finish, sometimes longer. It could be a report that aggregates years of data, a bulk import job, or a workflow that calls three external services and a database before it can respond. The first time this happens, most teams just let the request run and hope the connection does not time out.
That approach creates two problems at once. Your users stare at a loading spinner for several minutes, and your API holds the connection open the entire time, tying up a thread, a database connection, and a slot in your concurrency budget. A small traffic spike on that one slow endpoint can take the rest of your API down with it.
In this article, I want to walk through the progression I use to fix this problem, starting from the simplest blocking version and ending at a fully decoupled, queue backed worker pool. Depending on your traffic and requirements, you might stop at any point along this path, but it helps to understand the full picture before deciding where to stop.
Step 0: The Naive Version
In the naive version, a user sends a request, the application server does the work inline, the work takes five minutes, and the connection stays open for the entire duration. There is nothing technically wrong with this. It is just paying for correctness with availability.
The user experience is poor and the blast radius is large. Every long running request you accept is capacity you cannot use anywhere else, whether that capacity was needed for another user’s request or for your own health checks.
The first mental shift you need to make is realizing that response time and work duration do not have to be the same thing. Your API can respond quickly even when the actual work behind it takes much longer to finish.
Step 1: Accept the Work, Don’t Do It
The first real fix is to stop doing the work inside the request thread. I add a jobs table that represents the piece of work I intend to do, and the API endpoint changes to do three things.
- Validate the incoming request.
- Insert a row into the jobs table with status Pending.
- Return 202 Accepted along with a job ID.
A background processor running inside the same API process picks up Pending rows and works through them one at a time. The client can either poll a GET /jobs/{id} endpoint to check status, or, better, you push updates through SignalR, Server-Sent Events, or a notification once the job finishes.
This step alone buys you a lot. The endpoint now returns in milliseconds, the user gets a job ID to track, and a spike of incoming requests just becomes a spike of rows in a table, which is cheap to write to. But there is a ceiling here, and most teams reach it faster than they expect.
Step 2: Decouple the Worker From the API
The background processor from Step 1 still lives inside your API process. It competes with your regular endpoints for the same CPU, memory, and database connection pool. When processing gets heavy, your API starts feeling it, which is exactly the situation you were trying to avoid in the first place.
The fix is to pull the background processor into its own deployable unit and put a message queue between the API and the workers. The API publishes a message to the queue, optionally still writing a job row for tracking and status lookups, and a pool of workers consumes from that queue and does the actual work. This is the competing consumers pattern, the same shape you would use in any event driven architecture built around a broker like RabbitMQ or Azure Service Bus.
Three things change once you cross this line. The queue absorbs traffic spikes, so your API keeps accepting work at a steady rate while the workers drain the queue at their own pace. You can scale workers independently of the API, so more throughput on background jobs does not force you to run more API instances. Failures also become routine rather than exceptional, since a worker crash just means the message goes back on the queue instead of the user seeing a 500 error.
You also get retryability, pause and resume, structured error handling, and a dead letter queue for messages that keep failing, mostly for free, because the queue infrastructure already gives you these.
What This Costs You
None of this is a free upgrade, and it is worth being honest about the cost before you commit to it. You are now running a queue, a fleet of workers, and a notification path, which means more moving parts to deploy, monitor, and set alerts on.
Your “is this done yet” logic is no longer obvious from the HTTP response. The client has to ask, or you have to push a notification to it. Every job you write also needs to be idempotent, because most queues guarantee at least once delivery, which means your workers will occasionally see the same message twice.
If you only have one slow endpoint and modest traffic, this entire setup is overkill. A simple fire and forget approach with status polling inside the same process works fine at that scale. I would not reach for a queue until the pain in production actually justifies the extra moving parts.
When I’d Reach for a Managed Cloud Service Instead
You do not always need to assemble all of this from parts yourself. A few alternatives worth considering, depending on the shape of the workload, are listed below.
- AWS SQS with Lambda, or Azure Service Bus with Azure Functions, when you want the worker pool to scale to zero and do not want to manage hosts yourself.
- Azure Durable Functions or AWS Step Functions when the work is really a multi step workflow involving timers, retries, and human approval steps. Orchestration is exactly what these tools are built for.
- Temporal when the workflow is long lived, running for hours or days, and you need durable execution, versioning, and visibility across runs as first class features.
The trade-off here is the usual one in cloud architecture: less operational work in exchange for tighter vendor coupling and a pricing model you need to model carefully as throughput grows. I have seen teams get surprised by their Lambda or Functions bill once a workload that looked occasional turned into a steady background load.
Summary
The progression is straightforward, and it generalizes well beyond this one example.
- Do not do slow work inside the request. Accept it, persist it, and return 202 Accepted.
- Do not run workers inside the API process. Put a queue in between and scale the two sides independently.
- Tell the user when the work is done. Polling works, pushing a notification is better.
This is not really an argument for microservices. It is a separation between accepting work and doing work, two concerns that have very different scaling profiles and very different failure modes. Treating them as one thing is usually where the pain starts.
Leave a Reply