Production Runtimes

In plain words

When you order on Swiggy, the app does not freeze until the food arrives. It replies at once and updates you later. Your agent needs the same thinking. Before you pick any library, decide the shape it runs in: does the user wait, does the answer stream, does it go into a background queue, does it wake on an event, or does it run on a nightly timer?

How it flows

A tiny example

def handle(task):
    if task.seconds < 30:
        return run_agent(task)      # user waits
    job_id = queue.push(task)       # user does not wait
    return {"job_id": job_id}

def worker():
    job = queue.pop()
    try:
        run_agent(job)
    except Exception as e:
        dlq.append((job, e))        # never lose a failed job

Notice that the agent logic never changes; only the shell around it decides who waits and where failures land.

What you will learn

• The six shapes an agent can run in when it goes live.
• Why a long task must never sit on a normal HTTP request.
• What durable execution, queues, dead-letter queues and cron actually do.
• How to pick the shape first, and the library later.

The problem, simply

Think about ordering food on Swiggy. You tap "Place order" and the app does not freeze for forty minutes with a spinner until the delivery partner reaches your hostel gate. It replies in one second with "Order placed", and then keeps updating you separately.

Imagine if it did freeze. Screen stuck, no back button, and if your network dropped for two seconds the whole order is gone. Nobody would use that app.

Your agent has exactly this problem. In your Jupyter notebook, the agent runs in one cell and you sit and watch. In production it is different. The network dies at step 37. The user closes the tab in the middle. The server reboots and your nightly job is wiped out.

See, the code inside the agent is the same in both cases. What changes is the shell around it — where it runs, who waits for it, and what happens when something breaks. That shell is called the runtime shape. Choosing it wrongly is the most common production mistake.

The idea

There are six shapes. Learn the shapes first. The framework you use is a detail after that.

The six shapes

• Request-response. User asks, waits, gets the answer. Plain synchronous HTTP. Fine only for short tasks, roughly under 30 seconds.
• Streaming. The answer comes out piece by piece as it is generated, like ChatGPT typing on screen. Same total time, much better feeling.
• Durable execution. The runtime saves the agent's state after every single step. If the machine dies at step 37, it restarts from step 37, not from step 1. This saving is called checkpointing.
• Queue-based (background). The job is dropped into a queue. Separate worker processes pick jobs up and run them. The result comes back later through a webhook or a notification.
• Event-driven. The agent sits idle and wakes up when something happens — a new email arrives, someone opens a pull request, a payment fails.
• Scheduled (cron). The agent runs on a timer. Every night at 2 AM, every Monday, every hour. Cron is just the old Unix name for "run this on a schedule".

IMP

Note: These are not competitors. One real product usually uses three or four of them together.

A worked example

Suppose Priya is building an agent for a placement cell. It reads a company's job description, checks 400 student profiles, and writes a shortlist with reasons.

She first tries request-response. The HR person clicks a button and waits. The agent takes about 6 minutes. The browser times out at 60 seconds, so the HR person clicks again. Ten clicks later her server has ten heavy jobs running and falls over.

So Priya switches shape. The click now just drops a job into a queue and instantly returns "Shortlist is being prepared". A worker process picks it up, runs the 6 minutes quietly, and emails the result. If a worker crashes at student 250, a durable runtime resumes from student 250. And every night at 2 AM, a cron-shaped agent recomputes the college's placement summary.

Same agent logic. Four different shells. Only the shell changed, and the product became usable.

Queues need a dead-letter queue

When a background job fails, where does it go? If you did not plan for this, the answer is: nowhere. It just vanishes, and nobody finds out until a student complains that his shortlist never came.

A dead-letter queue, DLQ for short, is a parking lot for failed jobs. After a job fails its retries, it lands in the DLQ instead of disappearing. Now you can count them, look at them, and replay them.

IMPRemember: a queue without a DLQ is a queue that silently eats your users' work.

You cannot debug what you cannot see

Long agents run dozens to hundreds of steps for a single task. Anthropic said this openly when they announced computer use — that many steps per task is normal, not unusual.

Now your agent fails at step 40. Which step? What did it call? What came back? If all you have is a "failed" log line, you have nothing. You must run the whole thing again with extra print statements.

So production agents export traces — a record of every step with timing and inputs and outputs. OpenTelemetry (OTel) is the common standard for this, and tools like Langfuse, Phoenix and Opik store and display those traces. This is not a nice-to-have for multi-step agents. It is the difference between fixing a bug in ten minutes and fixing it in two days.

Build it

This one file runs the same tiny "agent" through four shapes so you can see the difference in behaviour.

"""Same agent logic, four different runtime shells."""
import time
from collections import deque

def agent(task, steps=3):
    """Toy agent. Yields one line per step. No real model call."""
    for i in range(1, steps + 1):
        time.sleep(0.05)              # pretend the model is thinking
        if "bad" in task and i == 2:  # a deliberate failure for the demo
            raise RuntimeError("step 2 blew up")
        yield f"step {i} of '{task}' done"

# 1. Request-response: caller waits for everything, gets one answer.
def request_response(task):
    return list(agent(task))[-1]

# 2. Streaming: caller sees each step as it happens.
def streaming(task):
    for line in agent(task):
        print("   chunk:", line)

# 3. Queue-based with a dead-letter queue.
jobs, dlq, done = deque(), [], []

def submit(task):
    jobs.append({"task": task, "tries": 0})

def worker(max_tries=2):
    while jobs:
        job = jobs.popleft()
        try:
            done.append(list(agent(job["task"]))[-1])
        except RuntimeError as e:
            job["tries"] += 1
            if job["tries"] < max_tries:
                jobs.append(job)       # retry it
            else:
                dlq.append((job["task"], str(e)))   # park it, never drop it

# 4. Event-driven: handlers wake up when an event fires.
handlers = {}

def on(event):
    def register(fn):
        handlers.setdefault(event, []).append(fn)
        return fn
    return register

def fire(event, payload):
    for fn in handlers.get(event, []):
        fn(payload)

@on("new_submission")
def review(payload):
    print("   event handler ran for:", request_response(payload))

print("1. request-response ->", request_response("shortlist Priya"))
print("2. streaming ->")
streaming("shortlist Rahul")
submit("shortlist Sneha"); submit("bad profile"); worker()
print("3. queue -> done:", done, "| dlq:", dlq)
print("4. event ->")
fire("new_submission", "PR from Karthik")

Look at the queue line in the output. The good job lands in done, and the failing job is retried once and then parked in dlq — it is never silently lost. Compare shape 1 and shape 2: identical work, but streaming shows progress instead of one long silence.

Where you will see this

• Claude Code and Cursor stream tokens to your editor while the agent is still working.
• ChatGPT's normal chat is streaming; its longer research-style tasks are queue-shaped and notify you when done.
• A Swiggy or Zomato support bot answers instantly in request-response, but refund checks go to a background queue.
• GitHub Copilot's pull-request review agents are event-driven — they wake when a PR is opened.
• Cost reports, nightly evaluation runs and daily digest emails are all cron-shaped agents.

Common mistakes

• Putting a 5-minute task on a synchronous HTTP request. Users hang up, click again, and each click starts another heavy run until the server dies.
• A queue with no DLQ. Failed jobs vanish quietly. You find out only when a user complains, and by then you have no record of what failed.
• Background work with no traces. If nobody is watching the worker, failures are invisible. The agent can be broken for days.
• Skipping durable state on long runs. If a 20-minute job restarts from step 1 on every crash, it may never finish on a bad day.
• Choosing a framework first. People pick a library and then bend the product around it. Decide the shape from the task, then pick tools that support it.

If they ask in an interview

Q: Your agent takes 4 minutes per request. How will you deploy it?

A: Not on a synchronous endpoint — browsers and load balancers time out well before that. I would accept the request, push a job to a queue, return a job id immediately, and have workers process it with retries and a dead-letter queue. The client polls or gets a webhook when the result is ready.

Q: What is durable execution and when do you need it?

A: The runtime checkpoints the agent's state after every step, so a crash resumes from the last successful step instead of the beginning. You need it when the number of steps is unknown and re-running from scratch is expensive in time or money — long research tasks, multi-hour data jobs.

Q: Why is tracing so important for agents specifically?

A: A single agent task can be dozens to hundreds of steps, so "it failed" tells you nothing. Traces record each step's input, output and timing, so you jump straight to the step that broke instead of re-running everything with extra logging.

Try these

1. Take the queue demo and make jobs fail randomly about 10 percent of the time. Run 50 jobs and print how many ended in the DLQ.
2. Convert the streaming function so it also prints the time gap between chunks. That gap is your per-chunk latency.
3. Write a tiny cron-shaped loop that runs a task every 2 seconds, five times, and keeps a counter in a small text file so it survives a restart.
4. Take any small agent you have written and write two lines for each of the six shapes saying whether it fits and why.

Words, simply

Quick recap

• Pick the runtime shape from the task first; pick the framework after.
• Anything longer than about half a minute belongs in a queue or a durable runtime, never on a plain HTTP request.
• Without a DLQ and without traces, your agent fails silently and you debug blind.