An agent harness takes a task and works it across many steps, many tool calls, and increasingly across long stretches of wall-clock time. A single model call is stateless and finishes in seconds. A harness running a real task is neither stateless nor fast. It carries context from one step to the next, and at some of those steps it has to stop and wait for something outside itself before it can continue.

That wait is an architectural problem, and most agent code today doesn’t treat it as one.

The wait is the hard part

When an agent blocks on something external, it is waiting on a clock it doesn’t control. There are three common shapes of this, and they show up in most real systems.

The first is a human decision. The agent proposes an action that needs sign-off before it runs, and a person approves or rejects it on their own schedule. Sometimes that is immediate. Often it takes hours, and sometimes it takes days.

The second is an agent-to-agent handoff. One agent delegates to another that is itself mid-task, and the caller has to wait for a result that may be minutes or much longer away.

The third is a long external callback. The agent kicks off a job, a webhook, or a batch process and has to resume when that process reports back, which might be well after the agent’s own turn would normally end.

All three have the same requirement. The run has to stay alive across a gap that can be long and unpredictable, and it has to survive whatever happens to the infrastructure during that gap.

The naive way to hold the wait is in memory. The agent calls a tool, reaches the point where it needs approval, and sits on an open await with the conversation and the pending decision held in the process. This works only as long as the process stays up. Workers redeploy and pods get rescheduled, and either of those routine events destroys an in-flight run whose only record was the memory of the process running it.

Holding the wait durably means the run’s state lives somewhere that outlives the process. If the worker dies, the run is restored somewhere else and continues from where it stopped. This is the difference between an agent that survives a deploy and one that has to be restarted from scratch every time the platform underneath it moves.

A concrete wait: refund approval

Consider a support agent that issues refunds. It takes a request like “Refund $40 for order A123,” decides to call an issue_refund tool, and has to wait for a human to approve before any money moves.

Walk through what a routine failure does to the in-memory version.

At 17:47 on a Friday, the agent reaches the approval step and pauses. The approver has already left for the weekend. On Saturday night, the cluster autoscaler drains the node the worker is running on, and the worker pod is rescheduled. On Monday morning the approver clicks approve. In the in-memory version there is nothing left to approve, because the run died with the pod on Saturday. The conversation and the pending refund are gone.

This is the case the rest of the post builds. The agent runs on Microsoft Agent Framework, the model is served by Microsoft Foundry, and Temporal makes the run durable. I kept the refund agent small on purpose. It is here to show how the pieces fit, not because refunds are special.

The real tax: state that has to outlive the process

The tax you pay for a durable wait is that the agent’s state has to persist across the gap, and persist correctly when the process handling it disappears mid-run.

The tempting solution is a second state system. You write the pending approval to a database, add a poller, reload the conversation when the approval lands, and reconstruct where the agent was. Now there are two systems, the agent runtime and the state store, and the bugs live in the seam between them. If the process dies after the tool is approved but before the agent resumes, you are writing reconciliation logic for a distributed system you did not set out to build.

Temporal removes the need for a separate application-owned checkpoint store for the orchestration state. The workflow’s event history becomes the source of truth. If the process running it dies, another worker can replay the workflow from the beginning against that history, reconstruct its state, and continue from the first point that requires new progress.

LLM in an activity, workflow stays deterministic

Temporal recovers a run by replaying it, and replay only works if the workflow is deterministic. Given the recorded event history, the workflow has to make decisions compatible with the ones it made before. An LLM call breaks that requirement because the same prompt can return different tokens and different tool choices on each run.

The design follows from that constraint. The non-deterministic work goes in an activity, and the workflow stays pure orchestration.

The agent turn, which includes the LLM call, the network I/O, and the tool decision, runs inside a Temporal activity called run_agent_turn. Activities may be non-deterministic. Once an activity completion is recorded, workflow replay returns that recorded result instead of running the activity again. An attempt whose completion was not recorded can still execute again, which is why external side effects must be idempotent. The workflow never calls a model. It moves serialized messages between activity calls and decides when to pause.

@workflow.run
async def run(self, prompt: str) -> str:
    history: list[dict] = []
    approvals: list[dict] | None = None
    first_prompt: str | None = prompt

    while True:
        result = await workflow.execute_activity(
            "run_agent_turn",
            args=[history, approvals, first_prompt],
            start_to_close_timeout=timedelta(seconds=120),
            retry_policy=RetryPolicy(maximum_attempts=5),
        )
        first_prompt = None
        history += result["messages"]
        if not result["pending"]:
            return result["text"]

        # The durable pause. This can span minutes or days with no compute running.
        self._pending = result["pending"]
        await workflow.wait_condition(
            lambda: all(req["id"] in self._decisions for req in self._pending)
        )

        approvals = [
            {"request": req, "approved": self._decisions[req["id"]]}
            for req in self._pending
        ]
        self._pending = []

The workflow imports no agent_framework, no Azure SDK, and no model client. It loads inside Temporal’s workflow sandbox, which restricts known non-deterministic operations but does not prove that workflow code is deterministic. The conversation moves as a list of plain dicts, and the workflow never needs to know the wire shape of a Microsoft Agent Framework Message. The activity handles that translation at the boundary, which keeps the deterministic core clean.

The agent itself is ordinary. It is a model client pointed at Foundry with a single tool, and the tool is gated behind approval.

@tool(approval_mode="always_require")
def issue_refund(order_id: str, amount_usd: float) -> str:
    """Issue a customer refund. Requires human approval before it runs."""
    return _perform_refund(order_id, amount_usd)

approval_mode="always_require" does the work on the agent side. In the framework’s automatic function-invocation loop, the tool does not run until approval arrives. Pending input is exposed through result.user_input_requests; a function approval is currently a Content item with the type function_approval_request. The activity returns that item to the workflow as a pending approval. The framework raises the question, Temporal holds it open, and a human answers whenever they get to it.

The durable pause: wait_condition

One call carries the durability.

await workflow.wait_condition(
    lambda: all(req["id"] in self._decisions for req in self._pending)
)

This is neither a poll nor a sleep. Temporal schedules no workflow task merely to check the condition, so the wait consumes no worker compute while it is pending. When the human decides, a signal updates the state, a worker processes the new event history, and the workflow continues. The pause can last thirty seconds or three days, and because no process is holding it open, there is nothing to keep alive when infrastructure moves underneath it.

The decision arrives as a Temporal signal, which is a method on the workflow that updates its durable state:

@workflow.signal
async def submit_approval(self, decisions: list[dict]) -> None:
    for d in decisions:
        self._decisions[d["id"]] = bool(d["approved"])

Because the approval is a signal against durable state, it is decoupled from the run. The process that started the agent can be gone. The approver signals from a different machine, a script, or an internal tool, minutes or days later, and Temporal routes the signal to the parked workflow. Driven from the command line, the flow is two decoupled steps:

# start a run — it pauses at the approval gate and prints the request
uv run python -m durable_agent.client start "Refund $40 for order A123"

# approve later, from anywhere. --deny to reject.
uv run python -m durable_agent.client approve <workflow_id>

The gap between start and approve is where the durability earns its place. Everything else is supporting plumbing.

Proving the recovery path offline

Designing for failure means you have to be able to test the recovery path, and test it without spending real model tokens every time. The approval logic is the part most likely to break, so it is worth exercising in isolation. Temporal’s time-skipping test environment and a stub activity let you drive the whole pause-and-resume loop with no Azure and no LLM in the loop. Time skipping becomes useful once the workflow also has timers, approval deadlines, or retry backoff; the signal-only test here does not need to advance the clock.

@activity.defn(name="run_agent_turn")
async def stub_turn(history, approvals=None, prompt=None) -> dict:
    if not approvals:
        return {"messages": [], "pending": [_REQUEST], "text": ""}
    approved = approvals[0]["approved"]
    return {
        "messages": [],
        "pending": [],
        "text": "refund issued" if approved else "refund denied",
    }

Swap the real turn for that stub, drive the workflow to the approval gate, signal it, and assert both branches resolve. Approval produces “refund issued” and denial produces “refund denied.” The test runs the same workflow code that production runs, because only the activity is stubbed. The determinism split is what makes this possible, and the same property that makes the workflow recoverable is what makes it testable.

What I won’t claim

This is a minimal implementation, and a few edges matter if you take the pattern further.

The tool runs inside the turn activity, so a retry re-invokes it. The activity in the workflow above has RetryPolicy(maximum_attempts=5), and the refund side effect runs inside that turn, so a transient failure and retry can run it twice. In this example the tool returns a formatted string, so a double call is harmless. Against a real payments API it is a double refund.

The fix is a stable idempotency key for the specific refund operation, enforced by the payment API. For a dedicated refund activity, the workflow run ID and activity ID form a key that remains stable across retries. If you also want per-tool retry, timeout, and observability, the tool moves out to its own workflow-scheduled activity with its recorded result fed back. That activity must still be idempotent, and the extra boundary is worth adding once the tool does real I/O.

The conversation is application-managed message history rather than service-managed state. Every turn rebuilds the full message list and hands it back to the model, which is the simplest approach and keeps Temporal as the source of truth for orchestration. This is distinct from Temporal workflow replay. The messages passed into and returned from activities are persisted in event history, so an ever-growing conversation increases history size and may be readable to operators who can access that history. Longer-running agents need a strategy such as summarization, encrypted payloads, external payload storage, or Continue-As-New.

If you would rather use a service-managed conversation in Microsoft Foundry Agent Service, you persist its conversation or session identifier in the workflow instead of resending application-managed messages. That reduces message duplication in Temporal but reintroduces an external state dependency, so I would keep application-managed history until its size or model-context cost justifies the trade-off.

Durability is not free, and not every agent needs it. Temporal is real infrastructure, with a server, a worker, and a task queue to run. A short agent with no external waits and no valuable intermediate progress may not justify that machinery, even though retries and crash recovery can still be useful. The case becomes much stronger at the wait. Once an agent has to stop and depend on something slow and outside itself, in-memory state stops being sufficient and durable execution starts paying for itself.

Architecting for failure

Most agent frameworks today assume the happy path. The process stays up, the run lives in memory, and failure is treated as an exception to recover from by hand. That assumption is fine in a notebook and wrong in production, where deploys and evictions are normal events rather than rare ones.

Durable execution lets you invert that assumption. When the run’s state lives outside any single process, a failure mid-wait is something the architecture already handles, because the run is restored elsewhere and continues. You stop designing around keeping a process alive long enough and start designing on the assumption that any process can die at any time. For agents that touch money or wait on people, that assumption is the honest one to build on.

The code for the refund agent above, including the full workflow, the offline test, and an azd up path that provisions Foundry and AKS, is at alisoliman/temporal-maf-durable-agents if you want to work through a running version.