Prefix the unknown-tool error with the "cannot" convention and drop
the duplicate wrap around executeSingleTool: that helper already wraps
its generic error path, so the outer wrap produced messages shaped like
"cannot execute tool X: cannot execute tool X: ...".
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
MaxTurns is the only agent bound compared against a counter that is
serialised in the checkpoint (Turns). When config drifts between save
and restore -- typically because a deploy changed WithMaxTurns or a
different build of the agent is registered by name -- cp.Turns can
exceed agent.maxTurns on the resumed run, which previously surfaced
as a warning log and then a MaxTurnsExceededError on the first
iteration of the resumed coreLoop.
Capture MaxTurns in the new AgentConfig on every save, and on
restore clone the registry-resolved agent with WithMaxTurns applied
from the snapshot. The override flows through the outer Restore path
and through both inner-agent resolution sites in
restoreNestedSuspended and restoreAwaitingApproval, so nested
runs get the same treatment. Other loop bounds
(maxEmptyOutputRetries, maxToolDepth) reset per turn / per tool
depth and stay intentionally live so deploys can tune them without
invalidating in-flight checkpoints. Live references (tools, hooks,
LLM client, approval callbacks, guardrails) are not snapshotted for
the same reason.
With the snapshot in place, the "restored agent run has already
reached max turns" warning at the top of continueFromMessages is
structurally unreachable -- the live agent's bound is now the same
value cp.Turns was bounded by at save time -- and is removed.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
executeParallel ignored SuspendedError when checkpoint was nil,
treating it as a normal tool error. Nested suspension propagation
also dropped the in-memory checkpoint when persistence failed,
making runs non-resumable.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Poll-based supervisor that claims PENDING agent runs with FOR UPDATE
SKIP LOCKED, runs them with lease-based heartbeat, and handles
graceful shutdown. On infrastructure stop the row stays RUNNING so
stale recovery resets it to PENDING on restart; Restore picks up
from the last checkpoint. Heartbeat loss cancels execution without
committing a terminal status.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
coreLoop now saves incremental checkpoints after each tool-call turn
and checks a cooperative stop signal at turn boundaries. SuspendedError
is handled in finishRun, executeParallel, and executeSingleTool.
Approval-interrupted checkpoints are persisted for both flat and
nested interruptions.
Introduce RunOption, WithCheckpointStore, RunWithOpts, ResumeWithOpts,
and RunStreamedWithOpts so callers can provide checkpoint storage.
Add StreamEventSuspended and OnRunRestore hook.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
executeSingleTool had three exit paths but only the success path
emitted all end signals. The interrupted path (nested agent
approval) skipped OnToolEnd and StreamEventToolEnd entirely,
leaving hook consumers with an unpaired OnToolStart. The error
path also missed StreamEventToolEnd and AgentHooks.OnToolEnd.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
When the LLM returns tool calls after a handoff in the same
assistant message, they were silently dropped. This left
orphaned tool_call entries without matching tool-result
messages, causing protocol errors on the next LLM turn.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
