Skip to content

Event-Driven Agentic Flows

Editions: OSS. Contributor documentation for this repository.

This chapter covers the Flow subsystem: the Trigger Service, Flow Orchestrator, NATS queue, agent infrastructure, and data flows. Remote runners, matrix/batch fan-out, eval result artifacts, and evidence packs are included here.

Private runners are the default. resolve_runner_pool picks a pool in this order: trigger _runner override, flow.runner_pool, account.default_runner_pool, any online private runner (auto), then the hosted executor. The literal server at any explicit level opts into hosted compute. A resolved private pool uses RemoteRunnerExecutor to lease work to a matching self-hosted runner; if no runner in that pool has a free slot the job queues for 15 minutes (DEFAULT_QUEUE_TIMEOUT) and then fails. A runner holds min(owner ceiling, what the process reports) executions at once (default 2), one row per lease in flow_runner_assignment, and dispatch prefers the runner with the most free slots. Runners maintain an outbound WebSocket control plane for leases, heartbeats, status, logs, completion, and halt requests; durable lease metadata stays in PostgreSQL so temporary disconnects can recover without persisting short-lived API tokens. The CLI (preloop runner fg) redials that socket on unexpected close and keeps an in-flight Docker job, sending complete/logs after reconnect. Register/connect/disconnect also publish runner_updated on account-updates.{account_id} so the console Runners page updates without a refresh. When agent_config.execution_path is persistent, create_executor_for_execution selects AgentControlExecutor before that pool resolution and delivers the rendered prompt as one audited Agent Control send_message to target_agent_id; see Persistent flow execution.

Private Docker jobs carry a versioned transient launch specification generated by the hosted Codex/OpenCode adapters. Reconnect delivery rebuilds credentials and rejects changed leased configuration; scripts and credential environments are excluded from persisted lease JSON. Docker runs an explicit bootstrap, clears stale reports, and requires both exit zero and a recognized structured result.json verdict. The API checks this again before accepting success. The private protocol exports only the bounded report, not workspace archives. Unsupported harnesses fail explicitly. See the runner image contract.

The default private-runner job is still docker run of the flow image (or a custom image / docker_image). An opt-in host execution profile is a distinct private-only path: the runner advertises named local CLIs, the flow selects a name (agent_config.host_exec_profile with agent_type: cursor), and the host executes that fixed command under the operator's local login. Hosted compute rejects it. Native success uses completion_protocol: host_exec (structured Cursor stream-json result plus exit 0) and must not be treated as Docker launch v1. Remote checkout/setup, custom commands, workspace seeds, isolated publication and native CLI --resume fail closed on this path. Each job starts in a fresh empty directory. The workspace root controls working-directory placement, not filesystem isolation; Cursor has the runner user's local login, configuration, hooks and filesystem access. Flow MCP tool/server settings are not injected as a sandbox. Profiles advertise supported requested model identifiers mapped to local aliases; actual model attribution requires an observed Cursor result. This is not Agent Control and not preloop cursor.

Ad-hoc preset runs

POST /api/v1/flows/run-preset runs a catalog preset on a GitHub or GitLab issue without waiting for a tracker webhook. confirm_create=false only resolves the account flow (409 flow_missing when none exists). confirm_create=true clones the preset on first use, then triggers it. The auto-created flow has empty trigger_event_types and no trigger_event_source, so tracker events do not start extra runs until the operator edits the flow. Other tracker types (for example Jira) return 400.

Jira repository binding

A Jira-triggered flow with git clone enabled and no repositories of its own clones the code-host repository bound on the flow or the Jira project, and writes the opened pull request back to the issue as a comment and a remote link. See Jira project repository binding.

Prompt placeholders

Flow prompt_template strings are resolved before the agent starts. Besides {{project.*}}, {{account.*}}, and {{trigger_event.*}}, an execution resolver provides:

  • {{execution.id}}: this run's id.
  • {{execution.url}}: console URL {PRELOOP_URL}/console/flows/executions/{id}.
  • {{execution.resume_from}}: prior execution id when this run was started from a human comment on a PR this flow opened. Empty otherwise.
  • {{execution.ci_failure}}: when this run was started because GitHub CI failed on a PR this flow opened: provider, job name, and check URL. Empty otherwise.
  • {{flow.review_instructions}}: blocking review rules stored on the flow. Empty when unset. The Pull Request Reviewer reads this the same way it reads .preloop/review-policy.md, and truncates it at 16 KiB.

Any placeholder may take a |truncate(N) filter, for example {{trigger_event.payload.object_attributes.description|truncate(16384)}}. N is a byte cap (not a character count) so it matches the launch-payload and tokenizer costs that actually grow. The cut is on a UTF-8 boundary. {{name|truncate}} without N uses 16 KiB. When the resolved value is longer, the injected text is the prefix plus:

[truncated by Preloop: showing the first N bytes of TOTAL; fetch the full text
with the tool that owns this object, for example get_pull_request]

Unbounded webhook fields are capped because they dominate context window, reviewer attention, and the kernel execve string limit (one argv element or one NAME=value entry cannot exceed 128 KiB). Templates without the filter are unchanged. Operator-facing grammar also lives in Webhook Triggers.

Agent launch payload (container environment)

Custom agent images and private runners must not assume the rendered prompt arrives as one environment variable. Linux caps a single execve string (one argv element or one NAME=value entry) at MAX_ARG_STRLEN (131072 bytes). The control plane delivers large text as base64 chunks and reassembles files inside the container:

Variable Meaning
PRELOOP_AGENT_PROMPT_0..N Base64 chunks of the rendered prompt (96 KiB each).
PRELOOP_AGENT_PROMPT_CHUNKS Chunk count (0 when the prompt is empty).
PRELOOP_AGENT_PROMPT_BYTES Decoded byte length, used to abort a truncated reassembly.
AGENT_PROMPT_FILE Path the chunks materialize at (/tmp/preloop/prompt.txt).
AGENT_PROMPT Whole prompt, only when it is <= 64 KiB. Absent above that.
PRELOOP_INNER_SCRIPT_0..N Base64 chunks of the Kubernetes inner agent script.
PRELOOP_INNER_SCRIPT_CHUNKS / PRELOOP_INNER_SCRIPT_BYTES Count and decoded size for that script.
PRELOOP_INNER_SCRIPT Legacy whole-script value. Still honoured by the artifact wrapper so an old control plane can drive a new image.

A new control plane can drive an old image: small prompts still set AGENT_PROMPT, and the wrapper still accepts a whole PRELOOP_INNER_SCRIPT. An old control plane can drive a new image the same way. Custom images that read the prompt themselves should prefer AGENT_PROMPT_FILE after the launch script's materialization block, or reassemble PRELOOP_AGENT_PROMPT_* the same way (base64 -d, then wc -c against _BYTES). Do not require AGENT_PROMPT for prompts above 64 KiB.

OpenHands (the default agent_type) uses this prompt transport on Docker and Kubernetes. Gemini and OpenCode feed the materialized file to the CLI on stdin, so the prompt is not an argv element (issue #692). Residuals that still expand the file into argv are aider (--message "$(cat ...)"), OpenHands (-t "$(cat ...)"), and Codex nudge/recovery ("$(cat ...)" as a positional). Those are not part of this contract.

Model stream recovery

Hosted Docker/Kubernetes agent scripts and supported private Docker launches keep provider retries separate from session recovery. Codex uses four request retries and five stream retries (explicit for the gateway provider, matching its native defaults); OpenCode 1.18.29 already retries a model request up to five times; Gemini enables transport retries with four total chat-model attempts. After a transient CLI failure, the container can resume the captured parent conversation twice, with two- and four-second backoffs and a 600-second timeout per resume (forced termination after another five seconds). The execution's overall timeout still applies. These retry layers can multiply upstream attempts; they are bounded and do not guarantee recovery from a sustained outage.

Recovery requires an explicit captured session identifier. It does not start another conversation if capture fails, and it skips cancellation, terminal authorization/quota failures, and a terminal completion report written during this invocation. A content/stat fingerprint captured before the CLI starts preserves old result.json evidence from restored workspaces without mistaking it for completion of the new turn. The continuation prompt preserves conversation/tool results and asks the agent to inspect uncertain external actions before writing again. Initialization and the container's publication block execute once; model-controlled effects are not guaranteed exactly once. PRELOOP_STREAM_RECOVERY, PRELOOP_STREAM_RECOVERY_RESULT, and PRELOOP_STREAM_RECOVERY_UNAVAILABLE remain visible in agent logs. OpenCode/Gemini terminal JSON errors override a misleading zero CLI exit; truncated output without a terminal completion cannot count as success.

The separate private native-host execution profile supports only Cursor and retains its existing no-resume contract; OpenCode, Codex, and Gemini are not accepted on that path. Gemini is not currently a supported private Docker runner harness.

PR-comment resume

create_pull_request (and GitLab merge-request creation) records the opened HTML URL and source branch on flow_execution.result (flow_pr_binding). When a flow listens to both issue events (issue_labeled or issue_opened) and comment_created, a later human comment on that PR starts a new execution of the same flow with _resume in the trigger payload. The container clones and pushes the existing PR branch. Unmatched comments do not start a run. Native CLI --resume is a separate follow-up.

Wrapper-opened PRs bind too. Not every PR comes from the MCP tool: when git_clone_config.create_pull_request is set, the container's post-execution step opens the PR itself with a plain curl. Title and body are JSON-encoded in the container (json.dumps) so quotes and newlines in a commit message or a multi-line pull_request_description cannot produce a 400. The wrapper prefers pr_title / pr_body from /workspace/result.json, then interpolated git-config fields, then a flow-attribution fallback: an execution link plus the last commit body, or [Preloop] {flow_name} with a **Commits:** list when more than one commit landed. The create response is written to /workspace/evidence/pr.json, with a lookup by head branch (GitHub) or source branch (GitLab) when the PR already existed, and one log line PRELOOP_PR_OPENED {"url": ..., "branch": ..., "provider": ...}. The orchestrator reads that line off the log stream (rescanning the output summary if a stream reconnect lost it) and records the URL and branch on the execution result at terminal status, next to the MCP-recorded binding.

Implement, review, resume. The PR reviewer flow posts as the Preloop bot, and bot-sent events are dropped by the loop guard, so its findings never reached the implementer. A comment_created event from a known Preloop identity is now allowed through that guard only when the body carries the reviewer marker <!-- preloop-review:flow-id:<id>[:severity:<S>] -->; every other bot comment stays dropped. Three guards bound the resulting loop:

  • Self-loop: a comment whose marker flow id names the receiving flow (its UUID or its name slug) never resumes it, so a flow cannot restart itself on its own output.
  • Cap: max_resumes_per_pr (flow-level, agent_config.max_resumes_per_pr, default 5) limits how many resumes one PR can start. The running count lives on the opening execution's result (resume_count) and each resume carries its resume_index; comments past the cap are logged and skipped.
  • Queue one: a correlated comment that arrives while a run for the same PR is still going does not start a competing execution. It sets pending_followup (with pending_followup_comment_url) on that running execution; further comments during the same run are coalesced onto the same flag. When the run reaches a terminal status (succeeded or failed), the orchestrator clears the flag and starts exactly one follow-up resume for the PR.

CI-failure resume

Failing GitHub check_run, check_suite, and workflow_run events on a PR this flow opened resume the same way as a review comment, with {{execution.ci_failure}} filled in. The skip/resume filter is GitHub-only: existing GitLab pipeline / job flows keep firing on every event. GitLab payload extractors stay in flow_ci_feedback so an opt-in can reuse them later.

Matrix / Batch Fan-Out

One flow definition can drive an agent-harness × model evaluation grid without cloning the flow per combination:

  • Trigger: POST /api/v1/flows/{flow_id}/trigger reserves the top-level matrix key in the trigger body. When present, it must be a list of up to 25 {"agent_type"?, "ai_model_id"?} cells; the trigger fans out to one execution per cell (an empty object {} runs the flow defaults). Validation is all-or-nothing (allowed keys, agent types from the agent factory registry, account-visible models), and all execution rows are committed before any cell is dispatched. Note for existing users: matrix is now a reserved key in trigger bodies, it is stripped before template-variable resolution and never reaches {{trigger_event...}} placeholders; rename any pre-existing matrix field in custom trigger payloads.
  • Batch identity: all cells share a batch_id (indexed column on flow_execution). Per-cell overrides are persisted on each execution under the reserved _matrix key of trigger_event_details, making every cell self-describing. All paths that (re)build an agent executor (initial run, resume, background monitor, crash recovery) resolve the effective agent type through resolve_execution_agent_selection (matrix cell, then routing record, then flow default), so an interrupted cell is always handled by its own harness rather than the flow default.
  • Observation: GET /api/v1/flows/batches/{batch_id}/executions lists a batch (account-scoped, sorted by matrix index) with a rollup of status counts, tokens, tool calls, and estimated cost, so an eval matrix can be observed as a unit.
  • Response shape: non-matrix triggers are wire-identical to before; matrix triggers return batch_id plus per-cell execution references.

Execution lineage

Executions expose parent_execution_id, root_execution_id, and delegation_depth in detail and lightweight list responses. Existing and root runs have null parent/root IDs and depth 0. Later delegation writers supply these values at creation; this storage contract does not start child runs.

The parent is an indexed self-reference with ON DELETE SET NULL, so removing a parent preserves its children. The indexed root ID has no foreign key and remains a grouping label after root deletion; recorded depth is also preserved. A whole tree consists of the root row plus executions whose root ID points to it. CRUDFlowExecution.get_children requires an account, returns direct children only, and orders them by start time then execution ID. The list query loads the lineage fields and existing parking timestamps with the row so serializing a page adds no per-row reads.

Label-based model routing

A flow can optionally store ordered routing rules in agent_config.model_routing (no extra column). Each rule has a stable id, labels.any and/or labels.all against the issue's current labels, and an account-owned ai_model_id plus compatible agent_type. The first matching rule selects the model and harness for that execution. If none match, or the key is absent, the flow's selected model and harness are used. Examples in docs use operator-defined labels such as documentation or bug; there is no built-in taxonomy.

The controller writes the chosen rule or default onto the execution under reserved _model_routing after validating ownership and _model_usable_for_agent. Webhook bodies, tracker payloads, and authenticated trigger JSON cannot supply ai_model_id, _matrix, _model_routing, or _resume as authorized overrides. Retries pin from the persisted source execution (retry_of_execution_id); native continuation pins from a controller-loaded prior execution after account and lineage checks. A live rule or default change does not swap the model mid-conversation. Every new execution records its selected defaults, including flows without rules. Durable feedback copies this identity before reserving a repair turn. A missing legacy identity or changed native selection blocks continuation and requires an explicit new execution; no automatic model handoff occurs. New matrix cells also materialize their effective defaults. Authorized matrix eval cells still win over routing. Account budgets still apply. Operator guide: model routing.

Eval / observe runs (structured result artifact). The Observe / Eval preset (backend/presets/003-observe-eval.yaml) establishes a run → capture → serve contract: the agent writes its final report to /workspace/result.json (preloop.eval.result/v1: status, summary, metrics, checks, optional artifacts), the orchestrator captures it first-class on every terminal path (success, failure, stop, timeout) via the Docker archive API (no log scraping or sentinels) and persists it to flow_execution.result (JSONB, size-capped; malformed or unfetchable artifacts are recorded as wrapped {"error": ...} objects so failures stay visible). It is served on GET /api/v1/flows/executions/{id} and GET /api/v1/flows/executions/{id}/result (404 when the run reported nothing); list payloads exclude it. The artifact is also an active success-confirmation channel alongside FLOW_EXECUTION_SUCCESS: success confirms the flow; error, failed, and failure fail it; and eval fail remains a subject verdict rather than a flow failure. Preloop transports and meters the artifact; scoring is owned by the customer's own verifier. On the Kubernetes backend a completed pod's filesystem is unreachable through the API, so the agent script is wrapped to emit result.json (and the evidence pack, see below) into the pod's log stream as base64 between structured PRELOOP_ARTIFACT_* marker lines; the runner parses the emission back out of the pod log with the same validation and error surfaces as the Docker archive path, and the marker lines are filtered out of operator-facing logs and summaries.

Completion-confirmation recovery ladder. An exit-0 run must confirm success through one of the two channels above (the exact-line FLOW_EXECUTION_SUCCESS sentinel, or a recognized result.json status/verdict); that requirement is unchanged for generic flows. When neither channel fired, the agent script has usually already reminded itself in place (see below) and the orchestrator walks a recovery ladder before failing closed: (1) post-exit log rescan: it refetches the complete runtime logs (docker logs / K8s pod logs, no tail limit) and rescans for the sentinel, armed at the PRELOOP_AGENT_EXEC_START marker, covering the edge where a late stream reconnect loses the log tail (logged as the post_exit_log_rescan milestone); (2) confirmation nudge: it re-invokes the agent once, at most one round per execution across retry attempts, with a minimal follow-up prompt (head of the original prompt plus a quoted output tail) asking it to confirm via the originally instructed channel or state failure plainly (logged as the confirmation_nudge_used milestone with its outcome). The nudge is side-effect free (git_clone_config and custom_commands are stripped) and only runs on runtimes that opt in via AgentExecutor.supports_confirmation_nudge (currently codex and opencode); all other runtimes skip straight to fail-closed. Two settings bound the round: FLOW_CONFIRMATION_NUDGE_MAX_TOKENS (default 4096) is a hard output-token ceiling (a larger limit inherited from the flow's model is clamped down; a tighter one is respected) and also bounds the embedded prior-context excerpt at ~4 chars/token, and FLOW_CONFIRMATION_NUDGE_TIMEOUT_SECONDS (default 300) bounds its wall clock. If the ladder ends without confirmation, or the nudge states failure, the execution is marked FAILED with the existing missing-confirmation message, which says so explicitly when the agent was already reminded in its own container.

In-place completion nudge (the cheap rung). no_confirmation was the largest single failure class on staging (33 of 100 failed executions, median duration 0.8 minutes): runs that exited 0 and simply never used a completion channel. The cheapest sound way to get the confirmation is to ask the agent that is still standing there, so the generated agent script now checks the completion contract itself and, when nothing confirmed it, re-invokes the SAME harness on the SAME session in the SAME container and workspace (opencode run --continue, codex exec resume --last) with a short reminder: do not redo or extend the task, do not take new side effects, write /workspace/result.json and print the sentinel for success, or report the failure plainly, then stop. The reminder's output is judged by exactly the same contract, because it lands on the same stdout and writes the same result artifact. Safety comes from placement rather than policy: the block is emitted before the container's post-execution git block and runs only when the harness exited 0, so a nudge can never re-run a push or a PR creation; it runs at most once; and its exit code is discarded, so a failed reminder leaves the run exactly where it was. Three marker lines make the round observable (PRELOOP_COMPLETION_NUDGE, PRELOOP_COMPLETION_NUDGE_RESULT exit=<n>, PRELOOP_COMPLETION_NUDGE_UNSUPPORTED <runtime>); the orchestrator reads them from the live stream and again from the post-exit log refetch, publishes them as the completion_nudge and completion_nudge_result timeline events, and then stands down from its own second-session nudge (skipped_inplace_nudge_used), which would be a second model call for the same answer from a session that knows strictly less. It also stands down when any action is already on the timeline (skipped_actions_recorded): a re-invoked model that decides to finish the job would repeat that posted comment. Runtimes that opt in declare AgentExecutor.supports_inplace_completion_nudge (currently codex and opencode). Settings: FLOW_COMPLETION_NUDGE_ENABLED (default true, flowExecution.completionNudge.enabled) is the fleet-wide kill switch, FLOW_COMPLETION_NUDGE_TIMEOUT_SECONDS (default 300, floor 30) bounds the round.

Wider completion signals for runtimes that cannot be resumed. Gemini, Aider, OpenHands and remote runners have no resume mechanism, so there is nobody left to ask. For those (and only those: the check is a strict identity test against supports_inplace_completion_nudge is False, so an unknown capability keeps failing closed) a non-empty result.json that the agent actually wrote is accepted as the completion signal even when its vocabulary is not one Preloop recognizes, logged as the completion_signal_accepted milestone with signal: result_artifact_present. An explicit failure status is decided earlier and still fails the run, and a runtime that was asked directly and declined to confirm never reaches this rung.

No-progress guard (a run that never edits anything). Some implementation runs read, plan, summarize and end with an empty workspace, having spent the full budget on nothing. Two rungs catch that, both driven by one piece of evidence: the workspace itself. (1) Live, on the poll loop: when agent_config.no_progress_after_seconds is set and that many seconds have passed with the checkout provably clean (git status --porcelain empty and no local commit ahead of the remote, probed in the running container at most once a minute), the orchestrator delivers ONE short reminder into the session that is still running, through the same resume mechanism the in-place completion nudge uses (codex exec resume --last, started detached so the loop is not blocked), logged as the no_progress_nudge milestone and published as a timeline event. After a further no_progress_grace_seconds (default 600) still clean, the run is stopped and classified agent_no_progress (no_progress_stop). A probe that cannot answer (Kubernetes, an unreadable checkout) returns "no answer" and never "clean": a stop always rests on a probe that actually ran, and a run that has any diff is never nudged and never stopped by this guard. Runtimes without an in-place resume get the stop without the reminder. The key is null by default, which disables both rungs. (2) Terminal: when the agent itself reports failure through result.json and the container's post-execution git block found no commit to push (PRELOOP_NO_COMMITS <branch>), the failure is classified agent_no_progress rather than the generic agent error, so "tried and could not" is separable from "never started". agent_config.retry_on_no_progress ({"enabled": true, "ai_model_id": ..., "reasoning_effort": ...}, off by default) then creates exactly one retry through the normal manual-retry path with retry_of_execution_id set, optionally escalating to a stronger model; a retry is never itself retried.

Per-flow timeout budget. flow.timeout_seconds (nullable, 60..86400, on the create/update API and settable from a preset YAML) is the wall-clock budget for one execution of that flow; NULL keeps the deployment default FLOW_EXECUTION_MAX_WAIT_SECONDS (3600, flowExecution.maxWaitSeconds). The orchestrator resolves and clamps it once per run, records it on the agent_monitoring_started milestone (timeout_seconds, timeout_source), enforces it in the monitor loop, and on expiry writes agent_execution_timeout and a message that names the budget that ran out ("this flow's timeout budget" vs "the default timeout budget"), still classified timeout. This is what separates "genuinely long" from "stuck": with one global ceiling, all 7 staging timeouts sat on exactly 3600 seconds and said nothing about which was which. A PR review that should finish in minutes (002-pull-request-reviewer.yaml, 1800) and a release audit that legitimately runs for hours (006-release-security-audit.yaml, 7200) now carry their own budgets.

Stream idle bound. A custom-provider Codex call (including the Preloop gateway) waits stream_idle_timeout_ms for the next byte of a streamed response before it treats the stream as dropped, then reconnects up to stream_max_retries (5) times inside the same call. agent_config.stream_idle_timeout_seconds (30..3600, validated on create/update) sets that wait per flow; unset keeps 600. Either way preloop.services.stream_stall.resolve_stream_idle_timeout_seconds caps it at half the run's timeout budget (the orchestrator passes flow_timeout_seconds in the execution context), because a wait as long as the budget can never fire: the run is stopped before Codex reconnects and the log never names the stall. The cap only changes an unconfigured flow when its budget is under 1200 seconds (a 900 second flow gets 450). Native OpenAI writes no provider block and keeps Codex's built-in wait. The Codex script also sets RUST_LOG so codex_core::responses_retry logs the reason for each reconnect (idle timeout waiting for SSE); without it the Reconnecting... n/m lines carry no reason. When a run times out and its log shows an idle signal with no model output (codex, exec, thinking, tokens used) after the last one, the message says it timed out "while waiting on a silent model stream", failure_category is model_stream_idle, result.stream_stall carries the evidence (idle_reconnects, retries_exhausted, stream_idle_timeout_seconds, last_signal), and an agent_stream_stalled milestone is written. A run that stalled, recovered and then ran long stays timeout (issue #872).

Hosted-monitor concurrency. Each flow-execution worker process may babysit FLOW_EXECUTION_MAX_INFLIGHT hosted monitors at once (default 10, flowExecution.maxInflight). The cap is a process-wide semaphore shared by execute_flow and resume_flow_execution. Other worker pools stay serial: they were not sized for fan-out. Helm injects a dedicated flowExecution.databasePool (size 10 / overflow 4) on the flow-execution pool so those short-lived sessions are not squeezed through the generic worker 2+4 pool.

Per-account admission cap. Admission is also bounded per account, so one account cannot fill every worker. FLOW_EXECUTION_MAX_RUNNING_PER_ACCOUNT (default 5, Helm flowExecution.maxRunningPerAccount) is enforced inside claim_execution. The cap bounds hosted compute only: an execution assigned to one of the account's private runners is bounded by that runner's own capacity, so it is neither counted nor held back. An account may override the cap through account.meta_data["flow_execution_max_running_per_account"]. A refused execution stays PENDING with queued_reason=account_concurrency_cap and its message is nacked for later redelivery. Independently, a flow gets at most one active execution per tracker object (issue, pull request, or merge request): a later matching trigger is recorded as skipped rather than queued. Comment and CI deliveries are exempt so they can still feed a live run. A labels condition on a labeled/unlabeled delivery tests the label the event carries, not the issue's full label list.

Stale-claim reaper. Every FLOW_EXECUTION_RECLAIM_INTERVAL_SECONDS (default 30, Helm flowExecution.reclaimIntervalSeconds) the reaper re-publishes active executions that are unclaimed or whose claim heartbeat went stale. That is the deploy-handoff safety net: an execution whose owning worker died is adopted by another worker inside one stale window. Three bounds keep it from becoming a storm, because "nobody has claimed this" and "nobody has a free slot for this" look identical from the database. (1) The pass runs under a session-level advisory lock (pg_try_advisory_lock) pinned to one connection for the whole pass, so one replica reaps per interval instead of all of them; losing the lease is not an error, the loser skips the pass and Postgres releases the lock if the holder's connection dies. (2) Each execution carries redispatch_count and last_redispatch_at: the gap between two re-dispatches of the same execution doubles from the reclaim interval up to FLOW_EXECUTION_REDISPATCH_BACKOFF_MAX_SECONDS (default 900, Helm flowExecution.redispatchBackoffMaxSeconds), so an hour of being unclaimed costs a handful of publishes rather than 120. A successful claim resets both, so a run that queued for an hour and then lost its owner is still recovered at once. (3) Before publishing, the worker asks JetStream whether flow tasks are already queued undelivered; when they are, executions that still need a container are left queued rather than republished into a queue nobody can drain. Executions that already have an agent session bypass the capacity and per-account filters, because an unmonitored container is worse than being one over a limit. Each pass logs exactly one line: candidates, re-dispatched, skipped_backoff, skipped_account_cap, skipped_no_capacity, failed.

Terminal notifications (flow.notifications). A flow can opt into posting a tracker comment when an execution ends, instead of asking the agent to ping the triggering issue from the prompt. on_success.comment_on_trigger_issue posts a short "PR opened: \<url>" comment when the run recorded a pr_url. The orchestrator posts it from the terminal path (including resume), and a notification failure never rewrites the execution status. The console offers the option only where it can apply: the flow opens the pull request itself (git_clone_config.create_pull_request) and the trigger is about an issue (a tracker trigger with an issue_* or comment_* event).

Failed executions always appear as console attention items of kind flow, and there is no failure comment: on_failure.comment_on_trigger_issue (removed 2026-09) and on_failure.attention_item are parsed and ignored so flows stored before the removal still load. notifications is a JSONB column, so nothing is migrated; a save from the console writes the blob back without the on_failure block.

Evidence packs. Audit-style flows (backend/presets/004–007) write a human-readable evidence pack under /workspace/evidence/. Two transports move that pack to the control plane. Direct upload (FLOW_ARTIFACT_DIRECT_UPLOAD) gives hosted containers and private Docker runners an execution-scoped PUT to the encrypted artifact store (kind=evidence). Kubernetes logs then carry status markers only, including when the upload fails: there is no plaintext fallback. The legacy transport is the default. Docker copies the directory through the engine archive API and does not use the log channel, so FLOW_EVIDENCE_LOG_PLAINTEXT does not change Docker capture. Kubernetes, with the plaintext switch left at its default true, emits a size-capped base64 block (MAX_EVIDENCE_ARCHIVE_BYTES, 2 MiB) of result.json, the evidence pack, and the workspace snapshot into the pod log. That default is an exposure window: base64 is not encryption, and anyone who can read retained pod logs can read the artifacts. Set FLOW_EVIDENCE_LOG_PLAINTEXT=false to refuse that channel. Without an upload token the wrapper fails closed (markers unavailable / skipped with reason plaintext_disabled, no artifact bytes). The receipt is failed with error plaintext_disabled, which means unavailable by policy, and result capture reports a missing result rather than a decoded payload. The switch does not cover pod-spec access or ordinary agent stdout. Encrypted log transport with per-execution keys is a separate decision tracked in issue #268. GET .../result and GET .../evidence-status report the persisted receipt (kind=evidence) without decrypt; download verifies digest and tenancy and reports available / missing / expired / failed distinctly. Packs are signed at capture with the account Ed25519 key; the signature lives beside the archive so the content-addressed digest does not change. See evidence-storage.md. This is operational retention, not object-lock.

Workspace persistence (snapshot, restore, retention). An execution that fails after the agent committed used to lose the work: on Docker the named volume agent-workspace-<execution_id> survived by accident with no way to find it, and on Kubernetes the emptyDir died with the pod. Every hosted run now captures a size-capped tar.gz of /workspace on every terminal path, success or failure, next to the evidence pack. .git is kept whole (the unpushed commits are the point) and a git bundle --all of every repository is written into it first as an independent second copy of the commit graph; regenerable caches (node_modules, .venv, __pycache__, and friends) are excluded. The cap is WORKSPACE_SNAPSHOT_MAX_BYTES (default 512 MiB) and it is enforced inside the container: the tar stream is piped through head -c, so an oversized workspace never writes more than the cap to disk, never crosses the runtime boundary, and is skipped with a logged PRELOOP_WORKSPACE_SNAPSHOT_SKIPPED size_exceeds_limit reason. Docker builds the archive in a short-lived helper container mounted on the workspace volume (the agent container has already exited and cannot run tar); Kubernetes emits it through the same PRELOOP_ARTIFACT_* log channel as the evidence pack, which caps it at 2 MiB there, so larger Kubernetes workspaces are reported as skipped until snapshots move to object storage. The archive is stored on flow_execution.workspace_snapshot (BYTEA, excluded from the execution schemas) and served by GET /api/v1/flows/executions/{id}/workspace. A correlated resume (a PR comment on a PR this flow opened) loads the prior execution's snapshot, unpacks it into the new Docker workspace before the entrypoint runs, and the init script then skips the clone and does fetch + checkout + merge --ff-only of the PR branch inside the restored repository, so unpushed commits are continued rather than redone. The decision is made in the container ([ -d <repo>/.git ]), so a restore that did not land silently falls back to the normal clone; Kubernetes always falls back, because an emptyDir cannot be seeded before start. Retention is a decision rather than a leak: an hourly janitor task (cleanup_flow_workspaces) clears snapshots and removes agent-workspace-* Docker volumes older than WORKSPACE_SNAPSHOT_TTL_HOURS (default 24; 0 means delete on the next pass). Private runners keep their workspaces on the operator's own machine and never upload them.

Codebase setup (git_clone_config.setup_commands). An implementation agent that cannot run the real test suite is guessing, and "the DB-backed tests need a Postgres this container does not have" is not an agent failure. setup_commands is a list of shell strings run inside the container after the clone or workspace restore and before the agent starts, from the first repository's clone path. All output is captured to /workspace/evidence/setup.log (inside the evidence pack, so it is retrievable even when the workspace snapshot was too large to keep); the first failing command stops the block, prints PRELOOP_SETUP_FAILED exit=<n> plus the log tail, and fails the execution with the dedicated setup_failed failure category, so a broken environment never reads as a broken agent.

git_clone_config:
  enabled: true
  setup_commands:
    - pip install -r requirements/dev.txt
    - docker run -d --name pg -e POSTGRES_PASSWORD=preloop -p 5432:5432 postgres:16
    - until pg_isready -h 127.0.0.1 -U postgres; do sleep 1; done
    - echo 'DATABASE_URL=postgresql+psycopg://postgres:preloop@127.0.0.1:5432/postgres' > /workspace/.env

Setup commands run in a subshell (set -e; cd <clone path>), so export, cd, and other shell state do not reach the agent process. Persist anything the agent needs in a file (as above) rather than relying on the environment.

Services are the caveat. A docker run inside a setup command needs a runner that is allowed to touch a Docker socket: on a private runner that means the trust flags the operator opts into explicitly, and on the hosted runners it needs a per-execution service sidecar, which is the next phase of this work (a declarative services: block with a Docker network on the Docker runner and pod sidecars on Kubernetes). Until then, keep setup commands to what a plain container can do: install dependencies, build, seed fixtures, start in-process services.

Publication gate (git_clone_config.verification). A prompt that asks an implementation agent to run tests is a request, not a contract: the agent's result.json is a claim about what it did, and the publisher has no way to tell a tested change from an asserted one. The gate turns verification into something the runner owns. A flow opts in by setting verification.mode: "gate" with a trusted test profile: a versioned (v1) mapping of changed-file patterns to required checks (always inexpensive hooks required on every diff, rules matched against the diff against the PR base, and an unknown_default applied to every unmatched changed path so a mixed documentation/code diff cannot hide unknown impact). The profile ships with the flow configuration, never inside the target repository, so the agent cannot narrow required checks or edit the profile to bypass the gate within its own PR. It may propose additional checks through proposed_checks in its result artifact; only the profile decides what is required.

After the agent exits and commits, before the push and before pull-request creation, the post-execution block runs a runner-controlled verifier inside the container: it derives the changed files from the merge base, selects the required checks with the same selection implementation the runner-side contract uses (preloop.utils.verification_selection, embedded verbatim into the verifier script so the two can never drift), executes each check with PRELOOP_DISABLE_TELEMETRY=true, a per-command timeout, and the PRELOOP_VERIFY_BASE/PRELOOP_VERIFY_HEAD range contract, and records evidence: commands, exit codes, per-check logs, an environment digest, the profile version, and the exact commit SHA and tree hash it verified. The legacy verifier always runs required checks: files in the agent sandbox cannot safely certify a cached pass. Reuse requires an authenticated runner/controller cache bound to the commit, tree, complete profile and environment. Selection reasons are recorded with the evidence, so operators can read why each check was required.

The verdict is fail-closed. Publication (push and PR creation) runs only when every required check passed on the exact commit and tree being published and the tracked tree is clean; a failed check denies publication as verification_failed, a check that could not run (missing interpreter, exhausted gate budget) as verification_blocked, and anything uncertain (no verdict file, a crash, a commit that appeared after the gate) denies too. A denial exits non-zero, which fails the execution with its commits and evidence kept recoverable; nothing unverified leaves the workspace. There is no draft-publication exception in this mode. The orchestrator captures the compact evidence from the PRELOOP_VERIFICATION marker line and stores it under verification in the execution result (with source: runner), while an agent-authored verification key in result.json is demoted to verification_reported: the implemented status (status) and the verification status (verification.status) are two different, separately auditable fields. Evidence logs land in /workspace/evidence/verification/ inside the normal evidence pack. The full credential boundary for test processes (no write credentials) is a separate workstream (issue #432).

Flows saved before the gate existed resolve to mode: "off" through effective_verification_policy() (an explicit, visible "ungated, because no policy is configured" rather than a silent default), while an explicitly configured malformed policy blocks publication; it never silently downgrades to off. The flow API also validates the profile shape on save.

Transient Failure Handling

A flow run crosses two systems that fail independently and briefly: the Kubernetes API that creates the agent Job, and the model provider behind the gateway. Neither should end a run on its own. Three bounded, jittered retry layers cover them, and every terminal failure is stamped with a category so the remaining failures can be counted instead of read.

Layer 1: agent Job creation (ContainerAgentExecutor). The Job name is derived from the execution id, so a second session for the same execution collides with the first. That is exactly what the run-level retry (layer 3) and the confirmation nudge do, and on staging it turned a recoverable provider blip into Failed to start agent Job: (409) Conflict: 8 of 100 failed executions. Two changes: each additional session now asks for its own name (agent-<execution-id>-a2, …-nudge; the first attempt keeps the historic unsuffixed name so in-flight runs stay addressable), and Job creation is wrapped in a bounded retry. On a 409 AlreadyExists the existing Job is read rather than blindly recreated: if it belongs to this execution and is still live it is adopted (an idempotent create, e.g. a duplicate dispatch, must not fail the run); if it belongs to this execution and has finished it is deleted with propagationPolicy=Background, waited for, and recreated, but only while a creation attempt remains, because deleting a leftover nothing will recreate would cost its logs and fail the run anyway; if it belongs to a different execution, or carries no preloop.execution_id label to prove it is ours, the run fails immediately as runner_conflict rather than adopting or deleting someone else's agent. 429/5xx from the API server are retried. Bounds: AGENT_JOB_CREATE_MAX_ATTEMPTS (default 3, flowExecution.agentJobCreate.maxAttempts) and AGENT_JOB_CREATE_RETRY_BASE_SECONDS (default 0.5, exponential with equal jitter). Set max attempts to 1 to disable the layer: a 409 then fails the run as runner_conflict without touching the conflicting Job.

Layer 2: upstream model calls (model gateway). Inside one client request, the gateway retries an upstream failure that happened before any token was forwarded to the client: connection errors, 429, 5xx, and a stream that dies during prefetch. A partially streamed response is never retried, because the client has already seen output. Retry-After is honoured and capped so a hostile header cannot pin a worker. Bounds: MODEL_GATEWAY_UPSTREAM_RETRY_MAX_ATTEMPTS (default 3), MODEL_GATEWAY_UPSTREAM_RETRY_BASE_SECONDS (default 0.2), MODEL_GATEWAY_UPSTREAM_RETRY_AFTER_CAP_SECONDS (default 8), under environment.modelGatewayUpstreamRetry in Helm. The number of retries spent on a request is recorded on the usage row (meta_data.upstream_retries) and published to the execution timeline as retried: n on the model_gateway_call event, so a slow run that was quietly rescued is visible rather than mysterious.

Layer 3: whole-attempt retry (_run_agent_with_retries). When the agent process itself dies, the run is retried from scratch only where that is provably safe; otherwise it fails with a category. The safety boundary is the container's post-execution block (git push, PR/MR creation), which the entrypoints run only on exit 0. So an attempt is retried only when all of: the agent exited non-zero (no push/PR happened), no actions were recorded on the timeline, and the failure was classified transient by the executor's analysis of the full logs. An unknown exit code counts as unsafe. Bounds: FLOW_EXECUTION_MAX_ATTEMPTS (default 2) and FLOW_EXECUTION_RETRY_BACKOFF_SECONDS (default 15, doubling). Retries are never silent: each one writes an execution_retry_scheduled milestone, a timeline warning, and an execution_retry update; exhaustion writes execution_retries_exhausted.

Failure categories. Every terminal non-success execution stores a coarse failure_category (flow_execution.failure_category, indexed; exposed on the execution list and detail schemas) from a closed vocabulary: runner_conflict, runner_error, model_transient, model_auth, provider_billing, budget_exceeded, model_quota, model_config, no_confirmation, agent_no_progress, publication_missing, setup_failed, verification_failed, verification_blocked, tool_error, agent_error, model_stream_idle, timeout, cancelled, unknown. It is derived once at failure time by preloop.services.flow_failure_category.derive_failure_category from the best evidence available, in order: a category named by the raising code (e.g. AgentStartError), structural message shapes that identify the failing layer regardless of provider noise (runner conflict, runner error, timeout, cancellation, missing completion confirmation, publication-gate denial), then the agent executor's failure analysis of the full logs (which shares the upstream_errors taxonomy with the gateway), then provider-message patterns. Anything unmatched is unknown rather than being folded into agent_error: a rising unknown share is the signal to extend the module. error_message remains the human-readable detail; the category is what you group by. Existing rows are not backfilled: categories are derived from live failure context, and guessing them from historical prose would launder a heuristic into stored data.

The legacy verifier runs in the same sandbox as the agent. Its producer label, files and log markers are diagnostic data, not an authenticated attestation. Agents or repository commands can reproduce a marker. Credential-isolated publication must use a separate controller/runner-host verifier that observes command completion outside that sandbox on an immutable commit artifact. Never promote PRELOOP_VERIFICATION JSON directly to trusted publication authorization.

Isolated publication (integration preview)

git_clone_config.publication_mode defaults to legacy for saved-flow compatibility. Legacy mode executes publication in the agent container and shares tracker write credentials with that runtime; it is not a credential isolation boundary. Operators must explicitly migrate a flow to isolated. Hosted execution has a controller-owned verification adapter. Private runner publication additionally requires the authenticated runner-host protocol. An isolated execution without controller-issued evidence fails closed before issuing write access.

Isolated flows require verification.mode: gate, a nonempty trusted profile, and verification.image set to a digest-pinned generic toolchain image. That image must provide Python 3, Git, a shell, and the dependencies needed by its configured checks. Verification does not reuse the agent's virtual environment, node_modules, writable caches, or workspace. A check may perform bounded setup using dependencies already available in the image, including local package caches and service binaries; it cannot download dependencies during verification. This is a generic toolchain contract, not a requirement to bake an application or its database into the harness image. Missing dependencies fail the check and leave the captured work recoverable.

The controller resolves the exact base commit before starting the agent. After committing recovery artifacts to durable storage it deletes the owned agent runtime and confirms its absence, including residual Kubernetes pods. A missing or failed recovery capture retains the original runtime and blocks publication. It derives changed paths from the frozen bundle, selects relevant checks from the trusted profile, and runs each check in a fresh credential-free checkout of the same head. Docker has no network, host mounts, or host namespaces. Kubernetes disables service-account automount and applies a deny-all NetworkPolicy. Because policies are additive, existing permissive policies matching the verifier's labels block startup. Hosted clusters must enforce NetworkPolicy and prevent concurrent policy or admission changes from weakening the verifier namespace's isolation. The controller observes process exit codes and confirms every verifier runtime was removed before minting writer credentials. Check diagnostics retain a bounded scrubbed output tail, per-check elapsed time, and Docker log rotation limits runtime log growth. If Kubernetes deletion cannot be confirmed, the deny-all NetworkPolicy remains with the execution's verifier label for operator recovery; removing it first would restore network access to residual pods. Log markers and result files never authorize publication.

The isolated path binds a single repository and its base/target branches from account-owned flow/project records. It never accepts a webhook clone URL as publication authority. GitHub App installation tokens are minted for exactly that repository: contents:read for the agent, then contents:write and pull_requests:write for the publisher after verification. Stored PATs and GitLab publication are rejected in this mode until a broker can enforce their scope and lifetime. The standalone metadata/provider client supports both GitHub and GitLab. Private runners require an authenticated current capability advertisement confirming protocol v1 and a locally available digest-pinned trusted helper image. Older runners cannot receive an isolated lease. Private source stays on the runner; the controller receives only bounded manifests, check outcomes, and the provider receipt.

Private verification is an ordered, nonce-bound handshake. The runner removes the agent and residual volume writers, freezes the bundle in independent storage, runs the exact controller-selected checks, and confirms verifier removal. Only then does the controller issue a repository-scoped writer to the trusted publisher helper. Agent result files and ordinary completion messages cannot advance publication. Both helper and controller revoke the writer; already-invalid token responses make repeated revocation safe.

Recovered private monitoring restores the protected policy and accepted receipt without relaunching the agent or replaying credentials. Queued isolated executions fail closed on worker recovery because their original lease cannot be safely replayed; retry explicitly after an eligible runner is available. Private frozen recovery artifacts remain local with a fixed 24-hour retention limit after verification or publication failure. A recovered hosted isolated execution currently cannot reconstruct its original trusted policy snapshot: it fails closed with an explicit diagnostic and retains its original runtime for recovery. It does not silently switch to ungated publication.

After runtime cleanup, the control-plane publisher imports a bounded, self-contained branch.bundle into a fresh bare object store. It never checks out repository code, imports agent Git configuration, or runs repository hooks, filters, credential helpers or shell commands. Each Git child has a clean environment and CPU, memory (Linux), file-size and time limits. HTTPS redirects are disabled. The publisher validates the exact verified head, checks the expected remote SHA and ancestry, then uses an atomic lease to reject concurrent remote changes. It never rewrites unexpected remote history. Provider failures mark publication failed and retain captured recovery evidence; retry reuses the existing branch and PR. Write tokens are revoked on success/failure and expire at the provider-issued deadline if revocation is unavailable.

The controller handoff is VerifiedPublication(execution_id, head_sha, bundle_sha256). It is an internal type, not an agent JSON schema. A trusted verifier adapter must construct it only after verifying the immutable artifact in an environment the agent cannot modify. Agent-written result files or log markers cannot establish this attestation. Hosted adapters construct it only after the complete trusted verification lifecycle. Deploy the control plane separately from agent workloads; never mount its process namespace, filesystem, Docker socket or signing keys into those workloads.

PR descriptions and execution provenance

Repository setup selects git_clone_config.pull_request_template when set, then the conventional GitHub/GitLab default, then the lexicographically first named Markdown template. It writes /workspace/evidence/pr-template.md for the agent to fill. No template uses Summary and Testing. Configured missing, invalid, oversized or escaping template paths fail setup. Tests that did not run remain unchecked. The implementation preset asks for problem, resulting behavior, acceptance evidence, verified commands, limitations and an issue link.

result.json retains #420's pr_title/pr_body fields and aliases. Valid agent text wins per field over configured text and commit text. Titles are one line, at most 256 UTF-8 bytes; bodies are at most 60,000 UTF-8 bytes. Invalid/missing metadata falls back with a diagnostic; no truncation silently changes meaning. All publication sources receive execution attribution. The isolated publisher owns only the preloop:executions HTML-comment region. It upserts trusted initial/repair execution links and published SHAs while preserving human edits outside that region, including metadata-only repairs. Links use PRELOOP_URL and existing authorization-protected console routes; tokens and transcripts are never provenance inputs. Legacy publication adds the current execution block on creation. When an open pull request or merge request already exists for the branch, legacy mode fetches that description, appends the current execution id and head SHA to the owned block when that pair is not already present, and updates only the body. Human prose and the title stay as they were. A malformed owned region or an oversized rewrite warns through PRELOOP_PR_METADATA_WARNING and leaves the owned provenance region unchanged; the independent failure-disclosure refresh still runs. A failed provider update is surfaced and never reported as successful publication. Isolated GitLab publication stays unsupported until a broker can enforce credential scope and lifetime.

Publication acceptance matrix (issue #431). Each cell is delivered (test name) or unsupported by design.

Mode Provider Create Continuation push to an existing PR Metadata-only retry Failure disclosure Human edits preserved Provider failure surfaced
legacy github delivered (TestWritePrPayloadPy.test_commit_fallback_single_commit_includes_execution_link; create script calls upsert_provenance once) delivered (test_github_continuation_appends_record_and_keeps_prose) delivered (test_repeated_continuation_is_idempotent_and_reuses_the_pr, test_missing_metadata_warns_and_keeps_existing_prose) delivered (test_already_pushed_commits_refresh_existing_failure_notice, test_existing_body_preserved_and_notice_idempotent) delivered (test_github_continuation_appends_record_and_keeps_prose) delivered (test_provider_update_failure_is_not_success; a create miss is test_no_url_anywhere_emits_no_marker)
legacy gitlab delivered (same create script, kind == "gitlab") delivered (test_gitlab_continuation_appends_record) delivered (test_repeated_continuation_is_idempotent_and_reuses_the_pr) delivered (same failure-disclosure tests, GitLab payload) delivered (test_gitlab_continuation_appends_record) delivered (test_provider_update_failure_is_not_success)
isolated github delivered (test_provider_create_retry_metadata_update_preserves_human_edits) delivered (same test, repair upsert) delivered (same test: one POST, later upserts only) out of scope (issue #599; the isolated publisher upserts provenance only) delivered (same test) delivered (test_provider_failure_is_observable)
isolated gitlab unsupported by design (flows.md: "Stored PATs and GitLab publication are rejected in this mode until a broker can enforce their scope and lifetime") unsupported by design (same) unsupported by design (same) unsupported by design (same) unsupported by design (same) unsupported by design (same)

Continuation append keeps the first execution record and the most recent 199 repair records (PROVENANCE_RECENT_RECORDS). The 201st continuation still lands (test_append_provenance_keeps_the_first_record_and_recent_199).

The standalone metadata client still accepts a GitLab payload shape. Isolated mode does not: validate_publication_tracker rejects PAT and GitLab credentials before a lease is minted.

Preset synchronization updates uncustomized fields and marks customized saved flows as having an available update. Inspect the effective saved prompt and configuration before expecting template behavior. The publication-mode switch is deliberate and is not silently enabled by updating the prompt.