feat(early-stop): opt-in early stop once armed criteria are decided#14
Open
mohsen-uipath wants to merge 7 commits into
Open
feat(early-stop): opt-in early stop once armed criteria are decided#14mohsen-uipath wants to merge 7 commits into
mohsen-uipath wants to merge 7 commits into
Conversation
added 6 commits
July 9, 2026 15:01
…uardrails (inert) Add the opt-in surface for early-stop-on-criterion and its resolution-time guardrails, all inert at runtime (nothing invokes an interrupt yet): - run_limits.stop_early (master opt-in) and per-criterion stop_when (pass/fail/decided) config fields. - BaseCriterion.live_verdict + live_stop_polarities observability contract; overrides on the two mid-run-observable criteria (skill_triggered's first-engagement policy, command_executed's monotone min/max rules, with a _matching_commands helper shared with _check_impl). - Agent.supports_cooperative_stop capability flag (True on ClaudeCodeAgent). - validate_early_stop(): rejects every unsupported arming (non-Claude agent, no armed criterion, unobservable criterion, unsupported polarity, simulation mode) as a hard error, wired into resolve_all_tasks, plan, and _setup.
…wired) Add the mechanism to interrupt a Claude turn cleanly between SDK messages, still unwired (the orchestrator does not pass should_stop yet — zero behavioral change): - should_stop: Callable[[], bool] | None kwarg on the communicate() ABC; the three non-Claude agents (codex/antigravity/noop) accept-and-ignore it for override compatibility. - ClaudeCodeAgent honors it: the message pump is extracted to _pump_messages (keeps communicate under the statement cap; query still resolved as a module global so test mocks keep working); the stop check runs AFTER dispatch so a watcher observing events on the deciding message stops before the next is pulled. A stopped_early_hit flag drives the finalize ladder (TIMEOUT -> STOPPED_EARLY -> COMPLETED) — max_turns promotion only fires for COMPLETED, and the post-finally timeout raise is gated on timeout_hit, so an early stop returns a clean crashed=False TurnRecord and never raises. - New STOPPED_EARLY member on TurnEndStatus + AgentEndStatus (consumers: renderers format status.value generically; TurnEndStatus(status.value) conversion round-trips; wire is by-value). - Seam tests: stop after the first dispatched message, one AgentEndEvent (STOPPED_EARLY, crashed=False), no raise; should_stop=None/False consume the full stream and finalize COMPLETED.
Activate early-stop-on-criterion: when run_limits.stop_early is armed, an EarlyStopWatcher composed into the agent's event stream evaluates each armed criterion's live_verdict on every tool completion and trips the cooperative should_stop the moment the armed set is decided (pass or definitive fail). - models/results.py: EarlyStopReason (StrEnum), EarlyStopInfo (why/when the run stopped), EvaluationResult.early_stop, and armed_criteria_passed — the armed-subset gate sibling of all_criteria_passed. Exports added. - orchestration/early_stop.py: EarlyStopWatcher (own EventCollector + the stop rule; fail-open on a raising verdict; latched decision; turn/tool/elapsed attribution). - orchestrator.py: build the watcher once in _setup when armed; compose it into the stream regardless of --stream; pass should_stop; populate result.early_stop before check_all; gate an early-stopped run on the armed subset (others advisory) vs the full gate on a completed run. - tests: 28 new (models, watcher stop-rule/fail-open/latching, orchestrator wiring, timeout-beats-stop precedence).
Surface the opt-in early-stop feature end-to-end and lock the live-verdict contract with a lint rule. All render/telemetry-only for unarmed runs, so the default path is behavior-unchanged. - reports_experiment: eval_result_to_task_dict emits stopped_early / early_stop_reason / turns_remaining_at_stop so downstream analysis never confuses a truncated run with a full one. - reports: _runtime_notes_lines adds a per-task early-stop NOTE (reads the task-dict keys), with the "<= N turn(s) avoided" clause when known. - reports_html: header badge + per-criterion "advisory — not gated" markers on the non-armed rows of an early-stopped run. - orchestrator: build_task_event gains EarlyStopped / EarlyStopReason dims on CoderEval.Task.End. - CE025 (live-verdict consistency): a criterion's live_stop_polarities and live_verdict must agree — a non-empty polarity set without an override arms a dead criterion; an override without polarities is unreachable. Scoped to criteria/, base.py exempt. 10 rule tests + 7 report-surface tests. - experiments/early-stop-ab.yaml: smoke vs. e2e flavors from one file via the stop_early boolean per variant. - docs: TASK_DEFINITION_GUIDE (stop_early subsection + stop_when field row), AB_EXPERIMENTS (smoke-vs-e2e recipe + ToC), CLAUDE.md (architecture bullet, criteria-fields mention, tree updates, CE range CE001-CE025).
- docs/tutorials/04-writing-a-task.md: add a "Where to go deeper" pointer to the stop_early / stop_when early-stop docs so the feature is discoverable from the intro tutorial (field detail stays in the Task Definition Guide). - .claude/harness-candidates.md: CE024 (discriminated-unions) and CE025 (live-verdict consistency) are now implemented rules, so the proposed candidates were renumbered off the taken ids to CE026 / CE027 / CE028, with a note that the tests/lint/runner.py id-uniqueness assert is the source of truth.
The five early-stop guardrails were unit-tested against validate_early_stop directly, but nothing asserted the resolution surfaces actually invoke it. Add integration tests driving the real wiring with real task YAMLs: - run surface: resolve_all_tasks raises EarlyStopConfigError for a bad arming (not demoted to a skipped task), accepts a valid armed task, and validates an arming introduced only by the layer-5 -D override (proving the guardrails run after _apply_cli_overrides). - plan surface: plan_command flips its exit code to 1 and names the early-stop config error for a bad arming; a valid armed task still plans clean.
mohsen-uipath
requested review from
akshaylive,
bai-uipath,
tmatup and
uipreliga
as code owners
|
Claude finished @mohsen-uipath's task in 1m 13s —— View job Code Review In Progress
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL flagged the Any and BaseCriterion imports as unused: their only references sat inside the quoted type-alias string tuple["BaseSuccessCriterion", "BaseCriterion[Any]"], which static analyzers do not resolve. Move the alias into the TYPE_CHECKING block unquoted so the usages are real name references. Behavior-neutral: the alias is referenced only from annotations, which are lazy under 'from __future__ import annotations'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Some tests only care about a specific part of a run, not the whole trajectory. Activation tests are the clearest example: today they pin the agent to a single turn so the run ends right after the skill is meant to activate. Teams want to raise that turn limit so the agent can investigate the starting project before recalling a skill — but naively raising it means agents that already activated on turn one keep going, doing unrelated and unpredictable work the test was never scaffolded for. That inflates turns, time, and cost, and adds noise to the result.
The same need shows up beyond activation: we'd like to stop as soon as the outcome is known — either the criterion we care about is satisfied, or a required early step is definitively missed (e.g. the wrong skill loads). No point running the rest when the verdict is already in. This has been asked for a few times, both for activation turn limits and for a general "early-exit on pass or fail".
What we want
An opt-in mechanism to end a run early once a designated criterion is decided — on pass or on definitive fail. This lets us give the agent more turns without wasting them or polluting the result once the thing we're measuring has already happened.
Ideally selectable by flavor, so a single test can serve as both a smoke check (stop after the key step) and an e2e check (run the whole thing), instead of maintaining two near-duplicate tests.
Prior art
coder_eval already has a related mechanism in simulation/dialog mode (
stop_on_criteria_pass) that ends a dialog once criteria pass. It's scoped to dialog mode and to the "all criteria pass" case, so it doesn't cover single-shot runs (which activation uses) or the stop-on-fail case — but it's a useful reference point for what the config surface and stop-reason reporting could look like.Possible approaches
A. Watch the live run and stop as soon as the signal appears. Some criteria — such as whether a skill was triggered, or whether a specific command ran — can be observed while the agent is still working, so the run can be cut off the moment the criterion is decided. Lighter-weight, and it naturally covers the stop-on-fail case for these signals. Limited to criteria that are observable mid-run.
B. Check a designated criterion between turns and stop. Drive the agent turn-by-turn and evaluate the chosen criterion after each turn, ending the run when it's decided. More general — works for any criterion — but a larger change to how runs are driven.
Considerations
Why this PR implements Approach A
There are two candidate methods for ending a run early once the measured criterion is decided.
Method A watches the live run: some criteria — whether a skill was triggered, whether a specific command ran — are observable in the event stream while the agent is still working, so a watcher evaluates them on each tool completion and cuts the run off at the next message boundary once the verdict is in.
Method B drives the agent turn-by-turn: the orchestrator regains control after every turn, evaluates the designated criterion against the sandbox, and ends the run when it is decided.
A has three decisive advantages over B. First, it fits the existing execution model: a single-shot run hands control to the SDK for the entire trajectory, so B would require rebuilding the drive loop into repeated single-turn calls with session continuity across every agent — while A rides the event stream the SDK already emits, stops at tool-call granularity (earlier than B's turn boundaries), and leaves the default path behaviorally unchanged. Second, it is sounder: A arms only criteria decided by positive events that cannot un-happen, so the live verdict and the authoritative re-check on the frozen trajectory agree by construction, whereas end-state criteria can flip after a mid-run "pass," making B's early verdicts guesses. Third, it is cheaper: live verdicts read the in-memory trajectory instead of B's per-turn sandbox inspections or judge calls. B remains the right tool only for genuinely end-state stop signals or agents that expose real turn boundaries — cases that are deferred behind hard resolution-time errors and can later slot in behind the same config surface.
In summary, A outperforms B on fit, correctness, and cost for everything this feature is meant to measure, which is why A is what ships here.
What's implemented
stop_when: pass|fail|decidedon any criterion (the "armed set") plus the opt-in master switchrun_limits.stop_early(defaultfalse— default behavior is unchanged).live_stop_polarities+live_verdictonBaseCriterion, implemented forskill_triggered(first-engagement policy) andcommand_executed(monotone count bounds, incl. the must-NOT-run form). Lint rule CE025 keeps the pair consistent.should_stoppoll at the Claude agent's existing between-message guard — cleanSTOPPED_EARLYfinalization (crashed=false, no SIGKILL), driven byEarlyStopWatcher(ownEventCollector+ stop rule) composed into the event stream.-D run_limits.stop_early=true; ships withexperiments/early-stop-ab.yaml(smoke vs. e2e from one task file) and docs recipes.EarlyStopInfo(reason, deciding criterion, SDK turn / tool-call index, elapsed seconds, turns-remaining upper bound) onEvaluationResult, surfaced as a run.md note, an HTML badge + advisory markers,stopped_earlyfields in run.json rows, andEarlyStopped/EarlyStopReasontelemetry dimensions.planandrun— never a silent no-op. A runtime live-verdict bug fails open to a full run.Deferred, not foreclosed: Approach B for end-state criteria / other agents / dialog mode slots in later behind the same config surface; evalboard badge (task.json already carries
early_stop).Testing
make verifygreen: 3382 passed, 3 expected skips, coverage 90.8% (gate 80%). New suites:tests/test_early_stop.py(models, live verdicts, watcher stop rule, agent seam incl. timeout-beats-stop precedence, orchestrator wiring, plan/run guardrail surfaces, report surfaces) and CE025 lint tests.stop_earlyunset, behavior is unchanged.experiments/early-stop-ab.yamlon an activation-style task: identical pass/fail verdicts between thee2eandsmokevariants, with the smoke variant substantially lower on turns, duration, and tokens.