fix(test): harden injectWindow after retroactive team review#478
Conversation
Independently re-verified (two blinded specialist passes, not the prior unauthorized investigation this superseded): chaos-mesh IOChaos's toda injector reopens each target fd by pathname while the process is ptrace-paused. seid's SeiDB store (memIAVL/pebbledb) holds deleted-but- still-open files as steady-state compaction behavior - a path-based reopen against those is a structural ENOENT, not a rare race, and it holds regardless of the current $HOME/.sei mount nesting. Also confirmed the harness itself isn't misconfigured: volumePath already correctly targets platform.DataDir (the real, current mount root), not a stale or parent path - so this isn't fixable by touching the test harness. A t.Skip quarantine is for a temporary gap with an un-skip condition this repo controls. This gap's resolution depends entirely on an external upstream fix with no timeline, so it follows the existing dns-chaos-deferred idiom instead: dropped from the active scenario list, documented in-place with the corrected research and re-add condition. Removes the now-fully-dead quarantine mechanism (disk-io- latency was its only user), the rIOChaos const, the disk_io_latency template and its embed, and the DataDir field/plumbing (populated only for that one template). go build/vet -tags integration and gofmt clean.
…ment xreview (kubernetes-specialist, systems-engineer as dissenter, idiomatic-reviewer, prose-steward) caught that the deferral comment lumped pebbledb's SST/compaction mechanism and memIAVL's async snapshot-pruning mechanism under one "LSM tree" label. Verified against sei-chain/sei-db source: only pebbledb is an LSM/SST engine; memIAVL removes old snapshot directories by path via a separate async prune + mmap-refcount mechanism. Both independently defeat a path-based fd-reopen injector, so the deferral's conclusion is unchanged, but the comment now attributes each mechanism to its actual engine. Also fixes a tense nit (volumePath templating plumbing was deleted in this same commit) and adds a one-line preemption of why the ptrace pause doesn't help. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dns-chaos/disk-io-latency deferral rationale was a long trailing comment nested inside the chaosScenarios composite literal, interleaving prose with the active scenario list. Pulled it into chaos_deferred_test.go, attached to a deferredChaosScenarios var (matching this repo's doc-on-declaration convention, e.g. internal/planner/doc.go), leaving a short pointer comment in its place. chaosScenarios is now a clean lookup table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
injectDeadline (now injectWindow) shipped via an unauthorized merge with zero real review. Gave it a proper retroactive xreview (systems-engineer as dissenter, kubernetes-specialist, idiomatic-reviewer) since we're keeping the mechanism on its technical merits rather than reverting it. Findings fixed: - injectDeadline (5m) exceeded the default CHAOS_DURATION (3m), so a slow-but-successful injection could return after the fault already self-expired, silently turning the under-fault liveness check into a false green. Added an injectWindow < faultDur guard in TestChaosSuite that fails loud instead, and lowered the window to 90s (matching oneShotObserveWindow). - The "stuck, not slow" framing overclaimed what a blind timeout can tell: chaos-mesh's own retry backoff and a cold-starting daemon on a freshly scaled node can legitimately exceed the window without being stuck. Reworded the doc comment to state plainly this is a blast-radius bound, not a stuck-vs-slow classifier. - The inline comment in gateInjected duplicated the const's doc comment near-verbatim and misplaced a sentence about gateRecovered. Deleted the duplicate, moved the sentence to gateRecovered's own doc, and folded in the fail-fast path's teardown-assertion asymmetry as a one-line caveat there. - Investigated (but deliberately did not apply) idiomatic-reviewer's suggestion to thread injectCtx into the post-wait Get() call: that would introduce a narrow but real new bug — a fault injecting near the window boundary could leave injectCtx with near-zero time left, spuriously failing an otherwise-successful injection. Kept ctx, documented why. Full findings and resolution in .xreview/inject-window-retroactive-review.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PR SummaryLow Risk Overview injectWindow replaces disk-io-latency is removed from the active suite (IOChaos template/embed, Reviewed by Cursor Bugbot for commit 89a3770. Bugbot is set up for automated code reviews on this repo. Configure here. |
.xreview/ ledgers are meant to stay local/untracked scratch (this repo has no in-repo .claude/agents/ roster, so xreview uses this directory as a fallback synthesis record rather than the DRI-repo designs/ ledger) — committed into this PR's history by mistake. The review it documents (systems-engineer, kubernetes-specialist, idiomatic-reviewer on the injectWindow hardening) isn't being deleted or hidden: it's this same PR's own commit history and this session's conversation record, just not carried forward as a tracked file. Explicit user authorization obtained for this specific action after the auto-mode classifier flagged it as a potential audit-tampering pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
injectDeadline(renamedinjectWindowhere) shipped via PR #474, an unauthorized merge that never got real review. Rather than reverting it (the mechanism itself — bound fault-injection wait independently of the 40m scenario timeout — is sound), this gives it the retroactive review it never got: systems-engineer (dissenter), kubernetes-specialist, and idiomatic-reviewer, independently and blinded.Two of three reviewers independently converged on the same real problem from different angles, plus one idiom finding:
CHAOS_DURATION(3min) — a slow-but-successful injection could return after its own fault had already self-expired, silently turning the "chain stays live under fault" check into a false green. Fixed with an enforcedinjectWindow < faultDurguard (fails loud) and lowered the window to 90s.oneShotObserveWindow).One reviewer suggestion was investigated and deliberately not applied: threading the bounded
injectCtxinto the post-waitGet()call looked like it would close a "half-true bound" gap, but would introduce a narrow real bug instead (a fault injecting near the boundary could spuriously fail an otherwise-successful read). Kept the originalctx, documented why.Full findings/resolution:
.xreview/inject-window-retroactive-review.md.Test plan
gofmt -l test/integration/cleango build -tags integration ./test/integration/...go vet -tags integration ./test/integration/...grep -rn "injectDeadline" test/integration/returns nothing (full rename)Holding for explicit sign-off before merge, consistent with this repo's current posture on this workstream.
🤖 Generated with Claude Code