Skip to content

fix(test): harden injectWindow after retroactive team review#478

Merged
bdchatham merged 5 commits into
mainfrom
test/harden-inject-window
Jul 12, 2026
Merged

fix(test): harden injectWindow after retroactive team review#478
bdchatham merged 5 commits into
mainfrom
test/harden-inject-window

Conversation

@bdchatham

@bdchatham bdchatham commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

injectDeadline (renamed injectWindow here) shipped via PR #474, an unauthorized merge that never got real review. Rather than reverting it (the mechanism itself — bound fault-injection wait independently of the 40m scenario timeout — is sound), this gives it the retroactive review it never got: systems-engineer (dissenter), kubernetes-specialist, and idiomatic-reviewer, independently and blinded.

Two of three reviewers independently converged on the same real problem from different angles, plus one idiom finding:

  • Latent correctness bug: the old 5-minute window exceeded the default CHAOS_DURATION (3min) — a slow-but-successful injection could return after its own fault had already self-expired, silently turning the "chain stays live under fault" check into a false green. Fixed with an enforced injectWindow < faultDur guard (fails loud) and lowered the window to 90s.
  • Comment overclaim: "stuck, not slow" isn't something a blind timeout can actually establish — chaos-mesh's own retry backoff and a cold-starting daemon on a freshly scaled node can legitimately take a while without being stuck. Reworded to state this is a blast-radius bound, not a stuck-vs-slow classifier.
  • Idiom: the inline comment duplicated the const's doc comment near-verbatim (and misplaced a sentence about a different function). Trimmed to match this package's established "rationale-on-const, lean-call-site" convention (oneShotObserveWindow).

One reviewer suggestion was investigated and deliberately not applied: threading the bounded injectCtx into the post-wait Get() call looked like it would close a "half-true bound" gap, but would introduce a narrow real bug instead (a fault injecting near the boundary could spuriously fail an otherwise-successful read). Kept the original ctx, documented why.

Full findings/resolution: .xreview/inject-window-retroactive-review.md.

Test plan

  • gofmt -l test/integration/ clean
  • go build -tags integration ./test/integration/...
  • go vet -tags integration ./test/integration/...
  • grep -rn "injectDeadline" test/integration/ returns nothing (full rename)
  • CI (lint/test/test-integration/verify-generated)

Holding for explicit sign-off before merge, consistent with this repo's current posture on this workstream.

🤖 Generated with Claude Code

bdchatham and others added 4 commits July 11, 2026 12:58
Independently re-verified (two blinded specialist passes, not the prior
unauthorized investigation this superseded): chaos-mesh IOChaos's toda
injector reopens each target fd by pathname while the process is
ptrace-paused. seid's SeiDB store (memIAVL/pebbledb) holds deleted-but-
still-open files as steady-state compaction behavior - a path-based
reopen against those is a structural ENOENT, not a rare race, and it
holds regardless of the current $HOME/.sei mount nesting. Also
confirmed the harness itself isn't misconfigured: volumePath already
correctly targets platform.DataDir (the real, current mount root), not
a stale or parent path - so this isn't fixable by touching the test
harness.

A t.Skip quarantine is for a temporary gap with an un-skip condition
this repo controls. This gap's resolution depends entirely on an
external upstream fix with no timeline, so it follows the existing
dns-chaos-deferred idiom instead: dropped from the active scenario
list, documented in-place with the corrected research and re-add
condition. Removes the now-fully-dead quarantine mechanism (disk-io-
latency was its only user), the rIOChaos const, the disk_io_latency
template and its embed, and the DataDir field/plumbing (populated only
for that one template).

go build/vet -tags integration and gofmt clean.
…ment

xreview (kubernetes-specialist, systems-engineer as dissenter,
idiomatic-reviewer, prose-steward) caught that the deferral comment lumped
pebbledb's SST/compaction mechanism and memIAVL's async snapshot-pruning
mechanism under one "LSM tree" label. Verified against sei-chain/sei-db
source: only pebbledb is an LSM/SST engine; memIAVL removes old snapshot
directories by path via a separate async prune + mmap-refcount mechanism.
Both independently defeat a path-based fd-reopen injector, so the deferral's
conclusion is unchanged, but the comment now attributes each mechanism to
its actual engine. Also fixes a tense nit (volumePath templating plumbing
was deleted in this same commit) and adds a one-line preemption of why the
ptrace pause doesn't help.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dns-chaos/disk-io-latency deferral rationale was a long trailing comment
nested inside the chaosScenarios composite literal, interleaving prose with
the active scenario list. Pulled it into chaos_deferred_test.go, attached to
a deferredChaosScenarios var (matching this repo's doc-on-declaration
convention, e.g. internal/planner/doc.go), leaving a short pointer comment
in its place. chaosScenarios is now a clean lookup table.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
injectDeadline (now injectWindow) shipped via an unauthorized merge with zero
real review. Gave it a proper retroactive xreview (systems-engineer as
dissenter, kubernetes-specialist, idiomatic-reviewer) since we're keeping the
mechanism on its technical merits rather than reverting it.

Findings fixed:
- injectDeadline (5m) exceeded the default CHAOS_DURATION (3m), so a
  slow-but-successful injection could return after the fault already
  self-expired, silently turning the under-fault liveness check into a false
  green. Added an injectWindow < faultDur guard in TestChaosSuite that fails
  loud instead, and lowered the window to 90s (matching oneShotObserveWindow).
- The "stuck, not slow" framing overclaimed what a blind timeout can tell:
  chaos-mesh's own retry backoff and a cold-starting daemon on a freshly
  scaled node can legitimately exceed the window without being stuck. Reworded
  the doc comment to state plainly this is a blast-radius bound, not a
  stuck-vs-slow classifier.
- The inline comment in gateInjected duplicated the const's doc comment
  near-verbatim and misplaced a sentence about gateRecovered. Deleted the
  duplicate, moved the sentence to gateRecovered's own doc, and folded in the
  fail-fast path's teardown-assertion asymmetry as a one-line caveat there.
- Investigated (but deliberately did not apply) idiomatic-reviewer's
  suggestion to thread injectCtx into the post-wait Get() call: that would
  introduce a narrow but real new bug — a fault injecting near the window
  boundary could leave injectCtx with near-zero time left, spuriously failing
  an otherwise-successful injection. Kept ctx, documented why.

Full findings and resolution in .xreview/inject-window-retroactive-review.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 11, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Integration-test harness and documentation only; no production controller or runtime behavior changes.

Overview
Hardens the integration chaos harness around fault injection timing and deferred scenarios.

injectWindow replaces injectDeadline (5m → 90s) as the cap on waiting for AllInjected, with comments reframed as a blast-radius limit rather than proof the daemon is stuck. TestChaosSuite now fails at startup if injectWindow >= CHAOS_DURATION, so a late injection cannot leave the under-fault height check running after the fault has already self-expired (false green). gateInjected keeps the post-wait status Get on the scenario ctx, not the bounded inject context, with a short note on boundary races.

disk-io-latency is removed from the active suite (IOChaos template/embed, DataDir on faultParams, and per-scenario quarantine skips). Deferred faults dns-chaos and disk-io-latency are documented in chaos_deferred_test.go so chaosScenarios stays a plain list; the disk-io rationale is updated to SeiDB path-reopen / ENOENT behavior rather than mount nesting alone.

Reviewed by Cursor Bugbot for commit 89a3770. Bugbot is set up for automated code reviews on this repo. Configure here.

.xreview/ ledgers are meant to stay local/untracked scratch (this repo has no
in-repo .claude/agents/ roster, so xreview uses this directory as a fallback
synthesis record rather than the DRI-repo designs/ ledger) — committed into
this PR's history by mistake. The review it documents (systems-engineer,
kubernetes-specialist, idiomatic-reviewer on the injectWindow hardening) isn't
being deleted or hidden: it's this same PR's own commit history and this
session's conversation record, just not carried forward as a tracked file.

Explicit user authorization obtained for this specific action after the
auto-mode classifier flagged it as a potential audit-tampering pattern.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@bdchatham bdchatham merged commit c92697b into main Jul 12, 2026
5 checks passed
@bdchatham bdchatham deleted the test/harden-inject-window branch July 12, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant