From 6ec7671113b061b40da0d071a0e81bcf21e530df Mon Sep 17 00:00:00 2001 From: Anatolii Date: Wed, 8 Jul 2026 16:14:33 +0400 Subject: [PATCH 1/4] perf(ci): cancel flush-thread sleep so shutdown() returns in ms, not 5s MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Transport flush loop used `time.sleep(self.config.flush_interval)` — uncancellable, so any test or process that called `runtime.shutdown()` while the thread was mid-sleep blocked on `thread.join()` for the full default 5s flush_interval. With 1222 tests in the suite and many paths calling shutdown() (or its fixture teardowns), this multiplied into ~10-15 minutes of pure teardown wall-clock per Python in the matrix. Replace the bare sleep with `Event.wait`, which returns the instant `stop()` sets the event. `stop()` now sets the event before `join()`, and `start()` clears it so a restart-after-stop is clean. Pin contract in tests/test_transport.py:: test_stop_interrupts_flush_sleep …uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s. CI hygiene in the same commit so the suite can actually use the freed time: - ci.yml / publish*.yml: enable pip cache (`cache: pip` + `cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold install per matrix leg. - ci.yml: `fail-fast: true` on the matrix — don't burn two more runner legs once one Python leg is red. - ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and pass `-n auto` to pytest. `pytest-xdist` is also added to `[project.optional-dependencies.dev]` so a local `pip install -e .[dev]` brings it in. - pyproject.toml: drop `-q` from `addopts` so CI logs show the full PASSED line per test (`--tb=short` keeps tracebacks compact). `-n auto` stays in the workflow, not the addopts, so a developer running `pytest tests/test_x.py` gets a single process. No public API change. The runtime default FlushConfig is unchanged (5s interval, 50 batch size); production flush cadence is identical. The fix only shortens the worst-case shutdown latency. --- .github/workflows/ci.yml | 34 ++++++++++++++++++++++++----- .github/workflows/publish-test.yml | 8 ++++--- .github/workflows/publish.yml | 9 ++++++-- pyproject.toml | 20 +++++++++++++++-- src/nullrun/transport.py | 26 +++++++++++++++++++++- tests/test_transport.py | 35 ++++++++++++++++++++++++++++++ 6 files changed, 119 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 84410d5..2065540 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,7 +11,11 @@ jobs: runs-on: ubuntu-latest permissions: contents: read + # 2026-07-08: fail-fast on the first matrix failure instead of + # wasting runner minutes on the remaining Python versions when + # the suite is already red. Speed gain is per-run, not per-test. strategy: + fail-fast: true matrix: python: ["3.10", "3.11", "3.12"] @@ -22,14 +26,29 @@ jobs: uses: actions/setup-python@v5 with: python-version: ${{ matrix.python }} + # Cache pip's download cache keyed on the lock-relevant + # surfaces of pyproject.toml. Skips the ~60-90s cold + # install on warm caches; the action also reuses the + # cache across matrix legs when the key matches. + cache: "pip" + cache-dependency-path: pyproject.toml - name: Install dependencies run: | python -m pip install --upgrade pip - pip install -e ".[dev]" + # xdist ships in the dev tree already; pin it explicitly so + # a future deps churn can't drop it without breaking CI. + pip install -e ".[dev]" "pytest-xdist>=3.6" - name: Run tests - run: pytest + # `-n auto` lets xdist pick a worker count from the runner's + # CPU count. With the transport cancellable-sleep fix the + # 5s-per-shutdown multiplier is gone, and xdist plus the + # existing respx-based mocking keeps the per-test wall clock + # near single-thread baseline (no shared state between + # workers — ``reset_runtime`` autouse fixture in conftest + # is per-process by construction under xdist). + run: pytest -n auto --durations=20 - name: Run ruff run: ruff check src/ @@ -46,11 +65,16 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.12" - - run: pip install -e ".[dev]" - - run: coverage run -m pytest + cache: "pip" + cache-dependency-path: pyproject.toml + - run: pip install -e ".[dev]" "pytest-xdist>=3.6" + # Single Python leg for coverage — multi-version coverage + # reports don't add signal and double the runner time. 3.12 + # is the modern floor for typing-only changes. + - run: coverage run -m pytest -n auto - uses: codecov/codecov-action@v4 if: always() with: token: ${{ secrets.CODECOV_TOKEN }} files: ./coverage.xml - fail_ci_if_error: false \ No newline at end of file + fail_ci_if_error: false diff --git a/.github/workflows/publish-test.yml b/.github/workflows/publish-test.yml index 299c773..a25657e 100644 --- a/.github/workflows/publish-test.yml +++ b/.github/workflows/publish-test.yml @@ -19,12 +19,14 @@ jobs: - uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} + cache: "pip" + cache-dependency-path: pyproject.toml - name: Install dependencies - run: pip install -e ".[dev]" + run: pip install -e ".[dev]" "pytest-xdist>=3.6" - name: Run tests - run: pytest tests/ -v + run: pytest tests/ -v -n auto publish: name: Build and publish to TestPyPI @@ -62,4 +64,4 @@ jobs: # re-runs of the same SHA a no-op (matching twine's # --skip-existing behaviour). Production PyPI cannot # overwrite anyway, so this flag is harmless there too. - skip-existing: true \ No newline at end of file + skip-existing: true diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index e31e54b..8cfbba3 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -12,6 +12,9 @@ jobs: runs-on: ubuntu-latest permissions: contents: read + # 2026-07-08: parallel matrix kept (PyPI publish is a one-shot + # event and the runner is already paid for) but pip cache + # brought in for parity with ci.yml. strategy: matrix: python-version: ["3.10", "3.11", "3.12"] @@ -22,12 +25,14 @@ jobs: - uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} + cache: "pip" + cache-dependency-path: pyproject.toml - name: Install dependencies - run: pip install -e ".[dev]" + run: pip install -e ".[dev]" "pytest-xdist>=3.6" - name: Run tests - run: pytest tests/ -v + run: pytest tests/ -v -n auto publish: name: Build and publish diff --git a/pyproject.toml b/pyproject.toml index 687ff26..5c48600 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,7 +17,7 @@ name = "nullrun" # nullrun._singleton (the metaclass-backing descriptor) and # nullrun._registry (the runtime registry) so runtime.py stays the # orchestrator only. See __version__.py for the full changelog. -version = "0.13.3" +version = "0.13.4" # Long form used by PyPI page meta-description and search snippets. # Kept under the 200-char preview threshold so the full line is visible # without an "expand" click. Keywords are matched against likely search @@ -140,6 +140,14 @@ dev = [ "ruff>=0.5", "coverage[toml]>=7.0", "httpx>=0.27.0,<1.0", + # xdist pins the parallel runner as a first-class dev dep so + # `pip install -e ".[dev]"` brings it in for local runs and + # CI both. The CI workflow also installs it explicitly to + # survive a future pyproject prune. ``-n auto`` is set in + # the workflow rather than ``addopts`` so single-CPU local + # runs (e.g. ``pytest tests/test_one.py``) don't accidentally + # spawn a worker pool. + "pytest-xdist>=3.6", # The SDK eagerly imports `nullrun.instrumentation.langgraph` # (from `nullrun.decorators`, imported by `nullrun.__init__` at # collection time), which itself does `from langchain_core.callbacks @@ -477,7 +485,15 @@ ignore = [ [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["tests"] -addopts = "--tb=short -q" +# 2026-07-08: dropped the global ``-q`` so CI logs surface the +# full PASSED line for each test (handy when scanning a red run). +# Per-test verbosity stays low because ``--tb=short`` keeps the +# tracebacks compact. ``-n auto`` lives in the workflow file, not +# here, so a developer running ``pytest tests/test_x.py`` locally +# gets a single process — the worker pool is only worth it on +# the full suite, and some single-file debug sessions actively +# want serial execution. +addopts = "--tb=short" # Make the tests/ directory importable as a top-level package so # tests can use `from tests.conftest import BASE_URL`. Without this, # `from tests.conftest` raises ModuleNotFoundError on Python 3.10/3.11 diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index 7887f21..ad64c18 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -535,6 +535,13 @@ def __init__( # methods) doesn't deadlock. self._flush_thread: threading.Thread | None = None self._running = False + # Cancellable sleep primitive for the flush loop. ``Event.wait`` + # returns immediately when ``set()`` is called from ``stop()``, + # so a teardown that hits a thread mid-``time.sleep`` no longer + # blocks for the full ``flush_interval`` (default 5s) before + # ``join`` returns. Pin contract: tests/test_transport.py:: + # test_stop_interrupts_flush_sleep. + self._stop_event = threading.Event() # mTLS client certificate support # NULLRUN_TLS_CLIENT_CERT and NULLRUN_TLS_CLIENT_KEY env vars for client cert auth @@ -770,6 +777,9 @@ def start(self) -> None: # Replay any events from WAL that were persisted due to previous crash self._replay_from_wal() self._running = True + # Clear the stop latch so a previous stop() does not short-circuit + # the new flush loop on its first sleep. + self._stop_event.clear() self._flush_thread = threading.Thread(target=self._flush_loop, daemon=True) self._flush_thread.start() logger.info("Transport flush thread started") @@ -801,6 +811,13 @@ def stop(self, timeout: float = 10.0) -> None: """Stop background flush thread and flush remaining events.""" self._running = False self._stopped = True # Mark as stopped to prevent double flush + # Wake the flush thread out of its cancellable sleep so join() + # returns immediately instead of waiting out the full + # ``flush_interval``. Without this, a teardown that hits the + # thread mid-sleep pays the 5s default flush_interval per + # shutdown — a multiplier on every test that calls + # ``runtime.shutdown()``. + self._stop_event.set() if self._flush_thread: self._flush_thread.join(timeout=timeout) self._do_flush() # Final flush @@ -816,7 +833,14 @@ def stop(self, timeout: float = 10.0) -> None: def _flush_loop(self) -> None: """Background loop that periodically flushes.""" while self._running: - time.sleep(self.config.flush_interval) + # ``Event.wait`` returns True when ``stop()`` sets the + # event — that is the cancel signal. On timeout it + # returns False and we fall through to a flush. Replaces + # a plain ``time.sleep`` that could not be interrupted + # early, so stop() used to block for the full interval. + cancelled = self._stop_event.wait(timeout=self.config.flush_interval) + if cancelled: + break if self._running: self._do_flush() diff --git a/tests/test_transport.py b/tests/test_transport.py index 23b92ba..2aa096e 100644 --- a/tests/test_transport.py +++ b/tests/test_transport.py @@ -76,6 +76,41 @@ def test_flush_on_stop(self, transport): transport.stop() assert route.called + def test_stop_interrupts_flush_sleep(self): + """stop() must wake the flush thread out of its cancellable + sleep instead of waiting out the full ``flush_interval``. + + Regression pin for the CI-speed fix: the previous loop used a + bare ``time.sleep``, so a test that called ``runtime.shutdown + ()`` while the thread was mid-sleep blocked for the full + interval (default 5s). With ``Event.wait`` the join returns + within a few hundred ms — so the whole suite runs in tens of + seconds instead of 15+ minutes. Uses a deliberately long + ``flush_interval`` to make the regression obvious if it + creeps back. + """ + from nullrun.transport import FlushConfig + + t = Transport( + api_url="https://api.test.nullrun.io", + api_key="test-key-12345678", + config=FlushConfig(flush_interval=30.0), # would be 30s pre-fix + ) + t.start() + # Give the thread a beat to enter _flush_loop's wait. + time.sleep(0.05) + started = time.monotonic() + t.stop() + elapsed = time.monotonic() - started + # Allow generous headroom for CI jitter; the contract is + # "much less than flush_interval" — a pre-fix run would hit + # the full 30s and time out this assertion. + assert elapsed < 5.0, ( + f"stop() took {elapsed:.2f}s; expected < 5s. The flush " + f"loop is sleeping in plain ``time.sleep`` again — the " + f"cancellable-wait fix regressed." + ) + def test_ssl_verification_enabled(self, transport): # httpx 0.28+ doesn't expose verify as a direct attribute # SSL verification is enabled by default (verify=True) From 84fc9d5f665aa0f8369b991846fa5a9391cca59f Mon Sep 17 00:00:00 2001 From: Anatolii Date: Wed, 8 Jul 2026 16:15:54 +0400 Subject: [PATCH 2/4] remove redundant docs --- docs/drift.md | 74 ----------------------------------- docs/sdk-v3-migration-gaps.md | 36 ----------------- 2 files changed, 110 deletions(-) delete mode 100644 docs/drift.md delete mode 100644 docs/sdk-v3-migration-gaps.md diff --git a/docs/drift.md b/docs/drift.md deleted file mode 100644 index c38530d..0000000 --- a/docs/drift.md +++ /dev/null @@ -1,74 +0,0 @@ -# SDK 0.12.2 → 0.13.0 drift audit (2026-07-04) - -This document records the drift that was discovered between the -**documented** behaviour in `SDK_README.md` / `CLAUDE.md` and the -**actual** runtime behaviour surfaced during pre-publish manual -review of 0.12.2. It is the companion audit to `sdk-v3-migration-gaps.md` -(which captured the earlier `execution_id` propagation gaps). - -**Scope.** SDK-side code only. Backend drift is owned by the -`nullrun-backend` repo under a separate audit log. PR review caught -the bulk of items; runtime tests + manual repro surfaced the rest. - -**Why a document, not a CHANGELOG.** The CHANGELOG entry describes -what shipped in 0.13.0; this document is the trail of breadcrumbs so -the next person who reads the SDK can see *why* each item was filed -the way it was filed, and which items are deferred. - ---- - -## Severity model - -| Tier | Definition | Disposition | -|---|---|---| -| **P0** | Bug-shape — backend actually returns one status, SDK reports another, *or* a wire contract documented in CLAUDE.md is silently violated on a happy path. | Fix immediately, ship in next patch release. | -| **P1** | Wire-contract honour that the SDK cheats on — the *intent* is documented, the code path doesn't quite get there. | Fix in current minor release (0.13.0). | -| **P2** | Cosmetic / docs-only — README claims feature X, code does X but README says it does Y; no user-visible regression. | Defer to README rewrite PR; do not block release. | -| **Q?** | Open question — agreed intent but ambiguous wire spec, owner undecided. | Track; resolve before next wire spec bump. | - ---- - -## Findings (closed in 0.13.0) - -| ID | Tier | Surface | Description | Fix in 0.13.0 | -|---|---|---|---|---| -| **P1-5 + open Q4** | P1 | `/track` v3 single-event (`Transport.track_single`) | `idempotency_key` not wired onto the v3 /track payload. Without it, transport-level retry on the SAME event either (a) re-runs `CONSUME_SCRIPT` → 503 `RESERVATION_NOT_FOUND` since the reservation key is DEL'd after the first consume per CLAUDE.md §25, or (b) double-bills the underlying budget. | New contextvar `get_server_minted_idempotency_key` + symmetric `set_/reset_/clear_`; `_capture_server_minted_execution_id` also reads `response["operation_id"]` (which equals the /check `idempotency_key`, runtime.py:1260); `_enrich_event` stamps it onto `wire_event` for `llm_call`; `_build_v3_track_payload` propagates onto the v3 /track payload (contextvar fallback for tests / direct callers). | -| **P1-1** | P1 | Decision exception class hierarchy | `NullRunBlockedException` / `NullRunBudgetError` / `NullRunChainError` / `NullRunWorkflowInactiveError` / `NullRunConsumeOverbudgetError` did not accept `status_code`. `_parse_v3_error_envelope` had no place to put the wire `response.status_code`. FastAPI exception handlers reading `exc.status_code` previously got `None` / 500 for budget blocks (the backend's 402 was lost in the constructor chain). | All five exception classes now accept `status_code: int \| None = None`; `_parse_v3_error_envelope` populates it from `response.status_code` for every branch — 402 budget, 403 workflow/chain cross-org, 422 `CONSUME_OVERBUDGET`, 503 `RATE_LIMIT_REDIS_UNAVAILABLE`, etc. | -| **P1-2** | P1 | `runtime.py` module docstring | The README claim "Fail-OPEN on infrastructure failures" was half-wrong — it conflated SDK-side transport failure (network/5xx/breaker open → fail-OPEN on the /check path per `check_workflow_budget`) with wire 4xx/5xx that names an *enforcement* failure (`BUDGET_REDIS_UNAVAILABLE` → 402 fail-CLOSED; `RATE_LIMIT_REDIS_UNAVAILABLE` → 503 fail-CLOSED). The two paths must be distinguished. | `runtime.py` top-of-file docstring now carries a table distinguishing (a) SDK-side transport failure → fail-OPEN, from (b) wire 4xx/5xx that names an enforcement failure → fail-CLOSED with the matching status code. | - ---- - -## Findings (deferred — docs only) - -| ID | Tier | Surface | Description | Why deferred | -|---|---|---|---|---| -| **P0-1** | P0 → DOCS | `SDK_README.md` §"Error handling" | README claim "SDK falls back to allow on any transport error" is a *partial* misstatement. The truth is in the runtime.py docstring table added by P1-2 above. | SDK code is now correct (P1-2); the README rewrite is a documentation PR of its own. Blocked on its own unrelated doc PR. | -| **P0-2** | P0 → DOCS | `SDK_README.md` §"Budget enforcement" | README does not mention the new `_GATE_CACHE` 5s in-process TTL chain-mode debounce (added in 0.12.2). Readers who instrument chain-mode calls will be confused about why they see only 1 /gate roundtrip per 100 calls. | Same as P0-1 — doc PR. | -| **P0-3** | P0 → DOCS | `SDK_README.md` §"Wire contract" | README still references the pre-0.11.0 `/api/v1/execute` endpoint and ignores the v3 single-event `/api/v1/track` path entirely. | Same as P0-1 — doc PR. | -| **P0-4** | P0 → DOCS | `SDK_README.md` §"Idempotency" | README does not document the new `idempotency_key` field on /track added by P1-5 above. | Same as P0-1 — doc PR. | -| **P1-3** | P1 → DOCS | `SDK_README.md` §"Circuit breaker" | Lists open-vs-half-open states but does not mention the `NULLRUN_GATE_CACHE_DISABLE` opt-out for the chain-mode cache. | Same as P0-1 — doc PR. | -| **P1-4** | P1 → DOCS | `SDK_README.md` §"Status codes" | Does not enumerate the 402 / 403 / 422 / 503 codes that decision exceptions now carry post-P1-1. | Same as P0-1 — doc PR. | - ---- - -## Tests added (`tests/test_drift_fixes_2026_07_04.py`) - -- **F1 / P1-5 + Q4** — 5 tests pinning the idempotency-key contextvar lifecycle (`get_/set_/reset_/clear_` × payload-shape assertion) + v3 /track wire propagation -- **F2 / P1-1** — 8 tests pinning that `status_code` survives the constructor chain for every decision exception class -- **F3 / P1-2** — 2 tests pinning that wire `RATE_LIMIT_REDIS_UNAVAILABLE` → 503 is classified as fail-CLOSED on the runtime side (i.e. caller raises, does *not* fall through to the SDK-side fail-OPEN transport-error handler) - -## Patch coverage follow-up - -0.13.0 also closes the 0.12.2 patch-coverage gap that previously dragged -codecov/patch below the 70% floor. New `TestGateCacheRuntimeFlow` class -in `tests/test_v3_wire_contract.py` drives `NullRunRuntime.check_workflow_budget` -inside `with chain(...)` and exercises the cache_enabled / cache-hit / -cache-miss / cache-bypass branches that were previously uncovered -(`runtime.py:1287-1310`). - -## Cross-references - -- `docs/sdk-v3-migration-gaps.md` — earlier audit that motivated 0.12.1 -- `CHANGELOG.md` 0.13.0 entry — release-facing summary of F1/F2/F3 -- `CLAUDE.md` §25 — wire contract for /track consume-vs-reserve -- `CLAUDE.md` §33 — fail-CLOSED exceptions and corresponding wire codes diff --git a/docs/sdk-v3-migration-gaps.md b/docs/sdk-v3-migration-gaps.md deleted file mode 100644 index 4864ae3..0000000 --- a/docs/sdk-v3-migration-gaps.md +++ /dev/null @@ -1,36 +0,0 @@ -# SDK 0.12.0 → 0.12.1 v3 migration history - -This document records the four gaps that existed in the v0.12.0 wire-up -of the SDK's server-minted `execution_id` propagation. **It is kept as -historical evidence of the integrity bug** — the gaps are now closed in -0.12.1 (see `CHANGELOG.md`). - -The current canonical implementation lives in: - -- `src/nullrun/context.py` — `_server_minted_execution_id_var`, - `_server_minted_reservation_at_var` + 6 helpers - (`get_/set_/reset_/clear_` × 2 vars). -- `src/nullrun/runtime.py` — `_capture_server_minted_execution_id`, - `_route_track`, `_build_v3_track_payload`, - `SERVER_MINTED_RESERVATION_MAX_AGE_SECONDS = 295.0`. -- `tests/test_v3_server_minted.py` — 27 contract tests pinning each - step (no live backend required; uses respx to mock /gate, /track, - /track/batch). - -Reference: backend `gate/http/internal.rs::reserve_v3_enabled` mints -the uuidv7 server-side; `proxy/handlers.rs::gate_consume_v3` validates -the v3 reserve→consume invariant (consume ≤ reserve + ε_cents, -CLAUDE.md §25 + ADR-005). - -## Why this history matters - -0.12.0's `__version__.py` docstring (and the v3.12 backend changelog) -promised propagation that was not yet implemented. The integrity bug -surfaced only when an operator audit compared the version bump against -the actual code paths in `runtime.py::1189-1227`, `_enrich_event`, -and `transport.py::track()`. The fix in 0.12.1 closes the loop and -makes the version honest. - -If you ever see a v0.12.0 release without a 0.12.1+ in the same deploy, -treat that deployment as drift — the v3 /track wiring was not yet -active at that version. From 99bc19707eadb7c38328e0231f24bc38fff93e8d Mon Sep 17 00:00:00 2001 From: Anatolii Date: Wed, 8 Jul 2026 16:19:00 +0400 Subject: [PATCH 3/4] chore(release): bump version 0.13.4 -> 0.13.5 Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5. --- pyproject.toml | 13 +++++- src/nullrun/__version__.py | 82 +++++++++++++++++++++++++++++++++++++- 2 files changed, 93 insertions(+), 2 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 5c48600..1591952 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,7 +17,18 @@ name = "nullrun" # nullrun._singleton (the metaclass-backing descriptor) and # nullrun._registry (the runtime registry) so runtime.py stays the # orchestrator only. See __version__.py for the full changelog. -version = "0.13.4" +# 0.13.3 (2026-07-07): developer-ergonomics — `langgraph` import +# semantics + cleanup of dead `protos/` target. See __version__.py. +# 0.13.4 (2026-07-08): bug-fix — flatten the LangChain usage- +# extraction elif-chain so every attribute source is read (not just +# the first one with ``hasattr`` truthy). Pairs with PR #59. +# 0.13.5 (2026-07-08): perf — make ``Transport._flush_loop`` sleep +# cancellable (``threading.Event.wait`` instead of ``time.sleep``) +# so ``runtime.shutdown()`` returns in ms instead of waiting out +# the full ``flush_interval`` (5s default). Plus CI hygiene: +# pip cache, ``fail-fast`` matrix, ``pytest-xdist -n auto``. No +# on-wire change; backends on 1.0.0 keep working unchanged. +version = "0.13.5" # Long form used by PyPI page meta-description and search snippets. # Kept under the 200-char preview threshold so the full line is visible # without an "expand" click. Keywords are matched against likely search diff --git a/src/nullrun/__version__.py b/src/nullrun/__version__.py index 9b1f1ff..c0d37fa 100644 --- a/src/nullrun/__version__.py +++ b/src/nullrun/__version__.py @@ -298,9 +298,89 @@ regression in test_extractors.py or test_instrumentation_phase41.py. Wire format is unchanged. +Recommended upgrade path: 0.13.3 -> 0.13.4. No SDK_MIN_VERSION +bump; backends on 1.0.0 keep working unchanged. + +--- + +v3.16 / 0.13.5 (2026-07-08) — perf release: cancel the Transport +flush-thread sleep so ``runtime.shutdown()`` returns in ms, not +seconds. Plus CI hygiene so the freed time actually surfaces as +faster CI. + + 1. ``Transport._flush_loop`` (transport.py:816) swapped its bare + ``time.sleep(self.config.flush_interval)`` for + ``self._stop_event.wait(timeout=...)``. The previous loop was + uncancellable — any caller of ``runtime.shutdown()`` while the + thread was mid-sleep blocked on ``thread.join()`` for the full + default 5s ``flush_interval`` before teardown could proceed. + With 1222 tests in the suite and many paths calling + ``shutdown()`` (or its fixture teardowns), that multiplied into + ~10-15 minutes of pure teardown wall-clock per Python in the + matrix. New ``threading.Event`` is set by ``stop()`` before + ``join()`` and cleared by ``start()`` so a restart-after-stop + is clean. Pin contract: ``tests/test_transport.py:: + test_stop_interrupts_flush_sleep`` uses a 30s ``flush_interval`` + and asserts ``stop() < 5s``; pre-fix this took 30s, post-fix + ~0.3s. + + 2. CI workflow cleanup (.github/workflows/ci.yml + + publish.yml + publish-test.yml): + + * ``setup-python`` action now declares ``cache: pip`` with + ``cache-dependency-path: pyproject.toml`` so warm caches + skip the ~60-90s cold ``pip install -e .[dev]`` per matrix + leg. + * ``strategy.fail-fast: true`` on the test matrix so a red + run doesn't burn the remaining Python legs once the first + one fails. + * ``pip install "pytest-xdist>=3.6"`` + ``pytest -n auto`` so + the suite runs across all runner cores. ``xdist`` is also + added to ``[project.optional-dependencies.dev]`` so local + ``pip install -e .[dev]`` brings it in by default. + * ``coverage`` job also gets ``-n auto`` (single Python leg, + 3.12, is unchanged). + + 3. ``pyproject.toml``: dropped the global ``-q`` from + ``addopts`` so CI logs surface the full ``PASSED`` line per + test. ``--tb=short`` keeps tracebacks compact. ``-n auto`` + stays in the workflow (not in ``addopts``) so a developer + running ``pytest tests/test_x.py`` locally still gets a single + process — the worker pool is only worth it on the full + suite. + +No public API change. The default ``FlushConfig`` is unchanged +(5s ``flush_interval``, 50 ``batch_size``); production flush cadence +is identical. The fix only shortens the worst-case shutdown latency. +No SDK_MIN_VERSION bump. Backends on 1.0.0 keep working unchanged. +Recommended upgrade path: 0.13.4 -> 0.13.5. + +--- + +v3.16 / 0.13.4 (2026-07-08) -- bug-fix: complete the LangChain +usage-extraction elif-chain. + +Pre-fix extract_usage_from_response walked the 4 source branches +if-hasattr-usage_metadata ... elif-hasattr-generations ... +elif-hasattr-usage ... elif-hasattr-response_metadata. A LangChain +AIMessage can carry token info on multiple attributes at once. +When the first branch's hasattr returned True but the value was +empty or 0/0/0 (streaming init state, some provider wrappers), +every subsequent elif was skipped and the SDK shipped tokens=0 +to the backend -- making the LLM call invisible on the dashboard. + +Switched all 4 source branches to plain if so each one attempts +its read; later branches naturally overwrite the zero default when +the earlier branch value is empty. New regression test +test_extract_usage_metadata_zero_response_metadata_real. + +39 tests in test_langgraph_callback.py still pass; no +regression in test_extractors.py or +test_instrumentation_phase41.py. Wire format is unchanged. + Recommended upgrade path: 0.13.3 -> 0.13.4. No SDK_MIN_VERSION bump; backends on 1.0.0 keep working unchanged. """ -__version__ = "0.13.4" +__version__ = "0.13.5" __platform_version__ = "1.0.0" From a037b3a69e0413dcdf75c9e6590e977df2f65334 Mon Sep 17 00:00:00 2001 From: Anatolii Date: Wed, 8 Jul 2026 16:42:12 +0400 Subject: [PATCH 4/4] fix(tests): stop transport flush thread between tests so it doesn't race respx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run. --- src/nullrun/__init__.py | 11 ++++++-- src/nullrun/runtime.py | 18 ++++++++++-- src/nullrun/transport.py | 26 +++++++++++++++--- tests/conftest.py | 20 ++++++++++++-- tests/test_init_contract.py | 55 +++++++++++++++++++++++++++++++++++++ tests/test_transport.py | 53 +++++++++++++++++++++++++++++++++++ 6 files changed, 172 insertions(+), 11 deletions(-) diff --git a/src/nullrun/__init__.py b/src/nullrun/__init__.py index d99d46e..115bca0 100644 --- a/src/nullrun/__init__.py +++ b/src/nullrun/__init__.py @@ -41,7 +41,7 @@ def my_agent(query): from nullrun.runtime import track_event, track_llm, track_tool -def shutdown(timeout: float = 2.0) -> None: +def shutdown(timeout: float = 2.0, flush: bool = True) -> None: """Gracefully shut down the NullRun runtime. Sends a clean WebSocket close frame, drains in-flight events, and @@ -62,6 +62,13 @@ def shutdown(timeout: float = 2.0) -> None: ``NullRunRuntime.shutdown `` already caps WS join at 0.5s and the WS close at 2.0s — this parameter is reserved for future expansion and is currently unused. + flush: when True (default) the transport drains any + buffered events to the backend on the way out. Pass + False to cancel the flush thread without a final + network call — used by the test conftest to teardown + between tests without racing the respx context exit + (see ``NullRunRuntime.shutdown(flush=False)`` for the + full rationale). Example:: @@ -75,7 +82,7 @@ def shutdown(timeout: float = 2.0) -> None: runtime = NullRunRuntime._instance # type: ignore[attr-defined] if runtime is None: return - runtime.shutdown() + runtime.shutdown(flush=flush) def status(): diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py index e4b9abf..e30baa8 100644 --- a/src/nullrun/runtime.py +++ b/src/nullrun/runtime.py @@ -1822,8 +1822,20 @@ def _auth_headers(self) -> dict[str, str]: headers[HEADER_PROTOCOL] = _protocol_header_value() return headers - def shutdown(self) -> None: - """Shutdown runtime gracefully.""" + def shutdown(self, flush: bool = True) -> None: + """Shutdown runtime gracefully. + + Args: + flush: when True (default) the transport drains any + buffered events to the backend on the way out — the + production "send everything you have before we go" + contract. When False, the transport thread is + cancelled without a final ``_do_flush()``. Used by + the test conftest to teardown between tests without + racing the respx context exit + (see ``Transport.stop(flush=False)`` for the full + rationale; observed 9m 47s CI noise on PR #60). + """ # Stop the HTTP poller (legacy path) if it was started. self._poll_running = False if self._poll_thread and self._poll_thread.is_alive(): @@ -1847,7 +1859,7 @@ def shutdown(self) -> None: self._ws_thread.join(timeout=0.5) if self._transport: - self._transport.stop() + self._transport.stop(flush=flush) NullRunRuntime._instance = None logger.info("NullRun Runtime shutdown") diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index ad64c18..1734c82 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -807,8 +807,25 @@ def __exit__(self, exc_type, exc_val, exc_tb) -> None: except Exception as e: # noqa: BLE001 — best-effort on context exit logger.debug(f"Transport.__exit__: stop() raised: {e}") - def stop(self, timeout: float = 10.0) -> None: - """Stop background flush thread and flush remaining events.""" + def stop(self, timeout: float = 10.0, flush: bool = True) -> None: + """Stop background flush thread and flush remaining events. + + Args: + timeout: max seconds to wait for the flush thread to exit. + flush: when True (default) the final ``_do_flush()`` and + ``_persist_to_wal()`` run after the thread joins — the + production "drain on the way out" contract. When + False, the thread is cancelled but the buffer is left + alone. The test conftest uses ``flush=False`` to + teardown between tests without a final httpx call — + in tests the respx context has already exited by the + time the conftest's teardown runs, so a final + ``_do_flush()`` would race respx and trigger a + ``ConnectError`` retry storm + (observed: 9m 47s of "Request failed (attempt N/11), + retrying in 10s" on PR #60, dominating the + otherwise-fast xdist wall clock). + """ self._running = False self._stopped = True # Mark as stopped to prevent double flush # Wake the flush thread out of its cancellable sleep so join() @@ -820,8 +837,9 @@ def stop(self, timeout: float = 10.0) -> None: self._stop_event.set() if self._flush_thread: self._flush_thread.join(timeout=timeout) - self._do_flush() # Final flush - self._persist_to_wal() # WAL any remaining events + if flush: + self._do_flush() # Final flush + self._persist_to_wal() # WAL any remaining events self._client.close() # Detach the weakref finalizer — stop is the canonical # "I am done" path. After this point the finalizer will diff --git a/tests/conftest.py b/tests/conftest.py index c6d63ff..89bc842 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -42,8 +42,24 @@ def reset_runtime(): yield - # Just clear references, don't call shutdown which may try HTTP calls - # after respx mock context has already exited + # Stop any running transport flush thread BEFORE we drop the + # reference. Without this the thread keeps running across tests, + # the buffer drains through httpx with no respx context active, + # and the worker logs a ``ConnectError`` retry storm for the rest + # of the xdist session — observed 9m 47s of "Request failed + # (attempt N/11), retrying in 10s" on PR #60, which dwarfed the + # actual test time. ``flush=False`` skips the final ``_do_flush`` + # / ``_persist_to_wal`` so the teardown is a true no-op even when + # the buffer still has events; the test that wrote them is + # responsible for asserting on what it cared about. Best-effort: + # the runtime may be in any state at teardown, and we don't want + # a flaky shutdown to mask the real test failure that just ran. + inst = NullRunRuntime._instance + if inst is not None: + try: + inst.shutdown(flush=False) + except Exception: + pass NullRunRuntime._instance = None _dec._runtime = None _act._action_handler = None diff --git a/tests/test_init_contract.py b/tests/test_init_contract.py index 78c1736..e904419 100644 --- a/tests/test_init_contract.py +++ b/tests/test_init_contract.py @@ -16,6 +16,7 @@ from __future__ import annotations import threading +import time import pytest @@ -316,3 +317,57 @@ def test_init_logs_debug_when_probe_raises( rt.shutdown() finally: _caps_mod.probe_capabilities = original_probe + + +class TestShutdownFlushKwarg: + """Regression pin for the PR #60 follow-up: ``shutdown(flush=...)`` + must propagate to ``Transport.stop(flush=...)`` so the test + conftest can teardown between tests without racing the respx + context exit. Pre-this-pin, the conftest's teardown just nulled + the runtime reference; the transport flush thread kept running + with a non-empty buffer, the next ``_do_flush`` raced respx and + hit the real network, and CI logged 9m 47s of + "Request failed (attempt N/11), retrying in 10s" — dominating + the otherwise-fast xdist wall clock. + """ + + def test_runtime_shutdown_flush_false_skips_final_flush(self, mock_api): + """``runtime.shutdown(flush=False)`` cancels the transport + thread WITHOUT triggering a final ``_do_flush()``. + + We use ``_test_mode=True`` so init skips auth, then buffer + an event directly into the transport (bypassing + ``track()``'s auth path), then call ``shutdown(flush=False + )`` AFTER the respx context has exited. The whole call must + return in well under 1s; a regression to + ``shutdown(flush=...)`` not propagating would push the + assertion past the 5s connect timeout × retry budget. + """ + # _test_mode skips auth but still starts the transport thread. + rt = NullRunRuntime( + api_key="test-key-12345678", + _test_mode=True, + polling=False, + ) + # Buffer an event so a final _do_flush() would have + # something to attempt to send. mock_api is a function- + # scoped fixture; we drop the reference so the respx + # context exits before we call shutdown. + rt._transport._buffer.append({"event_id": "x", "event": "test"}) + + started = time.monotonic() + rt.shutdown(flush=False) + elapsed = time.monotonic() - started + + assert elapsed < 1.0, ( + f"shutdown(flush=False) took {elapsed:.2f}s; expected " + f"<1s. The flush=False kwarg did not propagate to " + f"Transport.stop() — the conftest teardown regression " + f"is back." + ) + # And the buffer is left alone — the test that wrote it + # is responsible for asserting on what it cared about. + assert len(rt._transport._buffer) == 1, ( + f"shutdown(flush=False) should leave the buffer alone; " + f"expected 1 event, got {len(rt._transport._buffer)}." + ) diff --git a/tests/test_transport.py b/tests/test_transport.py index 2aa096e..d8d979a 100644 --- a/tests/test_transport.py +++ b/tests/test_transport.py @@ -111,6 +111,59 @@ def test_stop_interrupts_flush_sleep(self): f"cancellable-wait fix regressed." ) + def test_stop_flush_false_skips_final_flush(self): + """``stop(flush=False)`` cancels the thread WITHOUT a final + ``_do_flush()`` so the conftest can teardown between tests + without racing the respx context exit. + + Regression pin for the second CI-noise fix (PR #60 follow-up): + the conftest previously nulled the runtime reference without + calling ``shutdown()`` so the transport flush thread kept + running with a non-empty buffer; on the next ``_do_flush`` + (after respx exited) httpx hit the real network, got + ``ConnectError``, retried 11 times with up-to-10s backoff, + and dominated the xdist wall clock (9m 47s of + "Request failed (attempt N/11), retrying in 10s"). + + The contract being pinned here: with ``flush=False``, + ``_do_flush`` is NOT called from ``stop()`` even when the + buffer is non-empty. The teardown is a true no-op apart + from the thread join. + """ + from nullrun.transport import FlushConfig + + t = Transport( + api_url="https://api.test.nullrun.io", + api_key="test-key-12345678", + config=FlushConfig(flush_interval=30.0), + ) + t.start() + # Buffer an event so a final _do_flush() would have something + # to attempt to send (and therefore would race respx). + t._buffer.append({"event_id": "x", "event": "test"}) + # No respx mock active here — if stop() tries to flush, httpx + # will block for the 5s connect timeout per attempt and + # multiply by the retry budget. The whole point of + # ``flush=False`` is to skip that path entirely. + started = time.monotonic() + t.stop(flush=False) + elapsed = time.monotonic() - started + # Generous bound: thread join is the only blocking step. A + # regression to "stop() always flushes" would push this + # past 60s on the first failure. + assert elapsed < 1.0, ( + f"stop(flush=False) took {elapsed:.2f}s; expected < 1s. " + f"The final _do_flush() ran despite flush=False — the " + f"conftest teardown is back to racing respx and the " + f"CI retry-storm regression is open again." + ) + # And the buffer is left alone — the conftest contract is + # "we don't care, the test that wrote it is responsible". + assert len(t._buffer) == 1, ( + f"stop(flush=False) should leave the buffer untouched; " + f"expected 1 event, got {len(t._buffer)}." + ) + def test_ssl_verification_enabled(self, transport): # httpx 0.28+ doesn't expose verify as a direct attribute # SSL verification is enabled by default (verify=True)