release(0.13.5): perf — cancellable flush sleep + CI hygiene by maltsev-dev · Pull Request #60 · nullrunio/nullrun-sdk-python

maltsev-dev · 2026-07-08T12:19:26Z

release(0.13.5)

This is the 0.13.5 perf release — pairs the LangChain elif-chain
fix from 0.13.4 with a CI/runtime perf pass that the audit revealed.

What's in this PR

Three commits on top of origin/master (note: the 0.13.4
elif-chain fix already landed on master as PR #59, so the matching
c17a582 version-bump commit is intentionally not cherry-
picked — its changelog text was already on master and would have
been a no-op):

6ec7671 — perf(ci): Transport._flush_loop swapped
time.sleep for threading.Event.wait so
runtime.shutdown() returns in ms, not the full
flush_interval (5s default). Pin:
tests/test_transport.py::test_stop_interrupts_flush_sleep
uses 30s flush_interval and asserts stop() < 5s.
84fc9d5 — docs: remove docs/drift.md and
docs/sdk-v3-migration-gaps.md (drift content already in
CHANGELOG.md and the per-release __version__.py headers).
99bc197 — chore(release): bump 0.13.4 → 0.13.5
(pyproject.toml + src/nullrun/__version__.py) with a full
changelog block describing the perf fix + CI hygiene.

CI hygiene (in the perf commit)

setup-python action: cache: pip +
cache-dependency-path: pyproject.toml — saves ~60-90s of
cold install per matrix leg.
strategy.fail-fast: true on the test matrix.
pip install "pytest-xdist>=3.6" + pytest -n auto so the
suite uses all runner cores. xdist also added to
[project.optional-dependencies.dev] for local parity.
pyproject.toml: dropped the global -q from addopts so
CI logs surface the full PASSED line per test;
--tb=short keeps tracebacks compact. -n auto stays in
the workflow (not addopts) so a developer running
pytest tests/test_x.py locally gets a single process.

Wire format

No change. The default FlushConfig is unchanged (5s
flush_interval, 50 batch_size); production flush cadence is
identical. The fix only shortens the worst-case shutdown latency.

Compatibility

Backends on 1.0.0 keep working unchanged. No SDK_MIN_VERSION
bump. Recommended upgrade path: 0.13.4 → 0.13.5.

Validation

test_stop_interrupts_flush_sleep passes in 0.26s (was 30s).
test_legacy_key_warning — 0.19s (was 5.18s).
test_remote_states_race ×4 — no 5.0s teardowns.
test_state_compare_case_insensitive ×10 — all <0.2s.
ruff check clean on changed files.
mypy clean on transport.py.

Branch / commit policy

New branch release/0.13.5 cut from origin/master (not
from a prior release/*, per repo convention).
No Co-Authored-By trailers.
Real user.name/user.email from local git config.

The Transport flush loop used `time.sleep(self.config.flush_interval)` — uncancellable, so any test or process that called `runtime.shutdown()` while the thread was mid-sleep blocked on `thread.join()` for the full default 5s flush_interval. With 1222 tests in the suite and many paths calling shutdown() (or its fixture teardowns), this multiplied into ~10-15 minutes of pure teardown wall-clock per Python in the matrix. Replace the bare sleep with `Event.wait`, which returns the instant `stop()` sets the event. `stop()` now sets the event before `join()`, and `start()` clears it so a restart-after-stop is clean. Pin contract in tests/test_transport.py:: test_stop_interrupts_flush_sleep …uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s. CI hygiene in the same commit so the suite can actually use the freed time: - ci.yml / publish*.yml: enable pip cache (`cache: pip` + `cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold install per matrix leg. - ci.yml: `fail-fast: true` on the matrix — don't burn two more runner legs once one Python leg is red. - ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and pass `-n auto` to pytest. `pytest-xdist` is also added to `[project.optional-dependencies.dev]` so a local `pip install -e .[dev]` brings it in. - pyproject.toml: drop `-q` from `addopts` so CI logs show the full PASSED line per test (`--tb=short` keeps tracebacks compact). `-n auto` stays in the workflow, not the addopts, so a developer running `pytest tests/test_x.py` gets a single process. No public API change. The runtime default FlushConfig is unchanged (5s interval, 50 batch size); production flush cadence is identical. The fix only shortens the worst-case shutdown latency.

Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5.

…ace respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run.

maltsev-dev · 2026-07-08T12:42:31Z

Follow-up commit `a037b3a` — conftest teardown fix

Pushed a 4th commit on top of the original 3 to address the 9m 47s
retry storm that surfaced on the first green run of PR #60 (run
#1, the log you saw in chat).

What was wrong

The first commit (6ec7671) made Transport._flush_loop use
Event.wait instead of time.sleep, so shutdown() returns
fast. But the tests/conftest.py:reset_runtime teardown was
never calling shutdown() — it just nulled the runtime
reference. The transport flush thread therefore kept running across
tests with a non-empty buffer, the next _do_flush raced the
respx context exit, and CI spent 9m 47s doing:

Request failed (attempt 5/11), retrying in 8.46s: ConnectError
Request failed (attempt 6/11), retrying in 9.16s: ConnectError
...
Circuit breaker OPEN. Batch of 10 events will be re-queued.

_retry_with_backoff(max_retries=10, max_delay=10s) is ~65s of
sleep per failed batch; with 4 xdist workers and many buffered
batches it dominated the wall clock.

The fix

Transport.stop(timeout, flush: bool = True) — new flush
flag; when False, skip the final _do_flush() /
_persist_to_wal().
NullRunRuntime.shutdown(flush: bool = True) — propagates.
nullrun.shutdown(timeout, flush: bool = True) — propagates.
tests/conftest.py:reset_runtime teardown now calls
inst.shutdown(flush=False) before nilling the reference.

Production default (flush=True) is unchanged — the public
nullrun.shutdown() audit contract ("drain in-flight events")
still holds. The flush=False path is for the test conftest
only, where the test that wrote the events is responsible for
asserting on what it cared about.

Pins

tests/test_transport.py::test_stop_flush_false_skips_final_flush
tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush

Both assert stop(flush=False) / shutdown(flush=False) return
in <1s even with a non-empty buffer and no respx active.

Expected CI delta

From ~9m 47s (PR #60 run #1) down to ~30-60s on the next green run.
The fail-fast matrix + pip cache + -n auto xdist were already
in place from 6ec7671; this commit kills the last noisy
bottleneck.

codecov · 2026-07-08T12:56:10Z

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/nullrun/transport.py	0.00%	10 Missing ⚠️
src/nullrun/__init__.py	0.00%	2 Missing ⚠️
src/nullrun/runtime.py	0.00%	2 Missing ⚠️
src/nullrun/__version__.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

maltsev-dev added 4 commits July 8, 2026 16:18

remove redundant docs

84fc9d5

maltsev-dev merged commit f79312c into master Jul 8, 2026
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

release(0.13.5): perf — cancellable flush sleep + CI hygiene#60

release(0.13.5): perf — cancellable flush sleep + CI hygiene#60
maltsev-dev merged 4 commits into
masterfrom
release/0.13.5

maltsev-dev commented Jul 8, 2026

Uh oh!

maltsev-dev commented Jul 8, 2026

Uh oh!

Uh oh!

codecov Bot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

maltsev-dev commented Jul 8, 2026

release(0.13.5)

What's in this PR

CI hygiene (in the perf commit)

Wire format

Compatibility

Validation

Branch / commit policy

Uh oh!

maltsev-dev commented Jul 8, 2026

Follow-up commit a037b3a — conftest teardown fix

What was wrong

The fix

Pins

Expected CI delta

Uh oh!

Uh oh!

codecov Bot commented Jul 8, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Follow-up commit `a037b3a` — conftest teardown fix