Skip to content

test: bind moto server to an ephemeral port to fix xdist deadlock#4139

Merged
d-v-b merged 17 commits into
zarr-developers:mainfrom
d-v-b:claude/moto-ephemeral-port
Jul 10, 2026
Merged

test: bind moto server to an ephemeral port to fix xdist deadlock#4139
d-v-b merged 17 commits into
zarr-developers:mainfrom
d-v-b:claude/moto-ephemeral-port

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

This is a claude-authored PR that changes our S3 test fixture to use an ephemeral port managed at runtime by moto itself. I discovered this while commissioning claude to find new pytest features that we can use to improve our test suite.

🤖 AI text below 🤖

Problem

Running the full suite locally with pytest -n auto deadlocks near the end (~98%), with every xdist worker parked at 0% CPU. It has been this way since the shared moto fixture landed (#4018, 2026-06-02).

Root cause: the session-scoped moto_server fixture in tests/conftest.py bound a fixed port (MOTO_SERVER_PORT = 5555). Under pytest-xdist, session scope means once per worker, so when two or more workers reach an S3 test concurrently they race to bind the same port. The first wins; every loser blocks forever in ThreadedMotoServer.start() — its werkzeug thread dies on "Address already in use" before ever setting the ready Event that start() waits on (no timeout).

Minimal repro: construct two ThreadedMotoServer(port=5555) and call start() on both — the second never returns.

CI never sees this because run-coverage runs the suite single-process, and the one -nauto CI target (run-hypothesis) doesn't touch S3.

Fix

  • moto_server binds port=0 (ephemeral) and yields the actual endpoint from server.get_host_and_port(). This also makes concurrent pytest sessions on one machine safe, not just concurrent workers.
  • Since the endpoint is now only known at runtime, test_fsspec.py's module-level endpoint_url constant becomes an endpoint_url fixture (absorbing the old s3_base alias), and its consumers take it as a parameter. tests/test_docs.py already consumed the fixture value, so it needs no changes.

Verification

  • pytest tests/test_store/test_fsspec.py -n 4: 102 passed in 8.8s (previously a guaranteed deadlock shape — 4 workers, 4 servers).
  • Full suite pytest --ignore tests/benchmarks -n auto: 6196 passed, 277 skipped, 4 xfailed in 55.54s on a 10-core M-series machine, where it previously hung indefinitely.
  • Diagnosis receipts: sampling the hung workers showed all threads in __psynch_cvwait; kill -ABRT produced faulthandler dumps pointing at Event.waitThreadedMotoServer.startmoto_server fixture setup.

🤖 Generated with Claude Code

dependabot Bot and others added 15 commits May 31, 2026 19:28
…#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The session-scoped moto_server fixture bound a fixed port (5555). Under
pytest-xdist, session-scoped fixtures run once per worker, so concurrent
workers raced to bind the same port. The losers blocked forever inside
ThreadedMotoServer.start(): its werkzeug thread dies on "Address already
in use" before setting the ready event that start() waits on, and the
wait has no timeout. Any multi-worker run whose s3 tests landed on 2+
workers deadlocked near the end of the suite at 0% CPU. CI never saw
this because it runs the suite single-process.

Bind port 0 instead and yield the actual endpoint from the fixture.
Since the endpoint is now only known at runtime, test_fsspec.py's
module-level endpoint_url constant becomes an endpoint_url fixture
(replacing the s3_base alias), and its consumers take it as a
parameter.

With this fix the full suite completes in ~1 minute under -n auto on a
10-core machine, where it previously hung indefinitely.

Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
@d-v-b d-v-b changed the title Claude/moto ephemeral port test: bind moto server to an ephemeral port to fix xdist deadlock Jul 10, 2026
@d-v-b d-v-b marked this pull request as ready for review July 10, 2026 13:49
@d-v-b

d-v-b commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

this is a small test-time-scoped quality-of-life improvement to our test suite that does not affect public code. I will self-merge.

@d-v-b d-v-b merged commit 13279ca into zarr-developers:main Jul 10, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant