test: bind moto server to an ephemeral port to fix xdist deadlock#4139
Merged
Conversation
…#176) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@57e3a13...e79a696) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](j178/prek-action@6ad8027...bdca6f1) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v7...043fb46) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v7...3e5f45b) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The session-scoped moto_server fixture bound a fixed port (5555). Under pytest-xdist, session-scoped fixtures run once per worker, so concurrent workers raced to bind the same port. The losers blocked forever inside ThreadedMotoServer.start(): its werkzeug thread dies on "Address already in use" before setting the ready event that start() waits on, and the wait has no timeout. Any multi-worker run whose s3 tests landed on 2+ workers deadlocked near the end of the suite at 0% CPU. CI never saw this because it runs the suite single-process. Bind port 0 instead and yield the actual endpoint from the fixture. Since the endpoint is now only known at runtime, test_fsspec.py's module-level endpoint_url constant becomes an endpoint_url fixture (replacing the s3_base alias), and its consumers take it as a parameter. With this fix the full suite completes in ~1 minute under -n auto on a 10-core machine, where it previously hung indefinitely. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Contributor
Author
|
this is a small test-time-scoped quality-of-life improvement to our test suite that does not affect public code. I will self-merge. |
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a claude-authored PR that changes our S3 test fixture to use an ephemeral port managed at runtime by moto itself. I discovered this while commissioning claude to find new pytest features that we can use to improve our test suite.
🤖 AI text below 🤖
Problem
Running the full suite locally with
pytest -n autodeadlocks near the end (~98%), with every xdist worker parked at 0% CPU. It has been this way since the shared moto fixture landed (#4018, 2026-06-02).Root cause: the session-scoped
moto_serverfixture intests/conftest.pybound a fixed port (MOTO_SERVER_PORT = 5555). Under pytest-xdist, session scope means once per worker, so when two or more workers reach an S3 test concurrently they race to bind the same port. The first wins; every loser blocks forever inThreadedMotoServer.start()— its werkzeug thread dies on "Address already in use" before ever setting the readyEventthatstart()waits on (no timeout).Minimal repro: construct two
ThreadedMotoServer(port=5555)and callstart()on both — the second never returns.CI never sees this because
run-coverageruns the suite single-process, and the one-nautoCI target (run-hypothesis) doesn't touch S3.Fix
moto_serverbindsport=0(ephemeral) and yields the actual endpoint fromserver.get_host_and_port(). This also makes concurrent pytest sessions on one machine safe, not just concurrent workers.test_fsspec.py's module-levelendpoint_urlconstant becomes anendpoint_urlfixture (absorbing the olds3_basealias), and its consumers take it as a parameter.tests/test_docs.pyalready consumed the fixture value, so it needs no changes.Verification
pytest tests/test_store/test_fsspec.py -n 4: 102 passed in 8.8s (previously a guaranteed deadlock shape — 4 workers, 4 servers).pytest --ignore tests/benchmarks -n auto: 6196 passed, 277 skipped, 4 xfailed in 55.54s on a 10-core M-series machine, where it previously hung indefinitely.__psynch_cvwait;kill -ABRTproduced faulthandler dumps pointing atEvent.wait←ThreadedMotoServer.start←moto_serverfixture setup.🤖 Generated with Claude Code