test: enable pytest strict mode, fix duplicate parametrization ids, and add 10m timeout to individual tests#4140
Open
d-v-b wants to merge 19 commits into
Open
Conversation
…#176) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@57e3a13...e79a696) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](j178/prek-action@6ad8027...bdca6f1) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v7...043fb46) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v7...3e5f45b) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The session-scoped moto_server fixture bound a fixed port (5555). Under pytest-xdist, session-scoped fixtures run once per worker, so concurrent workers raced to bind the same port. The losers blocked forever inside ThreadedMotoServer.start(): its werkzeug thread dies on "Address already in use" before setting the ready event that start() waits on, and the wait has no timeout. Any multi-worker run whose s3 tests landed on 2+ workers deadlocked near the end of the suite at 0% CPU. CI never saw this because it runs the suite single-process. Bind port 0 instead and yield the actual endpoint from the fixture. Since the endpoint is now only known at runtime, test_fsspec.py's module-level endpoint_url constant becomes an endpoint_url fixture (replacing the s3_base alias), and its consumers take it as a parameter. With this fix the full suite completes in ~1 minute under -n auto on a 10-core machine, where it previously hung indefinitely. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Adopt pytest 9's `strict = true` config option, which enables
strict_config, strict_markers, strict_xfail, strict_parametrization_ids,
and future strictness options. This replaces the --strict-config and
--strict-markers addopts flags (which pytest silently ignored when
passed via addopts before 9.1, pytest#14442) and the separate
xfail_strict setting. minversion is bumped to 9 accordingly.
strict_parametrization_ids surfaced seven parametrizations whose ids
collided, which pytest previously deduplicated silently:
- test_common.py paired float("inf") with np.inf (and -inf/nan), but
np.inf IS a Python float, so three cases per test ran twice and the
intended numpy-scalar coverage did not exist. Use np.float64 scalars
with explicit ids, restoring the intended coverage.
- test_int.py deliberately includes np.dtype("i")/np.dtype("I"),
which compare equal to one of the explicit-endian dtypes and
stringify identically. The class-fixture hook in test_dtype/conftest
now suffixes colliding ids with their index instead of relying on
pytest's silent deduplication.
- test_group.py test_consistent_signatures id'd both create_hierarchy
pairs identically; ids now include module and qualname.
- test_array.py ("a" vs b"a"), test_vlen.py (str vs "str"), and
test_metadata/test_v2.py (3 vs "3") get explicit ids for the
colliding member.
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Set faulthandler_timeout=600 with faulthandler_exit_on_timeout (new in pytest 9): if a single test exceeds 10 minutes, every thread's traceback is dumped and the run is killed, instead of hanging indefinitely with no diagnostics. Motivated by the moto fixed-port deadlock, which hung local -n auto runs at ~98% for hours and required manual kill -ABRT forensics to diagnose; with this watchdog the same bug would have failed in 10 minutes with the offending stack in the log. The ceiling is ~30x the slowest legitimate test (~20s locally) to stay clear of slow CI runners, coverage overhead, and nightly stateful hypothesis runs. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
changed the title
Contributor
Author
|
I'm going to self-merge, since this is a purely internal set of changes that target the test suite. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR updates our pytest configuration to use strict mode, which was added here and was released with pytest 9.0. It turns on
strict_config,strict_markers,strict_xfail, andstrict_parametrization_ids.Strict mode checks for issues like multiple parameters in
pytest.parametrizeinvocations that use the sameidfield, which caught several problems in our codebase. They are fixed in this branch by explicitly creating an instance ofpytest.param(id=...)instead of relying on automaticidgeneration.Edit: this also adds a timeout of 600s to any single test, and any test that triggers this timeout triggers a dump of the traceback of all threads. This would have resolved the s3 test fixture deadlocks resolved in #4139.
These changes were written by claude: see d-v-b#215 for claude's output.
Author attestation
AI coding assistance is welcome, but a human must be the author and is responsible for the contents of the PR. The description and any review responses must be in your own words. Please read AI-assisted contributions before opening.
TODO
docs/user-guide/*.mdchanges/