ENH: igenerator pkl correctness fixes + CastXML content-addressed cache (supersedes #6486)

[hjmjohnson]

Cuts Python-wrapping CI build time by ~31 min per run on 2-core Azure agents (fleet median 175 min → 144 min, −18%; best same-code cold→warm pair −20%) and by up to −64% on GHA ubuntu-24.04 combined with warm ccache — via two latent pkl-tracking correctness fixes plus a two-level content-addressed CastXML cache. Populating the cache is free (cold runs match the fleet median). Validated by a 49-test cross-platform pytest suite wired in as a WrappingInfrastructure ctest. Supersedes #6486.

Measured CI performance — this PR's own builds (AzDO ITK.Linux.Python, 2-core)

Fleet baseline: the 32 most recent standard builds on this pipeline (other PRs + main, warm ccache, no CastXML cache) have median 174.9 min (range 158.7–235.8).

Build class	Builds	Duration
Standard (fleet median, n=32)	—	174.9 min
This PR, cache cold / populating	16147, 16173	177.1, 179.0 min
This PR, cache warm	16158, 16160, 16181	143.8, 145.0, 142.5 min

• Steady-state saving: ~31 min (−18%) vs fleet median; warm runs are also far more consistent (2.5 min spread vs 77 min fleet spread).
• Best controlled measurement: builds 16173 → 16181, identical source one day apart: 179m00s → 142m29s = −36m31s (−20%).
• Cache-population overhead is free — cold runs land at the fleet median.
• Build 16181 breakdown: configure 51s / build 131m / test 6m; ccache hit rate 99.97% — the CastXML cache removes the wrapping-generation time that ccache cannot touch.

On 4-core GHA ARM runners the total is flat (±1 min) as expected: with enough cores CastXML is not on the critical path. The cache targets core-count-limited runners.

Measured CI performance — prototype runs (PR #6486, GHA + AzDO)

GHA ubuntu-24.04 (2-core, combined CastXML cache + ccache cold→warm):

Phase	Cold (2026-06-28)	Warm (2026-06-29)	Δ
CastXML (816 jobs)	31m 45s	23 sec	−98%
Total	5h 1m 52s	1h 49m 50s	−64%

AzDO ITK.Linux.Python — CastXML cache isolated (ccache warm on both sides):

	Cold (16132)	Warm (16133)	Δ
Build and test	167m 54s	132m 15s	−35m 39s
Total	2h 52m 46s	2h 20m 23s	−32m 23s (−19%)

72-core local workstation (NVMe, ccache warm): warm speedup ~4% — CastXML parallelizes off the critical path; the cache is for CI-class machines.

What this fixes (correctness)

Ninja graph gap — igenerator.py writes one .pkl file per wrapped
class (~816 files per full ITK build) into itk-pkl/ as side effects.
None of those files appear as CMake OUTPUTs; only the .index.txt
manifests do. When itk-pkl/*.pkl files are absent (fresh clone, partial
ninja -t clean, CI agent reusing a workspace with a stale build tree)
while the .index.txt files survive, ninja considers igenerator.py
up-to-date. pyi_generator.py then fails with:

No pickle files were found in the pkl directory

Fix (commit 1): a per-module <Module>.stamp file is written by
igenerator.py after all pkl work for that module completes and is
declared as a CMake OUTPUT. Ninja now reruns igenerator.py whenever
the stamp is absent, guaranteeing the pkl files exist before
pyi_generator.py runs.

Parallel write race — the ~96 concurrent igenerator.py jobs (one per
ITK module) all write individual .pkl files into the same itk-pkl/
directory with no locking. Under adverse scheduling this produces torn
reads or silently truncated files.

Fix (commit 2): replace the directory of per-class .pkl files with a
single WAL-mode SQLite database. SQLite WAL mode is designed for
concurrent writers; the race is structurally eliminated. The .index.txt
manifests now hold DB keys (not file paths), and pyi_generator.py
queries the DB via those keys, self-healing if indexed keys are missing.

What this adds (performance)

Commit 3 defers import pygccxml until after the submodule list is
built from the .mdx files, separating the pure-I/O discovery phase from
the pygccxml/CastXML-dependent processing phase.

Commit 4 adds itk-castxml-cache.py, a transparent wrapper for the
CastXML binary with a two-level content-addressed cache:

• L1 (no subprocess, ~0.1 s): SHA-256 of CastXML binary content +
  .castxml.inc + .cxx content + flags → manifest with L2 key, a
  ccache-style dependency list, and an include-dir fingerprint that
  detects header shadowing.
• L2 (preprocessor pass, ~0.75 s): SHA-256 of castxml -E output with
  line markers stripped → output.xml.gz. Path-independent: any build
  directory on the same source tree produces the same L2 key.

Concurrency-safe (atomic staging + rename), size-capped with synchronous
eviction (ITK_WRAP_CACHE_MAX_SIZE, default 2 GB). Enable via
ITK_WRAP_CASTXML_CACHE (default ON); cache root override via
ITK_WRAP_CACHE (default ~/.cache/itk-wrap).

CI wiring: Cache@2 tasks in all three Azure DevOps Python pipelines and
GHA arm.yml; configure-python-ci / build-python-ci /
test-python-ci pixi tasks.

Validation suite (commit 6)

Wrapping/Generators/Tests/: 49 pytest tests covering cache key
composition, dependency invalidation, header shadowing, eviction,
concurrency (parallel writers), pkl DB integrity, pyi_generator
pruning/self-heal, and tar-relocation of the cache between build trees
(models actions/cache / Cache@2 restore). Runs in ~7 s using an
instrumented fake CastXML with exact subprocess-count assertions
(cold=2, L2-hit=1, L1-hit=0).

Registered as ctest WrappingInfrastructure when pytest is available
(graceful skip otherwise); pytest added to the four Python CI package
installs. The same suite runs standalone on
{ubuntu-24.04, macos-14, windows-2022} × {Python 3.11, 3.13} — all six
legs green.

Why this supersedes #6486

PR #6486 (ci/linux-azure-disk-management) was a work-in-progress branch
used to accumulate CI benchmark data. It contains temporary benchmarking
artifacts, a vestigial igenerator LRU cache that was added then removed,
benchmark key-version bumps, and two files already merged separately
(#6530, #6531). This PR is a clean rewrite from upstream/main with the
same final implementation in six independently-reviewable commits.

[greptile-apps]

Greptile Summary

This PR updates ITK's Python wrapping pipeline and adds a content-addressed CastXML cache. The main changes are:

• Build-local SQLite storage for pkl metadata used by pyi_generator.py.
• Per-module pkl stamp outputs so the build graph reruns igenerator.py when pkl state is missing.
• A two-level CastXML cache wrapper with CI cache wiring and post-build eviction.
• Wrapping infrastructure pytest coverage for cache behavior, pkl DB behavior, concurrency, and relocation.
• Contributor documentation for the Python wrapping architecture.

Confidence Score: 4/5

Merge safety is good overall, but the CastXML cache include-directory parsing needs attention before relying on warm-cache correctness across all generated response-file forms.

The changes are well-scoped and come with targeted wrapping infrastructure tests, but the cache invalidation gap can return stale XML when system include directories gain shadowing headers.

Wrapping/Generators/CastXML/itk-castxml-cache.py

T-Rex Logs

What T-Rex did

• I reproduced the -isystem cache invalidation scenario by running a Python harness that drives itk-castxml-cache.py with a fake CastXML and a quoted -isystem response file, observing the first run populating the cache from the lower-priority -I header and a second run producing an L1 HIT after shadow.h was added.
• I validated the pkl store behavior by comparing base and head states, noting base behavior with no DB and a missing pkl warning, versus head behavior with a WAL-enabled database and a 320-file pkl footprint and 320 WAL rows.
• I confirmed the head-state CastXML cache integration and relocation coverage, including a cache wrapper, relocation tests, concurrency eviction, and WrappingInfrastructure entrypoints as shown in the head artifacts.

_{Ran code and verified through T-Rex}

_{Reviews (2): Last reviewed commit: "ENH: Add validation suite for wrapping c..." | Re-trigger Greptile}

[hjmjohnson]

Yes — this file is the core deliverable of the PR.

itk-castxml-cache.py is a transparent two-level content-addressed wrapper that replaces the bare castxml invocation for all 816 wrapping jobs. CMake/itkWrapCastXMLCacheSupport.cmake points ITK_WRAP_CASTXML_CACHE_SCRIPT at it; Wrapping/macro_files/itk_auto_load_submodules.cmake:207–215 substitutes it for castxml when ITK_WRAP_CASTXML_CACHE=ON. On a warm cache it skips all 816 CastXML subprocesses, saving ~32 min per AzDO Python CI run (−19%) and ~3h 12m on GHA ubuntu-24.04 combined with ccache (−64%).

Documentation/docs/contributing/wrapping_architecture.md has the full picture (Step 1 and the "CastXML cache" section under Caches).

[hjmjohnson]

@dzenanz I'm running this through some stress tests, and trying to simplify it a little bit (I've tried to squeeze small performance increases, but added more complexity for some of the added features). I'm trying to find where the ROI on complexity is worth it. So far big wins come from the castxml cache ( 30+ minutes saved on many CI runs), and preliminary indications are that using a simple SQL db for data store rather than 1000's of small files is going to be a win as well.

This is not ready for review yet...Sorry.

[hjmjohnson]

some minor documentation generalizations.

[hjmjohnson]

@dzenanz @thewtex @blowekamp

FYI: I realize that everyone is busy, and this may not be a good time to review a change to the wrapping. If there are too many concerns with this PR or insufficient resources, let's shelve it and revisit it after other initiatives settle down.

I feel that a 20-64% decrease in CI build times is a good improvement, but this might not be the right time to try to introduce the caching of castxml.

[dzenanz]

I like this, but I leave it to you, Brad, and Matt to decide.