HyperTools 1.0: architecture refactor + bug hunt

[jeremymanning]

HyperTools 1.0 — architecture refactor + bug hunt

This PR merges the completed HyperTools 1.0 class-based refactor into dev-1.0, together with a broad bug hunt (open-issue triage + two animation-rendering fixes you reported + a legend-clipping fix) and the CI fixes needed to get all platforms green.

Note: originally developed under the working name "HyperTools 2.0"; renumbered to 1.0 (package version 1.0.0.dev0, branches dev-1.0/dev-1.0-refactor) per project decision. Older commit messages/notes may still say 2.0.

⚠️ Please do not merge. Opened for your review; I'll proceed however you decide. Base: dev-1.0 ← Head: dev-1.0-refactor.

Supersedes #271, which GitHub auto-closed when its head branch was renamed dev-2.0-refactor → dev-1.0-refactor; all commits, evidence, and CI history carry over unchanged.

---

1. Architecture refactor (Plans 1–8)

Reorganized the working dev-1.0 code into the class-based structure, on modern deps, with DataGeometry removed from the public API:

• Deps modernized: numpy ≥2, pandas ≥3, scikit-learn ≥1.4, matplotlib ≥3.8, plotly 6.x, via datawrangler 0.5 (funnel/stack/unstack/format-detection/model-dispatch). Supports pandas 3.0+.
• Class-based modules: core (eval-free apply_model), external (vendored PPCA + brainiak SRM/DetSRM/RSRM), manip (Normalize/ZScore/Smooth/Resample), reduce/align/cluster (generic sklearn dispatch by name), io, plot (matplotlib + plotly backends). Old tools/ names remain as shims.
• DataGeometry removed from the public API (kept only as a hidden unpickle shim so legacy hosted .geo datasets still load). plot() returns a Figure / (fig, ani); plot(..., return_model=True) returns {'fig','xform_data','models','animation'}; load() returns raw data.
• Docs/gallery/tutorials migrated to the 1.0 API and rebuilt.

---

2. Reported animation fixes

Animated bounding box "crowded / cut off on the right." Animated 3D plots zoomed the cube in too far. Fix: set_box_aspect zoom 1.25→1.125 + full-canvas axes. Min cube-to-edge margin over the full save_movie rotation: right 80→96px, bottom 51→72px.

before (crowded)	after (margin)

Duplicate animation-legend entries (GH #207). Faint "trail" artists carried the dataset label, so each label appeared twice. Fix: trails tagged _nolegend_; legend is now the static union of in-focus datasets. Verified ['first','second'].

Clipped static gallery legends. Wide legends (long labels / many entries) clipped off the right edge. Root cause: the fit routine measured the legend under seaborn's (narrower) font, but the figure is saved downstream under the default (wider) font. Fix: _fit_right_legend now measures the rasterized pixels under default rcParams and widens the figure (keeping the plot's size) until the legend has a margin. Regenerated plot_legend/plot_PPCA/plot_missing_data; pixel-based regression test added.

---

3. Open-issue triage → close-on-merge

The triage below has since been fully executed (2026-07-07): all 45 addressed/obsolete issues are CLOSED with per-issue run-code evidence comments; the 22 remaining feature requests carry research comments + low/medium/high effort labels; 6 issues were migrated from the jeremymanning fork (#273–#278, fork tracker now empty); and 6 residual gaps found during re-verification were fixed on this branch (#94, #141, #199, #206, #209, #244). See the audit summary comment. Originally: triaged all 67 open issues against this branch with real repros (see notes/issues-to-close-on-merge.md): 31 addressed/obsolete + 16 fixed or implemented on this branch (incl. §5's #169/#132 and §6's seven features) = 47 to close on merge; 20 stay open (feature requests / design decisions), each documented.

Bugs fixed from triage (all regression-tested):

#	bug	fix
#259	import hypertools mutated global rcParams['pdf.fonttype']	removed import-time mutation; editable-PDF default set inside manage_backend's snapshot/restore
#223	get_proj crash on 2D labeled plots	guard get_proj; match annotate/update tuple shapes for 2D vs 3D
#146/#190	DBSCAN/MeanShift/OPTICS crash on n_clusters	inject n_clusters only when the model's signature accepts it; register those clusterers
#148	show=False leaked the figure into pyplot	plt.close(fig) for static figures (skipped when the user passed ax, and for animated figures, whose timer must stay alive)
#132	DataFrame columns consumed positionally across datasets	format_data aligns named columns BY NAME to the first dataset's order (warns on reorder); mismatched column sets raise a clear ValueError
#214	wiki-model docstring vs wiki_model key	docstring corrected
—	reduce=<class/instance> → UnboundLocalError	initialize model_params in the custom-estimator branch

---

4. CI fixes (get all platforms green)

The first CI run surfaced two platform issues (unrelated to the features above); both fixed:

• Windows (all Pythons) — collection error. datawrangler 0.5.0 evaluates os.getenv('HOME') at import time to build its data dir; HOME is unset on Windows, so os.path.join(None, …) crashed dw's (and hypertools') import. Fixed hypertools-side by setting HOME=expanduser('~') before importing dw; filed upstream as Import crashes on Windows: config datadir evals os.getenv('HOME') which is None data-wrangler#32, fixed in dw 0.5.1 (released; verified import datawrangler works with HOME unset). The pydata-wrangler pin is bumped to >=0.5.1; the one-line HOME guard stays as belt-and-suspenders for environments still on 0.5.0.
• macOS/Ubuntu Python 3.11+ — 3 tests. matplotlib 3.11 (Python 3.11+ only) resets a figure's canvas after plt.close() (the disabling the figure doesn't work as intended #148 fix), so tests/callbacks that read fig.canvas.renderer/buffer_rgba() failed. Fixed by guarding the renderer in update_position and rendering the affected tests through an explicit Agg canvas. (savefig after close still works, so users are unaffected.)
• Windows — animated-figure draw crash. On Windows the animate backend switch to TkAgg actually succeeds (headless Linux/mac fall back to Agg), so the disabling the figure doesn't work as intended #148 plt.close() destroyed the FuncAnimation's real Tk timer and any later draw of the returned figure crashed ('NoneType' object has no attribute 'start'). Animated figures are now exempt from the show=False close — the disabling the figure doesn't work as intended #148 complaint was about static figures, and animations need their timer alive for playback.
• Ubuntu 3.12 — screenshot-verification step. The screenshot harness discovered figures via plt.get_fignums(), which the disabling the figure doesn't work as intended #148 close empties; it now uses the figure(s) returned by plot() (13/13 cases pass).

---

5. New: hyp.predict + hyp.impute (resolves GH #169)

Two new modules in the established class-based style (base class + one file per model + funnel dispatcher), integrated into hyp.plot/hyp.analyze like cluster/align:

• hyp.predict(data, model=..., t=...) — timeseries forecasting: Kalman, GaussianProcess, AutoRegressor (any sklearn regressor, recursive multi-step), ARIMA, Laplace (skaters ensemble), Chronos (HuggingFace foundation model, real chronos-t5-tiny test). t follows Use Kalman filter to fill in missing data #169's spec (int steps, or a datetime on time-indexed data — including past-date truncation). One forecast per input dataset, same dimensions, continued index.
• hyp.impute(data, model=...) — missing data: PPCA (default; clean interface over the vendored implementation — format_data's fill now routes through it, behavior-preserving), SimpleImputer/KNNImputer/IterativeImputer, and Kalman, which fills rows where every feature is NaN — the exact gap Use Kalman filter to fill in missing data #169 describes (PPCA cannot).
• return_model=True on both → (result, fitted) matching apply_model's convention; the fitted model can be passed back as model= on new data and is applied without re-estimation (verified: fit on A, forecast/impute B).
• hyp.plot(data, predict='Kalman', t=30) overlays one dashed, low-opacity, same-color forecast tail per dataset (2D + 3D, both backends, no legend duplication, frame always contains the forecasts). impute= selects the missing-data model in the plot/analyze pipeline.
• Dependencies: new [predict] extra (pykalman, statsmodels, skaters) and [predict-hf] (chronos-forecasting); GaussianProcess/AutoRegressor/sklearn imputers work on the base install; friendly ImportErrors otherwise (fresh-venv verified). yfinance is not a dependency — the tutorial self-installs it.
• Tutorials (executed, real data):
  • stock_forecasting.ipynb — scrapes 2y of real Yahoo Finance prices for 4 tickers, backtests all models against a 30-day holdout with an honest MAE/MAPE table (spoiler: ARIMA/Kalman ≈ the naive baseline, as efficient-market theory predicts — the tutorial says so).
  • projectile_kalman.ipynb — a real NBA SportVU jump-shot arc (25 Hz optical tracking): Kalman imputation of 5 fully-occluded frames recovers them to RMSE 0.20 ft vs the recorded truth; forecasting the arc's final 20 frames from the first 30 lands within MAE 4.2 ft.
• Forecast direction fix (review follow-up): the GaussianProcess default kernel is now DotProduct + RBF + WhiteKernel — the old stationary RBF reverted forecasts to the training mean beyond the data (drift −0.026/pt vs observed +0.0019/pt: reversed); the linear term extrapolates trends (+0.002..+0.010/pt: continues). Before/after renders + measurements in this comment.
• Gallery: plot_predict (helical forecasts) + plot_impute (PPCA-vs-Kalman panels on the Use Kalman filter to fill in missing data #169 case); API reference sections added.

---

6. Seven long-standing feature requests (GH #95, #100, #108, #109, #127, #142, #177, #191)

All seven implemented in the 1.0 design language, in both rendering backends, each with numeric + screenshot evidence in this comment:

• support multiindex Pandas DataFrames #95 MultiIndex DataFrames → leaf trajectories + per-level group means (thicker/opaquer means per level, colors by top-level index, exactly per the issue's spec).
• colorbar option #100 colorbar=True/dict — continuous hue (same palette as the lines, real value range) and discrete groups (segmented, labeled); coexists with legends without clipping.
• triangular meshes #109 surface=True/dict — smooth, lit convex-hull surfaces (3D, Blinn-Phong + plotly vertex shading) and smooth filled shapes (2D), dict-controlled properties, animatable (per-frame hull recompute); the axes cube auto-expands to contain the hull. New animate_surface_morph gallery demo.
• support kernel density plots in 2d and 3d #108/enhancement: replace matplotlib with ipyvolume #191 density=True/dict — subtle per-group KDE clouds (off by default; 2D alpha-ramp, 3D iso-surfaces/go.Volume with small-cluster auto-boost); enhancement: replace matplotlib with ipyvolume #191's ipyvolume ask is superseded by the plotly backend.
• feature request: support different selections of bullettime, precog, and chemtrails for different data matrices #127 chemtrails/precog/bullettime accept per-dataset lists (mixed trail styles in one animation); spin/serial warn instead of silently ignoring.
• BUG: legend isn't updated when color kwarg is specified #142 original legend/color symptom already fixed by the refactor (verified); re-verification exposed colors=/linestyles=/markers= were silent no-ops without their singular twins — fixed.
• Add support for Google Drive and other easy-to-add filetypes to hyp.load #177 hyp.load completed: Drive large-file interstitial (verified against a real 498MB public file), .xlsx/.xls, Google Sheets→CSV, remote-pickle trust policy.

Every graphical feature was adversarially screenshot-reviewed by fresh agents across a {2D,3D}×{mpl,plotly}×{static,animated} grid; their findings drove 6 further fix commits (plotly WebGL surface artifacts, bounding-box containment, MultiIndex colorbars, legend fitting, 3D density visibility, trail-mode warnings). New gallery examples: plot_surface, plot_density, plot_colorbar, plot_multiindex, animate_trails_mix, animate_surface_morph.

Maintainer-feedback rounds (tight hulls + morph, constant rotation speed + plotly parity + lighting controls, axes-clipping + gif corruption + full-sample morphs, multibyte text support, GH #205): hulls now hug the observations (hull-hugging smoothing: post-Taubin pull-back to the hull; cube-cloud oversize 1.63×→1.13×, ≥99% containment; axes box sized from actual meshes incl. mid-morph union bound, both backends) and morphing is a first-class animation style — animate='morph' (Hungarian point-cloud morphs between datasets, tagged-list form for static backdrops) with per-segment rotations lists ([1, 0.25, 2, ...] = per hold/transition). Both shape-morph gallery demos collapsed to single hyp.plot calls. Morph rotation speed is constant (segment duration ∝ rotation count); plotly marker sizes are empirically calibrated to matplotlib (15.1px → 5.0px for markersize=6) and volumetric shading retuned to matplotlib's subtlety; every surface color/lighting/shading knob is verified effective on both backends (incl. new lightdir; silent no-op keys removed). The "cut off bounding box" report was traced to two real rendering bugs, both fixed: 3-D scene artists were clipped at matplotlib's aspect-shrunk square viewport (now unclipped in every animation path), and animation GIFs saved at reduced dpi were corrupted by a matplotlib writer resize through the interactive-backend window (saves now dpi-safe). Morph animations use every dataset's full sample set (duplicate-to-largest, duplicates hidden at holds; hold frames are a 100% pixel match to static plots). GH #205 fixed: full multibyte (CJK) text support in both backends — automatic covering-font detection (excluding placeholder fonts like LastResort) + a font= kwarg (family/path/FontProperties) applied to labels/legends/colorbars/titles; plotly also gained real labels= annotations (was a silent no-op); CI provisions CJK fonts on all platforms with anti-tofu pixel tests.

---

7. Round 17 — every non-deferred open issue addressed (GH #103 #116 #123 #130 #138 #153 #154 #159 #161 #162 #174 #187 #198 #227 #273 #274 #275 #276 #277 #278)

Following the per-issue "current status (1.0 update)" triage, this round implements all 20 non-deferred open issues (the 5 marked defer/don't-implement were left untouched). Highlights: a unified cross-module API (every dispatcher takes manip=/normalize=/reduce=/align=/cluster=/return_model=, canonical order documented as a flowchart in docs/pipeline_order.rst) + public hyp.Pipeline with pipeline= reuse (#138 #153 #227 #161 #174); manip list-chaining and the story-trajectories animation — animate='window'/focused=/duration= (#274 #275, post-align inter-subject correlation −0.004→0.33, jumps 18.96→0.73); label_alpha=/axis labels/animate= dict/2-D animations (#103 #154 #123); six torch autoencoder reducers, sklearn/seaborn/538/kaggle loaders, LSL streaming, gensim text wrappers (#162 #273 #116 #130 #198, all opt-in extras); and a full docs pass — docstring coverage 131→0 with an enforcement test, complete api.rst, 7 new tutorials, regenerated README media (#276 #278 #159 #187 #277).

Executed as 19 review-gated tasks; the reviews caught and fixed five real reuse-contract bugs (silent pipeline refit, Aligner/SRM/Resample transform replaying fit-time data). Per-issue evidence comments (code + exact new API + numeric/screenshot proof) are posted on all 20 issues; see the round-17 summary comment for details.

---

Testing

• Local suite: 1271 passed, 0 failed (+492 tests across all rounds: surface/density/colorbar/multiindex/trails/meshutil/load/color-alias/morph-animation/hull-tightness suites); 6 plotly→kaleido image-export tests deselected locally only — they deadlock Chromium in this sandbox but run fine in CI.
• CI: all 12 jobs green (ubuntu/macos/windows x Python 3.10-3.13) on head 8c40499a -- run 28903254424
• Docs rebuilt (make html succeeds); gallery regenerated with the 6 new example pages (including two captured animations).

🤖 Generated with Claude Code

[jeremymanning]

Round 17 — every non-deferred open issue addressed

Following your per-issue "current status (1.0 update)" comments, this round implements all 20 non-deferred open ContextLab issues (the 5 you marked defer/don't-implement — #111, #112, #113, #163, #225 — were left untouched). Each issue now carries its own evidence comment (code + the exact new API + numeric/screenshot proof); all 20 are being closed against this PR.

The work was driven by a single API-consistency audit whose outcome is the binding "design language" for the round: one model-spec grammar everywhere (str / class / instance / {'model','args','kwargs'} / legacy {'model','params'}+DeprecationWarning / list-as-pipeline / fitted-model reuse), return_model= on every dispatcher, cross-module stage kwargs in one documented canonical order, and a public hyp.Pipeline.

What landed, by theme

• Unified cross-module API (extend cluster function api #138 sequences of analysis operations #153 Allow option to use DataGeometry objects à la scikit-learn pipelines #227 extend DataGeometry objects to provide access to "transform" and "inverse" methods #161 Support for scikit-learn's LatentDirichletAllocation and NMF, extend support for GaussianMixtureModel and BayesianGaussianMixtureModel #174). reduce/cluster/align/normalize/analyze/plot all accept manip=/normalize=/reduce=/align=/cluster= and return_model=, routed through a new hypertools.core.pipeline.build_pipeline in the canonical order manip → normalize → reduce → align → cluster (rendered as a flowchart in docs/pipeline_order.rst). New public hyp.Pipeline; fitted pipelines reapply to new data via plot(..., pipeline=p) (poison-pill tested: zero refit). reduce/ and cluster/ were migrated onto the same base-class+registry pattern as align/manip; mixture models are now valid reduce= targets (Support for scikit-learn's LatentDirichletAllocation and NMF, extend support for GaussianMixtureModel and BayesianGaussianMixtureModel #174).
• Manip chaining + story trajectories (hyp.plot(manip=...) does not exist; hyp.manip() cannot chain a list of steps -- blocks "smooth -> resample -> zscore -> UMAP" pipelines #274 "Story trajectories" demo pipeline (smooth -> resample -> zscore -> hyperalign -> UMAP -> windowed animation) cannot be reproduced -- missing manip=, animate='window', focused= #275). hyp.manip/hyp.plot(manip=[...]) chain a list of steps (any registry names). Your exact "Story trajectories" demo pipeline (smooth -> resample -> zscore -> hyperalign -> UMAP -> windowed animation) cannot be reproduced -- missing manip=, animate='window', focused= #275 snippet runs end-to-end on the real 36-subject weights data; new animate='window' + focused= + duration= deliver the requested "loosely aligned, interesting paths" (post-align inter-subject correlation −0.004 → 0.33; mean turning angle 21°). hyp.plot(manip=...) does not exist; hyp.manip() cannot chain a list of steps -- blocks "smooth -> resample -> zscore -> UMAP" pipelines #274's "jumps" are quantitatively fixed (max inter-frame jump 18.96 → 2.79 chained-manip → 0.73 gaussian-kernel).
• Plot/animation (ability to change label opacity #103 animate API (and style/labels) #154 2d animations #123). label_alpha=, xlabel=/ylabel=/zlabel=, animate= dict form, and full 2-D animation support in both backends.
• New reducers & data (Autoencoders as a dimensionality reduction algorithm #162 Extend hyp.load() to support additional source types (images, obj/nii files, seaborn/sklearn named datasets, audio) #273 Feature request: Loading functions for fivethirtyeight and kaggle #116 feature request: add support for LSL data streams #130 More text models #198). Six torch autoencoder reducers (hypertools[torch]); sklearn + seaborn named datasets and fivethirtyeight//kaggle/ explicit-prefix loaders in hyp.load; LSL streaming via hyp.io.lsl_stream (hypertools[lsl]); gensim vectorizer=/semantic= wrappers with the requested sklearn→gensim→HF parse order (hypertools[gensim]).
• Docs (Docstrings missing for hyp.manip and ~150 internal classes/methods across align/, predict/ #276 Sphinx docs: hyp.manip missing from docs/api.rst; two orphaned hypertools.tools.* pages not in any toctree #278 readthedocs API changes #159 text download idea #187 README example images/animations are unchanged since 2016-2017 (predate the 1.0 rewrite) #277). Docstring coverage 131 → 0 missing public defs (with a permanent AST enforcement test); api.rst now covers every public export and the two orphaned pages are gone; 7 new tutorials/examples (story-trajectories, datasets tour, pipelines & return_model, autoencoders, gensim, Wikipedia notebook, LSL notebook); all README media regenerated with 1.0 code (hero GIF 8.8 MB → 1.2 MB).

Process

Executed as 19 review-gated tasks (fresh implementer + independent reviewer per task; every graphical change verified by a fresh subagent viewing the rendered media at worst-case frames). The review gates caught and fixed five real bugs the reuse contract would otherwise have shipped — most notably build_pipeline silently refitting on every reuse, Aligner.transform/SRM replaying fit-time data instead of transforming new data, and Resample.transform ignoring its input.

Verification

• Local suite: 1271 passed, 0 failed (947 → 1271 this round; 6 kaleido image-export tests deselected locally only, run in CI).
• CI: all 12 jobs green on head CIHEAD — CIRUNLINK. (Two CI-only fixes this round: the fivethirtyeight/ loader now authenticates its GitHub API listing + CSV downloads via GITHUB_TOKEN so 12 concurrent jobs don't exhaust the anonymous 60/hr + raw.githubusercontent rate limits.)
• New optional extras: torch, gensim, lsl, kaggle (all opt-in; import hypertools never requires them).

PR remains open for your review — nothing merged.

🤖 Generated with Claude Code