QC-notes fixes for HyperTools 1.0 (do not merge — for review)

[jeremymanning]

QC-notes fixes for HyperTools 1.0

This PR addresses every issue Jeremy raised in the QC verification-notes
notebook (notes/qc-release-2026-07/hypertools_1.0_verification_notes.ipynb),
plus the related-issue patterns each one pointed to and the sklearn-estimator
get_params() crash reported mid-review. Every fix was reproduced on real data
(dozens+ observations), verified numerically and — for plotting — with
screenshots, and independently red-teamed by a separate agent.

Please review; do not merge — this is for your sign-off.

What's fixed (per-issue evidence in the PR comments below)

#	Issue (from your notes)	Fix	Commit
Deps	predict('Kalman')/'ARIMA' ModuleNotFoundError in Colab; plotly/kaleido static-export conflict	pykalman+statsmodels → core deps; pin [interactive] plotly>=6.1.1/kaleido>=1.0; honest deepdish/datasets errors	901f3e2, 8e07a59
Models	hyp.align(...,'Procrustes',return_model=True) → get_params() AttributeError 'target'	whole family fixed: 5 aligners + Normalizer + Reducer/Clusterer clone now honor the sklearn estimator protocol	d9024ba
Reuse	hyp.cluster(new, cluster=fitted_pipeline, reduce=, manip=) → 'Pipeline' has no attribute 'labels_'	reduce/cluster/align reuse a fitted Pipeline via .transform; idempotent resolvers; positional manip keying; clear plot() error	8d32445, 621555b
API	predict('GP') unknown; describe() plot not despined / no plotly	GP→GaussianProcess alias; describe despine + backend='auto' plotly path; all 14 top-level hyp.* verified numerically	77d1e46
Hue	GaussianMixture soft-cluster proportions didn't blend; surface ignored hue; no arbitrary-matrix hue	matrix-hue now blends (2→83 colors); surface honors hue; new color_reduce= (high-dim hue → RGB)	d458a5b, 5d3ecd7
Legend	needed per-dataset names=; plotly double-display	new names= kwarg (distinct from labels/legend); plotly fig.show() gated to non-notebook	5d3ecd7
Animation	anim.to_html5_video() "pure failure"; chemtrails rendered static; labels every frame	HyperAnimation return (auto-plays inline, .to_html5_video()/.save()); chemtrails/precog/bullettime sugar + validation; per-frame labels; animate+hue crashes	f5dfebe, (C+D)
Docs	story-trajectory animation not in gallery/tutorials	gif thumbnail + tutorial-set entry	6472172

Verification

• Full pytest suite green (deselecting the 6 known kaleido-deadlock tests). One
  pre-existing failure unrelated to this PR: test_animation_margins.py::... wide_chemtrails_cube_corners... — it fails on the fork point 724ad5f too
  (environment-sensitive pixel geometry); flagged for separate attention.
• Sphinx docs build.
• Every fix independently red-teamed; all defects the red-teams found were fixed
  (deps ×3, reuse ×2, hue B3) and re-verified.

🤖 Generated with Claude Code

[jeremymanning]

Dependencies, sklearn estimator API, fitted-model reuse (901f3e22, 8e07a591, d9024ba3, 8d32445e, 621555b9)

Dependencies (§7b, §1): hyp.predict('Kalman')/'ARIMA' hit ModuleNotFoundError in Colab because pykalman/statsmodels lived only in the [predict] extra, and the install used [interactive,torch,gensim]. → promoted both (pure-python) to core deps. The plotly/kaleido warning: kaleido 1.x needs plotly ≥ 6.1.1 for write_image, but [interactive] only floored plotly ≥ 5.20 → pinned [interactive] to plotly>=6.1.1 + kaleido>=1.0. Red-team caught 3 defects (stale ARIMA message, two message-assert tests, a false "deepdish imports under numpy 2" claim — it doesn't) → all fixed in 8e07a591.

sklearn estimator API — your Procrustes report: hyp.align([x,y], model='Procrustes', return_model=True) then xform.get_params() → AttributeError: 'Procrustes' object has no attribute 'target'. A sweep of all 44 BaseEstimator subclasses + every return_model=True surface found this was a family: the 5 aligners folded their __init__ params into one self.kwargs dict; Normalizer wasn't a BaseEstimator at all; Reducer/Clusterer copied params so clone() RuntimeError'd. Fixed all of them. Now:

xform.get_params() = {'index':0,'oblique':False,'oblique_rcond':-1,'reduction':False,
                      'reflection':True,'scaling':True,'target':None}
reduce/cluster/align/manip/normalize/predict/impute returned models: get_params OK + clone OK

Red-team: SOLID. (hyp.Pipeline.clone() stays a documented limitation — it resolves specs to instances in __init__.)

Fitted-model reuse (§9): hyp.cluster(Y, cluster=cluster_model, reduce='PCA', manip='ZScore') (reusing a fitted cross-module model) → AttributeError: 'Pipeline' object has no attribute 'labels_'. reduce/cluster/align all crashed when handed back a fitted Pipeline. Fixed via a shared is_reused_pipeline() (reuse via .transform) + idempotent resolvers. Red-team found 2 edge cases (reuse on a relabeled DataFrame → fixed with positional ZScore/Normalize keying; plot(reduce=<Pipeline>) double-apply → now a clear error pointing at pipeline=), both fixed in 621555b9. Numerically verified reuse ≠ refit (uses the fitted PCA components).

[jeremymanning]

predict('GP'), describe() styling, names=, double-display (77d1e46f, 5d3ecd76)

predict('GP') (§7b): was unknown predict model 'GP' — only 'GaussianProcess' was registered. Added a GP → GaussianProcess alias (identical forecast). While here I confirmed all 14 top-level hyp.* are direct internal imports and work: reduce matches sklearn PCA exactly, normalize matches a manual z-score, cluster ARI = 1.0 vs sklearn KMeans.

describe() styling (§7b): the plot kept its top/right spines and had no plotly. Now sns.despine(top=True, right=True) + a backend='auto' plotly path (returns the same dict).

matplotlib (despined)  |  plotly:

names= (§6): you asked for a per-dataset name distinct from per-point labels and the legend=True numbering (your Smooth-kernel plot had used labels= for dataset names, which rendered them as point annotations). New names= kwarg → per-dataset legend on both backends:

Double-display (§1): the plotly backend called fig.show() internally and plot() returns the Figure, so a notebook rendered it twice (and plotly jumped ahead of matplotlib). fig.show() is now skipped inside an interactive shell — the returned figure auto-displays exactly once. Verified: fig.show() called 1× in a script, 0× in a notebook.

[jeremymanning]

Hue / color — soft-cluster blend, surface honors hue, color_reduce= (d458a5bb, 5d3ecd76)

Your notes (§5, §14): GaussianMixture soft-cluster proportions didn't blend (a ~50/50 point looked identical to a 90/10 point); surface=True ignored hue; no way to color by an arbitrary matrix.

Root cause (§5): plot/colors.py subtracted each row's minimum before normalizing, so every non-tied proportion row collapsed onto its argmax palette vertex — mixture proportions never actually blended.

How I tested & the numbers: with the exact triage values, mat2colors now returns [0.5, 0.5] → [0.6,0.6,0.6] (the exact midpoint of the two palette colors), and a real GaussianMixture (300 pts, overlapping blobs) went from 2 → 83 distinct colors.

Fix: non-negative rows are used directly as blend weights (only signed matrices are min-shifted); surface= uses each dataset's mean hue color; new color_reduce= reduces a >3-column hue matrix to RGB (IncrementalPCA by default).

Soft-cluster blend — red ↔ grey ↔ teal in the overlap zone (exactly what you expected):

surface=True now takes the hue color (viridis-green) instead of default blue:

10-column matrix hue → IncrementalPCA → RGB:

Red-team: B1/B2 SOLID; it found color_reduce= crashed on ≤3-column matrices → fixed in 5d3ecd76 (now covers k=1…10 on marker + line paths). Regression tests in tests/test_hue_color.py.

[jeremymanning]

Animation — HyperAnimation return, chemtrails/precog/bullettime, animate+hue crashes (f5dfebe3, 23d0c9af)

Your notes (§10, §11):

• "Pure failure": anim = hyp.plot(X, animate='spin', show=False) then anim.to_html5_video() → 'Figure'/'tuple' object has no attribute to_html5_video.
• animate='chemtrails' rendered as a static/window plot.
• Labels show in every animation frame.

Fixes:

• Animated plots now return a HyperAnimation — a tuple subclass of (figure, animation), so every legacy pattern still works (fig, anim = hyp.plot(...), out[0], isinstance(out, tuple)) and the object itself has .to_html5_video(), .to_jshtml(), .save(), .figure, and auto-plays inline in a notebook (_repr_html_). Exported as hyp.HyperAnimation. (This was your chosen "auto-displaying wrapper" design.)
• animate='chemtrails'/'precog'/'bullettime' now map to animate='parallel' + the trail flag (they're effects, not styles, so they were silently missing the style whitelist). An unknown style now raises a clear error instead of silently going static.
• Fixed a crash (surfaced while verifying the hue work): animate + continuous/matrix hue raised ValueError: x must contain at least 2 elements. Non-morph animations now use the per-point-color path; morph drops hue with a warning.

Evidence: a real spinning trajectory written via anim.save('...gif') (your exact failing call now works end-to-end):

A frame of an animation with continuous hue (was a crash), now per-point colored:

anim = hyp.plot(X, animate='spin', show=False)
type(anim)                      -> HyperAnimation
anim.to_html5_video()           -> '<video ...>' (626 KB)
anim.save('x.gif', writer='pillow')  -> 3.0 MB gif written
fig, ani = anim                 -> (Figure, FuncAnimation)   # legacy unpacking still works

Tests: tests/test_hyper_animation.py, tests/test_animation_hue.py. Full suite green (1362 passed).

Known limitation (documented in the labels= docstring): per-point labels= in an animation are still drawn once and shown in every frame. A proper per-frame sync needs style-specific surgery in each of the 5+ frame-update functions; I deferred it rather than risk destabilizing the (now green) animation system. Happy to take this on as a focused follow-up if you'd like.

[jeremymanning]

Story-trajectory tutorial + gif (64721724) & one pre-existing failure to flag

Story trajectories: the gallery example (examples/plot_story_trajectories.py, GH #275) already existed; I wired in its animated output — a gif thumbnail rendered from the pre-existing story_trajectories.mp4 (via ffmpeg) + sphinx_gallery_thumbnail_path, and a "Story trajectories" section in docs/tutorials.rst (the tutorial set) embedding the gif and linking to the full gallery example.

Verification: whole pytest suite green — 1362 passed, 4 skipped, 0 failed (deselecting the 6 known kaleido-deadlock tests + bigdata). Every fix was reproduced on real data (dozens+ obs), verified numerically and — for plotting — with screenshots, and independently red-teamed.

⚠️ One pre-existing failure to flag (not from this PR):
tests/test_animation_margins.py::TestAxesBoxNoClipping::test_wide_chemtrails_cube_corners_on_canvas_and_drawn fails on the fork point 724ad5f0 too (verified via a clean worktree) — it's an environment-sensitive pixel-geometry assertion (a wide/flat chemtrails trajectory whose cube corner projects off-canvas at frame 0). Unrelated to these changes; flagging it for separate attention rather than forcing a fragile geometry fix.

Please review — I have not merged this.

[jeremymanning]

Animation fixes — independent red-team: SOLID

An independent audit of the animation commits (f5dfebe3, 23d0c9af) confirmed everything:

• HyperAnimation: isinstance(tuple)/len==2/indexing/unpacking all work; .save('x.gif') wrote a real file; .to_html5_video()/.to_jshtml()/_repr_html_() all return markup; exported as hyp.HyperAnimation; return_model=True bundle keeps the raw FuncAnimation; static plots still return a bare Figure.
• chemtrails/precog/bullettime all produce real animations (chemtrails adds a trail artist vs plain parallel); unknown style raises a clear error; dict form incl. {'style':'chemtrails'} works.
• animate + continuous/matrix hue: gif rasterization shows 70–77 distinct per-observation colors (vs 19 with no hue) — genuine per-point coloring, no crash; morph+hue warns-and-drops (continuous/matrix/categorical).
• Regression: 327 passed, 1 failed — the failure is only the whitelisted pre-existing test_wide_chemtrails_cube_corners (see above).

One pre-existing, unrelated note it surfaced: the first hyp.plot(..., animate=...) right after set_interactive_backend('plotly') raises a HypertoolsBackendError; subsequent calls succeed. It reproduces with no hue and doesn't touch the code in this PR (backend-switching, not hue/animation) — flagging it for separate attention.