diff --git a/LARGE_IMAGE_WEBGPU_PLAN.md b/LARGE_IMAGE_WEBGPU_PLAN.md new file mode 100644 index 00000000..b64e6dd8 --- /dev/null +++ b/LARGE_IMAGE_WEBGPU_PLAN.md @@ -0,0 +1,223 @@ +# Large-image WebGPU 2D rendering + binary transport — scoping document + +Status: **Phase 0 + core Phase 1/2 DONE, hardware-verified** (2026-07-06). Branch: +`feat/webgpu-2d-images`. Owner: @CSSFrancis + +## DONE (verified on an NVIDIA Pascal GPU via the SpyDE Electron consumer) +- **WebGPU 2-D image render path**: a `gpuCanvas` below `plotCanvas`; the normalized + uint8 frame → R8 texture, the 256-entry colormap → a 256×1 RGBA LUT texture, a + fullscreen-quad WGSL fragment shader re-stretches by clim (dmin/dmax uniform) and + samples the LUT. Replaces the 64M-iteration Canvas2D atob+LUT loop with one GPU draw. +- **Reuses the 3D contract**: the `_gpuDevice()` singleton, first-frame-canvas-then- + async-swap, `device.lost` → permanent Canvas2D fallback (extended to 2-D panels). +- **API**: `imshow(..., gpu="auto"|True|False)` → `gpu_mode`; `GPU_IMAGE_THRESHOLD` + (~1 Mpx) auto-gate; `plot.gpu_active` echo via the `gpu_status` event. +- **Fallback intact**: RGB images, sub-threshold images, `gpu=False`, and no-device + all render on Canvas2D (anyplotlib suite green; the DOM keeps `plotCanvas` first so + `querySelector('canvas')` still resolves the image canvas). +- **Correctness (review-hardened)**: the shader is a plain IDENTITY LUT lookup — + `_buildLut32` already bakes clim + scale_mode into the LUT, so the shader must NOT + re-apply the window (an earlier version double-applied clim; correct only at + full-range, wrong for any narrowed contrast/log/symlog — fixed). A zoomed/panned + view falls back to Canvas2D so the base image stays registered with the axes/ + overlays (the GPU quad is full-extent). Nearest sampling on both textures matches + Canvas2D's `imageSmoothingEnabled=false` pixel-for-pixel. +- **Zoom/pan v-window fix + headless GPU parity tests (2026-07-09)**: the shader + applied a global `1 - v` flip AFTER interpolating the `[v0,v1]` window, sampling + the MIRRORED window `[1-v1, 1-v0]` — correct only when v0+v1==1 (rest / centred + zoom), so it survived the readback tests but inverted pan-y and detached the + image from markers/overlays on any vertically off-centre view (base AND detail + passes; fixed by interpolating v from v1 at the screen bottom to v0 at the top). + Guarded by `tests/test_plot2d/test_gpu_parity_playwright.py`: GPU-vs-Canvas2D + PNG parity (rest/zoom/pan/markers/widgets/detail-tile) on REAL WebGPU in + headless Chromium — `channel="chromium"` (the full build; the default headless + shell has no `navigator.gpu`) + `--enable-unsafe-webgpu`, skipped when no + adapter. NB `page.screenshot()` DOES capture the WebGPU canvas there; the + "swapchain reads black" caveat below applies to Electron offscreen capture. +- **Verification**: `__apl_gpuReadback` renders the active panel to an OFFSCREEN + texture (the live swapchain reads black under automation) and copies it to CPU. + On a real 4k movie frame: min 0 / max 255 / 96% non-black / correct gray values. + A **narrowed clim [0.3,0.7]** matches the numpy windowed-colormap to meanDiff + 0.65/255 (regression guard for the double-apply bug; `tests/_gpu_clim_check.cjs`). + Movie scrub + playback re-upload the texture per frame and stay correct (5/5 + scrub, 5 distinct played frames). GPU resources are freed on panel close / figure + dispose / device loss (no leak on repeated large-image open/close). + +## DEFERRED (documented; not blockers now) +- **Mipmaps** (smooth downscale-on-zoom): the sampler uses `minFilter:'linear'` + without a mip chain. Marginal here because LOD already caps the uploaded texture + near display size; matters for deep zoom-out. Needs an R8 render-based mip chain. +- **Binary pixel transport** (base64-in-JSON → binary buffer): the Phase-0 headline, + but LOD decimation already cut the shipped payload ~35× (≤1536 px, ~2 MB not + ~85 MB) and the GPU shader removed the render cost, so this is now an incremental + transport optimization spanning Jupyter/Pyodide/standalone/Electron — do it with + the multi-environment verification it needs. + +Prerequisite reading: `WEBGPU_PLAN.md` (the 3D points/voxels WebGPU path this extends), +`anyplotlib/FIGURE_ESM.md` (the `figure_esm.js` section map), `AGENTS.md` (repo conventions). +Prerequisite reading: `WEBGPU_PLAN.md` (the 3D points/voxels WebGPU path this extends), +`anyplotlib/FIGURE_ESM.md` (the `figure_esm.js` section map), `AGENTS.md` (repo conventions). + +This extends the repo's existing hardware-verified `WEBGPU_PLAN.md` (3D instanced points + +voxels) to **2D large images**, deliberately lifting that doc's "No 2D pipeline changes" +non-goal (§2). Motivating consumer: a SpyDE in-situ movie viewer that must scrub/play through +8k×8k image frames smoothly. + +## 1. Goal + +Render **large 2D image frames** (up to 8k×8k) interactively — smooth scrub/playback of a +frame stream and smooth zoom — by moving the 2D image path from Canvas2D to **WebGPU** (texture +upload + WGSL colormap LUT + mipmap downscale) and moving the pixel bytes off **base64-in-JSON** +onto a **binary transport**. + +| Workload | Today (Canvas2D) | Target (WebGPU) | +|---|---|---| +| 8k×8k frame colormap+draw | ~64M-iter JS LUT loop → OffscreenCanvas | shader LUT, **<5 ms** | +| Scrub (new frame/tick) | ~85 MB base64/frame + full rebuild | binary uint8 + texture upload, **≥15–30 fps** | +| Zoom-in on a still | re-blit from OffscreenCanvas | GPU **mipmap**, **60 fps** | + +## 2. Non-goals + +- **Not** replacing Canvas2D for images — it stays the universal baseline, the fallback, the + small-image path, and the fully-CI-tested path. Default behaviour for small images is + byte-identical to today. +- No WebGL2, no three.js, no bundler — raw WebGPU + inline WGSL, mirroring `WEBGPU_PLAN.md`. +- 1D lines / bars stay Canvas2D. Only the 2D **image** path is affected. + +## 3. Coverage & the fallback contract + +Inherited verbatim from `WEBGPU_PLAN.md` §3: WebGPU is a progressive enhancement. `navigator.gpu` +present → `requestAdapter()` resolves → device created → *then* a panel may switch. Any failure at +any point (including mid-session device loss) lands on the Canvas2D path silently and permanently +for that session. **A figure must never render nothing because GPU was attempted.** + +## 4. Reuse (do NOT rebuild) + +- **Device singleton + fallback state machine**: `_gpuDevice()` (`figure_esm.js` ~L1955, + module-level `_gpuDevicePromise`), the `p._gpu ∈ {pending,active,unavailable}` per-panel state, + the "first frame always Canvas2D → swap on device resolve" pattern, and `device.lost` → permanent + per-session Canvas2D. All proven by the 3D path — the 2D image path plugs into the exact same + machinery. +- **The `gpuCanvas`-below-`plotCanvas` split**: decorations (axes, ticks, colorbar, scale bar, + overlay mask, markers, widgets) keep drawing on the 2D `plotCanvas` **verbatim** — only the image + raster moves to `gpuCanvas`. Mirrors `WEBGPU_PLAN.md` §4.3. +- **`gpu_mode` / `_gpu_active` plumbing**: the state field + echo already exist (`st.gpu_mode`, + ~L1998). Add an image-megapixel `GPU_IMAGE_THRESHOLD` alongside the existing point threshold. +- **The zoom/letterbox model**: `_imgFitRect` (~L1262) — the GPU draw honors the same fit-rect / + `zoom` semantics; no new zoom math. +- **The colormap LUT**: `_build_colormap_lut` (`_utils.py:118`) → `st.colormap_data` + ([[r,g,b]×256]) already ships to JS. Upload it as a 256×1 texture; the shader samples it. +- **The normalized payload**: `set_data` already produces `img_u8` (single-channel uint8 via + `_normalize_image`, `_utils.py:82`) + `display_min/max` (clim) + `raw_min/max`. The WebGPU path + uploads that **same uint8** (8× smaller than raw float64; keeps current visual behaviour) as an + **R8 texture**; the shader does the clim re-stretch + LUT — exactly what the JS loop at + `draw2d` L1379–1381 does today, but on the GPU. + +## 5. The function being replaced + +`draw2d` (`figure_esm.js` ~L1344): today it `atob`-decodes `image_b64`, runs a per-pixel LUT loop +(L1379–1381, 64M iters at 8k) into an `OffscreenCanvas` 2D context, caches the bitmap in +`blitCache`, and `_blit2d`-down-blits to the panel. The WebGPU path replaces the decode+LUT+blit +with: upload R8 texture → WGSL fragment shader (clim uniform + LUT texture) → mipmapped textured +quad over the `_imgFitRect`. `blitCache`'s "bytes unchanged → reuse" logic maps to "texture +unchanged → skip re-upload". Canvas2D `draw2d` remains as the fallback when `p._gpu !== 'active'`. + +## 6. Binary transport (generalize `WEBGPU_PLAN.md` §4.7 to images) + +Today `set_data` → `_encode_bytes(img_u8)` → `image_b64` string on the `panel__geom` trait +(base64-in-JSON). Change: send the raw `img_u8` **bytes** as an anywidget **binary buffer** +(`_repr_utils._widget_state` already handles `bytes`; the geom-trait split already isolates heavy +keys — `FIGURE_ESM.md` ~L237), with a small JSON header (`image_width/height`, `display_min/max`, +`raw_min/max`, dtype, an optional `lod` level). Keep base64 for **small images and the +standalone/Pyodide/`save_html` paths** (no binary channel there). Keep `FigureBridge` (`embed.py`) +transport-agnostic so an Electron embed can supply an even faster channel (shared memory / a +transferable `ArrayBuffer`) without the library caring. JS: pick up the binary buffer, upload to +the GPU texture; on the Canvas2D fallback, build the `ImageData` from the same bytes (no atob). + +## 7. API surface + +- `Axes.imshow(..., gpu="auto"|True|False)` and `Plot2D.set_data(..., gpu=...)` — mirror + `scatter3d`/`voxels`. Default `"auto"`: attempt WebGPU only above `GPU_IMAGE_THRESHOLD` + (initial ~4 megapixels — below it Canvas2D is already instant). `True` forces an attempt (still + falls back); `False` never attempts. +- `plot.gpu_active` — bool echo after first render (reuse `_gpu_active`). +- Optional `set_data(..., lod=k)` affordance so a consumer can mark a frame as decimated (scrub) + vs full-res (settle); at minimum, an uploaded full-res texture gets **free** GPU downscale-on-zoom + via mipmaps, so LOD-on-zoom needs no consumer work. + +## 8. Phases (with decision gates, mirroring WEBGPU_PLAN.md style) + +### Phase 0 — Binary transport + minimal GPU texture-blit (risk-retire; do first) + +Add the binary image trait + header (keep b64 fallback). Add a minimal WebGPU 2D image pipeline: +R8 texture + 256×1 LUT texture + clim uniform + a textured quad over `_imgFitRect`, **no mipmaps +yet**. Verify a single static 8k image renders identically to Canvas2D via **offscreen-texture +readback** (the 3D path's proven test method — the WebGPU swapchain doesn't snapshot reliably under +automation, per `FIGURE_ESM.md` ~L256). + +**Gate A:** GPU image matches Canvas2D within tolerance on a real GPU, and `gpu=False`/adapter-absent +renders identically via Canvas2D (automated). + +### Phase 1 — Mipmaps + zoom + scrub + +Generate mipmaps on upload; sample with trilinear filtering so zoom-out/downscale is smooth and +zoom-in honors `_imgFitRect`. Wire the scrub path: new bytes → texture re-upload (reuse +`blitCache`-style "unchanged → skip"). + +**Acceptance:** 8k still zooms at 60 fps; a frame stream scrubs ≥15–30 fps on a real GPU; benchmark +`js_gpu_image_8k` recorded. + +### Phase 2 — API + fallback hardening + tests + +`gpu="auto"|True|False` on `imshow`/`set_data`, `GPU_IMAGE_THRESHOLD`, `plot.gpu_active`, optional +`lod=`. Canvas2D-parity tests in the normal suite (assert `_gpu_active` false + pixel parity when +GPU absent); flagged headless-GPU smoke job (skip-on-null-adapter); `upcoming_changes/*.rst` +towncrier fragment. + +**Gate B:** full fallback matrix green (adapter-absent, mid-session device loss via forced +`device.destroy()`, `gpu=False`) + real-GPU image benchmark, before this is released. + +### Phase 3 (gated) — raw-float precision path + +If uint8 quantization proves visibly lossy for scientific contrast adjustment, add an optional R32F +texture upload (raw floats, 4× bytes) with the clim applied in-shader over true data range. Only +build on a concrete need (Gate C). + +## 9. Biggest risks / do-not-break + +- **The 3D WebGPU path and the Canvas2D 2D path must not regress** — the 2D image GPU path is + additive and plugs into the shared `_gpuDevice()` singleton; keep the device/fallback code shared, + not forked. +- **Canvas2D stays the default + fallback forever** — no figure may render nothing because GPU was + attempted (the `WEBGPU_PLAN.md` §3 contract). +- **Automation can't snapshot the WebGPU swapchain** — test via offscreen-texture readback or state + echo, not screenshot diffing of the live canvas (per `FIGURE_ESM.md`). +- **Small-image / standalone / Pyodide paths unchanged** — binary transport is large-image-only; + b64 remains for the rest. +- Repo norms (`AGENTS.md`): OO API only, state in `_state` dicts (no new traitlets on Plot2D — the + Figure adds panel traits dynamically), end every `_state` mutation with `_push()`, `uv run + pytest`, `uv run playwright install chromium` first, add the towncrier fragment. + +## 10. Verify + +`uv run pytest` (Canvas2D parity + fallback in the normal suite); the flagged GPU smoke job; +offscreen-readback comparison GPU-vs-Canvas2D on a large image; `js_gpu_image_8k` benchmark on a +real-GPU machine; a manual Electron-embed pass (the real consumer) once SpyDE pins the new version. + +## 11. API sketch + +```python +# Python +plot = ax.imshow(frame, gpu="auto") # auto-attempt WebGPU above GPU_IMAGE_THRESHOLD +plot.set_data(next_frame, clim=(lo, hi)) # scrub: binary bytes → texture re-upload +plot.set_data(decimated, lod=2) # scrub-time LOD hint (optional) +plot.gpu_active # bool, after first render echo +``` + +```js +// JS internals (figure_esm.js) +_gpuDevice() // existing module singleton → Promise +p._gpu // 'pending' | 'active' | 'unavailable' (existing) +_buildImagePipeline(device, p) // NEW: R8 sampled texture + LUT texture + clim uniform +_drawGpu2d(p) // NEW: image raster on gpuCanvas; decorations still 2D via draw2d +``` diff --git a/anyplotlib/_binary_frame.py b/anyplotlib/_binary_frame.py new file mode 100644 index 00000000..2f8bd526 --- /dev/null +++ b/anyplotlib/_binary_frame.py @@ -0,0 +1,78 @@ +""" +_binary_frame.py — a tiny length-prefixed binary wire format for shipping raw +image pixels to an Electron host without base64-in-JSON. + +Used ONLY by the Electron transport (``_electron.py``); Jupyter / Pyodide / +standalone keep the base64-in-JSON path untouched. The format is a single +self-describing frame written to a byte stream (stdout), interleaved with the +existing text ``PLOTAPP:{json}\\n`` lines: + + PLOTBIN::\\n + +- ``PLOTBIN:`` is the ASCII marker (distinct from ``PLOTAPP:``) so a host reading + the raw stream can tell a binary frame from a text line. +- ```` / ```` are ASCII decimal byte counts. +- ```` is UTF-8 JSON: ``{fig_id, key, ...metadata}`` (dims, + dtype, clim — everything the renderer needs EXCEPT the pixels). +- ```` is the raw payload (e.g. the single-channel uint8 image), + exactly ``payload_len`` bytes, NO base64, NO JSON. + +Both the Python producer and the JS host use this one definition, so the wire +format has a single source of truth and can be unit-tested in isolation (the +producer's ``encode_frame`` round-trips through ``decode_frame`` without any +Electron / stdout involved). +""" +from __future__ import annotations + +import json + +MARKER = b"PLOTBIN:" + + +def encode_frame(fig_id: str, key: str, header: dict, payload: bytes) -> bytes: + """Serialise one binary frame to bytes (marker + lengths + header + payload). + + ``header`` is arbitrary JSON-able metadata; ``fig_id`` and ``key`` are merged + in (so the decoder always recovers them). ``payload`` is the raw bytes.""" + hdr = dict(header or {}) + hdr["fig_id"] = fig_id + hdr["key"] = key + hdr_bytes = json.dumps(hdr, default=str).encode("utf-8") + prefix = MARKER + f"{len(hdr_bytes)}:{len(payload)}\n".encode("ascii") + return prefix + hdr_bytes + bytes(payload) + + +def decode_frame(buf: bytes): + """Parse ONE complete frame from ``buf`` (as produced by ``encode_frame``). + + Returns ``(header_dict, payload_bytes, n_consumed)`` where ``n_consumed`` is + the number of bytes the frame occupied. Raises ``ValueError`` if ``buf`` does + not begin with a complete frame. (A streaming host uses ``parse_prefix`` + + incremental reads instead; this is the whole-buffer convenience for tests.)""" + if not buf.startswith(MARKER): + raise ValueError("not a PLOTBIN frame") + nl = buf.find(b"\n") + if nl < 0: + raise ValueError("incomplete PLOTBIN prefix (no newline)") + prefix = buf[len(MARKER):nl].decode("ascii") + hdr_len_s, pay_len_s = prefix.split(":") + hdr_len, pay_len = int(hdr_len_s), int(pay_len_s) + start = nl + 1 + end_hdr = start + hdr_len + end_pay = end_hdr + pay_len + if len(buf) < end_pay: + raise ValueError("incomplete PLOTBIN frame (truncated body)") + header = json.loads(buf[start:end_hdr].decode("utf-8")) + payload = buf[end_hdr:end_pay] + return header, payload, end_pay + + +def parse_prefix(line: bytes): + """Parse a ``PLOTBIN::`` prefix line (the bytes up to, not + including, the newline). Returns ``(header_len, payload_len)``. For a + streaming host that reads the prefix line, then exactly ``header_len + + payload_len`` more bytes.""" + if not line.startswith(MARKER): + raise ValueError("not a PLOTBIN prefix") + hdr_len_s, pay_len_s = line[len(MARKER):].decode("ascii").split(":") + return int(hdr_len_s), int(pay_len_s) diff --git a/anyplotlib/_electron.py b/anyplotlib/_electron.py index fbbd5c1f..9f7edb5e 100644 --- a/anyplotlib/_electron.py +++ b/anyplotlib/_electron.py @@ -9,12 +9,109 @@ """ from __future__ import annotations +import base64 import json +import os import sys import uuid +import zlib + +from anyplotlib._binary_frame import encode_frame _figures: dict[str, object] = {} # fig_id -> Figure +# Opt-in binary transport for large image pixels (Electron host only). When on, +# the big ``image_b64`` / ``overlay_mask_b64`` pixel traits are shipped as a raw +# PLOTBIN binary frame (no base64, no JSON for the pixels) instead of a giant +# base64 string in the state_update JSON — the base64 encode + 5.6MB JSON line + +# JS atob cost ~200 ms/frame on a 4k movie. Off by default (base64 path) so this +# is zero-risk until the host opts in; SpyDE sets APL_BINARY_TRANSPORT=1. +_BINARY_TRANSPORT = os.environ.get("APL_BINARY_TRANSPORT") == "1" +# State keys whose value is a base64 pixel string worth sending as binary. +# detail_b64 is the zoom DETAIL TILE — it MUST be here too, else with binary +# transport on its _encode_pixels "\x00bin:" token stays in the geom JSON +# and the real bytes are never shipped, so the renderer can't decode it and the crisp +# zoom tile never displays (you only ever see the downsampled overview base). +_BINARY_KEYS = frozenset({"image_b64", "overlay_mask_b64", "detail_b64"}) + + +def _route_change(fig_id: str, name: str, value) -> None: + """Forward ONE trait change to the host — as a raw PLOTBIN binary frame for a + large image pixel trait (when binary transport is enabled), else as a + base64-in-JSON ``state_update``. Extracted from the observer so it's unit- + testable without a real Figure/stdout.""" + # Binary fast path: ship large image pixels as a raw PLOTBIN frame (the + # renderer rebuilds the bytes from the ArrayBuffer — no atob). + # + # The pixel keys ride INSIDE a panel geometry trait — ``panel__geom`` is + # a JSON string ``{"image_b64": , "colormap_data": …, …}`` (see + # Figure._push). Two producer modes: + # • RAW (Plot2D.set_data with binary transport): the geom carries a tiny + # ``"\x00bin:"`` change-token, and the real uint8 bytes sit in the + # Figure's ``_raw_pixels`` side-table. We emit those bytes directly — + # NO json-parse of a megabyte string, NO base64 decode. This is the hot + # scrub path (a 4k movie frame every tick). + # • BASE64 (initial __init__ frame / non-set_data pushes / robustness): the + # geom carries a real base64 string, which we decode once here. + # Either way the REST of the geom (LUT + flags, small) goes as a normal JSON + # state_update with the pixels removed. A bare pixel trait is handled too. + if _BINARY_TRANSPORT and isinstance(value, str) and value: + if name.startswith("panel_") and name.endswith("_geom"): + try: + geom = json.loads(value) + except Exception: + geom = None + if isinstance(geom, dict) and any(k in geom for k in _BINARY_KEYS): + panel_id = name[len("panel_"):-len("_geom")] + fig = _figures.get(fig_id) + raw_tbl = getattr(fig, "_raw_pixels", None) + sent_binary = False + for k in list(geom.keys()): + if k not in _BINARY_KEYS or not isinstance(geom[k], str) or not geom[k]: + continue + raw = None + if geom[k].startswith("\x00bin:") and raw_tbl is not None: + # RAW fast path: bytes are in the side-table (no decode). + raw = raw_tbl.get((panel_id, k)) + if raw is None: + # BASE64 path (initial frame / fallback): decode once. + try: + raw = base64.b64decode(geom[k]) + except Exception: + continue # leave it in geom → goes via JSON below + # Replace the pixel value with a small content TOKEN instead of + # popping the key: the renderer's caches (blitCache, GPU + # texture, maskCache) key their "unchanged → skip re-upload" + # checks on this string. Popping it left st. empty in JS, + # whose fallback fingerprint samples only 4 bytes of the + # buffer — two frames differing anywhere else COLLIDED and the + # skip froze the display on the old frame (found: a movie + # overview refresh that only moved a mid-frame feature). + tok = geom[k] + if not tok.startswith("\x00bin:"): + tok = f"\x00bin:{zlib.adler32(raw) & 0xFFFFFFFF}" + geom[k] = tok + # key carries which pixel field; geom= the trait name so the + # renderer routes it back into the right panel state. + emit_binary(fig_id, k, + {"nbytes": len(raw), "geom": name}, raw) + sent_binary = True + if sent_binary: + # Send the slimmed geom (LUT etc., pixels removed) as JSON. + emit({"type": "state_update", "fig_id": fig_id, "key": name, + "value": json.dumps(geom)}) + return + elif name in _BINARY_KEYS: + try: + raw = base64.b64decode(value) + emit_binary(fig_id, name, {"nbytes": len(raw)}, raw) + return + except Exception: + pass # fall back to the base64-in-JSON path below + if isinstance(value, (bytes, bytearray)): + value = {"buffer": base64.b64encode(value).decode()} + emit({"type": "state_update", "fig_id": fig_id, "key": name, "value": value}) + def register(fig) -> str: """Register *fig* for bidirectional state sync and return its fig_id.""" @@ -22,12 +119,7 @@ def register(fig) -> str: _figures[fig_id] = fig def _on_change(change): - name = change["name"] - value = change["new"] - if isinstance(value, (bytes, bytearray)): - import base64 - value = {"buffer": base64.b64encode(value).decode()} - emit({"type": "state_update", "fig_id": fig_id, "key": name, "value": value}) + _route_change(fig_id, change["name"], change["new"]) for name in fig.traits(sync=True): if not name.startswith("_"): @@ -72,3 +164,25 @@ def dispatch_event(fig_id: str, event_json: str) -> None: def emit(obj: dict) -> None: sys.stdout.write(f"PLOTAPP:{json.dumps(obj, default=str)}\n") sys.stdout.flush() + + +def emit_binary(fig_id: str, key: str, header: dict, payload: bytes) -> None: + """Write a raw PLOTBIN binary frame to stdout (pixels, no base64/JSON). + + Goes to the RAW ``sys.stdout.buffer`` so the payload is bytes on the wire; the + Electron host demuxes ``PLOTBIN:`` frames from the same stdout stream that + carries the text ``PLOTAPP:`` lines. Flushed so the host sees it promptly.""" + frame = encode_frame(fig_id, key, header, payload) + buf = getattr(sys.stdout, "buffer", None) + if buf is None: # no binary stdout (unusual) → base64 fallback + emit({"type": "state_update", "fig_id": fig_id, "key": key, + "value": base64.b64encode(payload).decode()}) + return + # NOTE: a host that redirects sys.stdout (e.g. SpyDE points sys.stdout at + # stderr to keep the protocol channel clean) MUST patch emit_binary to write + # to its real protocol stdout — otherwise these bytes go to the wrong stream. + # Flush the TEXT stream first so any pending PLOTAPP: line is on the wire + # before this binary frame — the host demuxes a single ordered byte stream. + sys.stdout.flush() + buf.write(frame) + buf.flush() diff --git a/anyplotlib/_repr_utils.py b/anyplotlib/_repr_utils.py index f8e57622..37332a7b 100644 --- a/anyplotlib/_repr_utils.py +++ b/anyplotlib/_repr_utils.py @@ -207,11 +207,49 @@ def _widget_px(widget) -> tuple[int, int]: // ── Inbound state updates from parent-page Pyodide ─────────────────────────── window.addEventListener('message', (e) => {{ - if (!e.data || e.data.type !== 'awi_state') return; - _fromParent = true; - model.set(e.data.key, e.data.value); - model.save_changes(); - _fromParent = false; + if (!e.data) return; + if (e.data.type === 'awi_state') {{ + _fromParent = true; + const _t0 = performance.now(); + model.set(e.data.key, e.data.value); + model.save_changes(); + if (e.data.key === 'image_b64') {{ + try {{ + globalThis.__apl_paint_ms = performance.now() - _t0; + globalThis.__apl_paint_n = (globalThis.__apl_paint_n || 0) + 1; + globalThis.__apl_paint_kind = 'base64'; + }} catch (_) {{}} + }} + _fromParent = false; + return; + }} + // Binary image frame (raw pixels, no base64). Set the bytes under a parallel + // `_bytes` trait (a Uint8Array) that the draw path uses directly — no atob. + // We also bump the base64 key to a short token so listeners keyed on it still + // fire, without carrying the (large) base64 string. + if (e.data.type === 'awi_state_binary') {{ + _fromParent = true; + const _t0 = performance.now(); + const hdr = e.data.header || {{}}; + const bytes = e.data.buffer instanceof Uint8Array + ? e.data.buffer : new Uint8Array(e.data.buffer); + // Stash the raw bytes in a global side-table (the anywidget model does NOT + // round-trip a Uint8Array set on an ad-hoc trait — model.get returns + // undefined), then bump a small TOKEN trait to fire the ESM's change: event, + // which reads the bytes from the side-table. Key = "::". + const slot = (hdr.geom ? hdr.geom : 'fig') + '::' + e.data.key; + (globalThis.__apl_pixbytes || (globalThis.__apl_pixbytes = {{}}))[slot] = bytes; + model.set(slot, (bytes ? bytes.length : 0) + ':' + (globalThis.__apl_paint_n || 0)); + model.save_changes(); // triggers the paint synchronously + // Paint-latency telemetry (message receipt → draw done) for the binary path. + try {{ + globalThis.__apl_paint_ms = performance.now() - _t0; + globalThis.__apl_paint_n = (globalThis.__apl_paint_n || 0) + 1; + globalThis.__apl_paint_kind = 'binary'; + }} catch (_) {{}} + _fromParent = false; + return; + }} }}); diff --git a/anyplotlib/_utils.py b/anyplotlib/_utils.py index f692addf..0c85708c 100644 --- a/anyplotlib/_utils.py +++ b/anyplotlib/_utils.py @@ -6,6 +6,8 @@ from __future__ import annotations +import functools + import numpy as np _LINESTYLE_ALIASES: dict[str, str] = { @@ -79,20 +81,48 @@ def _to_rgba_u8(data: np.ndarray) -> np.ndarray: return np.ascontiguousarray(data) -def _normalize_image(data: np.ndarray): - """Normalise data to uint8, returning (img_u8, vmin, vmax).""" - img = data.astype(np.float64, copy=False) - vmin = float(np.nanmin(img)) - vmax = float(np.nanmax(img)) - if vmax > vmin: - buf = np.empty_like(img) - np.subtract(img, vmin, out=buf) - np.divide(buf, vmax - vmin, out=buf) - np.multiply(buf, 255.0, out=buf) - img_u8 = buf.astype(np.uint8) +def _normalize_image(data: np.ndarray, clim: "tuple | None" = None): + """Normalise data to uint8, returning (img_u8, vmin, vmax) where vmin/vmax are + the QUANTISATION endpoints the 8-bit codes span (the caller stores them as + raw_min/raw_max so the renderer can reconstruct each code's value). + + When ``clim=(lo, hi)`` is given, quantise over THAT range instead of the raw + data min/max, clipping outliers to 0/255. This is what makes a single hot pixel + (or the saturating zero-order beam) NOT crush the real signal: without it the + 256 codes are stretched across the full data range, so a 60000-count hot pixel + over a 0-4000 signal leaves the signal only ~12 distinct codes (≈3.5-bit) before + it ever reaches the display window; clipping to the robust display range instead + gives the signal the full 256 codes and just saturates the outlier. When no + clim is given the behaviour is unchanged (quantise over raw min/max). + + Uses float32 for the rescale when the data range is small enough that float32's + ~7 significant digits don't lose low bits (the common case: uint8/uint16 movie + frames), else float64. float32 halves the memory bandwidth of the subtract/ + divide/multiply passes (~60ms → ~27ms on a 2048² frame — this runs every movie + frame) with a byte-identical result. vmin/vmax are always returned as full- + precision Python floats.""" + if clim is not None: + vmin, vmax = float(clim[0]), float(clim[1]) else: - img_u8 = np.zeros(data.shape, dtype=np.uint8) - return img_u8, vmin, vmax + vmin = float(np.nanmin(data)) + vmax = float(np.nanmax(data)) + if vmax <= vmin: + return np.zeros(data.shape, dtype=np.uint8), vmin, vmax + # float32 is safe when |values| and the span are within its ~1.6e7 exact-integer + # range with headroom; a huge-magnitude float source keeps float64 to avoid + # banding. (subtract of two near-equal large float32s loses precision.) + span = vmax - vmin + use32 = (max(abs(vmin), abs(vmax)) < 1e6) and (span > 1e-6) + dt = np.float32 if use32 else np.float64 + img = data.astype(dt, copy=False) + buf = np.empty_like(img) + np.subtract(img, dt(vmin), out=buf) + np.divide(buf, dt(span), out=buf) + np.multiply(buf, dt(255.0), out=buf) + # Clip into [0, 255] BEFORE the uint8 cast so out-of-clim values saturate + # instead of wrapping (a hot pixel above vmax would otherwise overflow uint8). + np.clip(buf, 0.0, 255.0, out=buf) + return buf.astype(np.uint8), vmin, vmax # Mapping from common matplotlib colormap names to their nearest colorcet @@ -115,9 +145,15 @@ def _normalize_image(data: np.ndarray): } +@functools.lru_cache(maxsize=64) def _build_colormap_lut(name: str) -> list: """Return a 256-entry ``[[r, g, b], ...]`` LUT for the named colormap. + CACHED (lru_cache) — it's a pure function of ``name`` but costs ~100 ms + (colorcet lookup + 256 hex parses), and it was being rebuilt on EVERY frame of + a movie scrub even though the colormap doesn't change. The returned list is + treated as read-only by callers (they JSON-serialise it); do NOT mutate it. + Priority order: 1. **colorcet** — preferred; common matplotlib names are remapped via diff --git a/anyplotlib/axes/_axes.py b/anyplotlib/axes/_axes.py index b7a65e68..a8e1f86a 100644 --- a/anyplotlib/axes/_axes.py +++ b/anyplotlib/axes/_axes.py @@ -33,13 +33,18 @@ def __init__(self, fig: "Figure", spec: "SubplotSpec"): # noqa: F821 self._plot: "Plot1D | Plot2D | None" = None # ------------------------------------------------------------------ - def imshow(self, data: np.ndarray, + def imshow(self, data, axes: list | None = None, units: str = "px", cmap: str | None = None, vmin: float | None = None, vmax: float | None = None, - origin: str = "upper") -> "Plot2D": + origin: str = "upper", + gpu: "str | bool" = "auto", + tile: "str | bool" = "auto", + integration_method: str = "mean", + overview_method: str = "mean", + tile_backend=None) -> "Plot2D": """Attach a 2-D image to this axes cell. Parameters @@ -74,7 +79,10 @@ def imshow(self, data: np.ndarray, x_axis = axes[0] if axes and len(axes) > 0 else None y_axis = axes[1] if axes and len(axes) > 1 else None plot = Plot2D(data, x_axis=x_axis, y_axis=y_axis, units=units, - cmap=cmap, vmin=vmin, vmax=vmax, origin=origin) + cmap=cmap, vmin=vmin, vmax=vmax, origin=origin, gpu=gpu, + tile=tile, integration_method=integration_method, + overview_method=overview_method, + tile_backend=tile_backend) self._attach(plot) return plot diff --git a/anyplotlib/callbacks.py b/anyplotlib/callbacks.py index 386eab34..ed30ab14 100644 --- a/anyplotlib/callbacks.py +++ b/anyplotlib/callbacks.py @@ -24,7 +24,7 @@ VALID_EVENT_TYPES = frozenset({ "pointer_down", "pointer_up", "pointer_move", "pointer_settled", "pointer_enter", "pointer_leave", "double_click", "wheel", - "key_down", "key_up", "close", "*", + "key_down", "key_up", "close", "view_changed", "*", }) @@ -79,6 +79,14 @@ class Event: # Wheel dx: float | None = None dy: float | None = None + # View (view_changed events — the current zoom/pan viewport) + zoom: float | None = None + center_x: float | None = None + center_y: float | None = None + image_width: int | None = None + image_height: int | None = None + display_width: int | None = None # panel device px (JS) → tile output resolution + display_height: int | None = None # Key key: str | None = None last_widget_id: str | None = None diff --git a/anyplotlib/figure/_figure.py b/anyplotlib/figure/_figure.py index fe5c16f6..6037c7d4 100644 --- a/anyplotlib/figure/_figure.py +++ b/anyplotlib/figure/_figure.py @@ -24,6 +24,16 @@ _ESM_SOURCE = (_HERE / "figure_esm.js").read_text(encoding="utf-8") +def _binary_wire() -> bool: + """True when the Electron binary pixel transport is live, so a pixel + change-token in the panel state should stay a token (the real bytes ride + PLOTBIN) rather than being resolved to inline base64. Read fresh from the + environment — the same gate ``Plot2D._encode_pixels`` uses — so producer and + serializer never disagree.""" + import os + return os.environ.get("APL_BINARY_TRANSPORT") == "1" + + class Figure(anywidget.AnyWidget): """Multi-panel interactive figure widget. @@ -124,6 +134,14 @@ def __init__(self, nrows=1, ncols=1, figsize=(640, 480), # when its values genuinely change. self._geom_rev: dict = {} self._geom_last: dict = {} + # Raw pixel side-table for the Electron BINARY transport: maps + # ``(panel_id, pixel_key)`` → the raw ``bytes`` of the current frame. + # When binary transport is active, ``Plot2D.set_data`` stashes the + # uint8 image bytes here (and puts a tiny change-token in the geom + # field instead of a 5.6 MB base64 string), so ``_electron._route_change`` + # ships them straight to a PLOTBIN frame with NO base64 encode/decode + # and NO megabyte JSON. Empty (and ignored) on every non-Electron path. + self._raw_pixels: dict = {} with self.hold_trait_notifications(): self.fig_width = figsize[0] self.fig_height = figsize[1] @@ -279,6 +297,14 @@ def _push(self, panel_id: str) -> None: state = plot.to_state_dict() geom_keys = getattr(plot, "_GEOM_KEYS", None) gname = f"panel_{panel_id}_geom" + # ``state`` may carry a ``"\x00bin:…"`` pixel change-token (binary + # transport: the real bytes ride PLOTBIN via ``_route_change``). Keep + # the token when that channel is live; otherwise — a standalone / + # save_html / Jupyter figure with no binary channel — materialise the + # real base64 inline so the pixels actually travel. ``_binary_wire`` + # matches the producer's gate (``Plot2D._encode_pixels``). + if geom_keys and not _binary_wire() and hasattr(plot, "resolve_pixel_tokens"): + plot.resolve_pixel_tokens(state) if geom_keys and self.has_trait(gname): # Split heavy geometry into its own channel. Detect change by # comparing the geom values themselves (the b64 strings / LUT @@ -501,8 +527,21 @@ def _dispatch_event(self, raw: str) -> None: plot = self._plots_map.get(panel_id) if plot is None: + if event_type == "view_changed": + import logging + # WARNING so SpyDE's log stream forwards it (it drops non-spyde.* INFO). + logging.getLogger("anyplotlib.tile").warning( + "[TILE] view_changed for UNKNOWN panel %r (have %r) — dropped", + panel_id, list(self._plots_map.keys())) return + if event_type == "view_changed": + import logging + logging.getLogger("anyplotlib.tile").warning( + "[TILE] view_changed RECEIVED panel=%s zoom=%s center=(%s,%s) disp=(%s,%s)", + panel_id, msg.get("zoom"), msg.get("center_x"), msg.get("center_y"), + msg.get("display_width"), msg.get("display_height")) + # GPU activation status echo (WebGPU path) — not a user event. if event_type == "gpu_status": if hasattr(plot, "_set_gpu_active"): @@ -537,6 +576,13 @@ def _dispatch_event(self, raw: str) -> None: group_index=msg.get("group_index"), dx=msg.get("dx"), dy=msg.get("dy"), + zoom=msg.get("zoom"), + center_x=msg.get("center_x"), + center_y=msg.get("center_y"), + image_width=msg.get("image_width"), + image_height=msg.get("image_height"), + display_width=msg.get("display_width"), + display_height=msg.get("display_height"), key=msg.get("key"), last_widget_id=msg.get("last_widget_id"), ) diff --git a/anyplotlib/figure_esm.js b/anyplotlib/figure_esm.js index a9fd8ff0..b1dde73b 100644 --- a/anyplotlib/figure_esm.js +++ b/anyplotlib/figure_esm.js @@ -12,6 +12,17 @@ function render({ model, el }) { // This guarantees pixel-perfect alignment of all panels in a row/column. const PAD_L=58, PAD_R=12, PAD_T=12, PAD_B=42; + // Tile-mode render diagnostics. OFF by default — a console.warn PER ANIMATION + // FRAME per panel floods the teed renderer console (and the host app's log + // relay) in normal use. Opt in with globalThis.__apl_tiledbg=true when + // debugging a render decision ("why is it blurry / white / flashing"): it logs + // which texture/passes drew, the detail region, and the contrast window. + if (globalThis.__apl_tiledbg === undefined) globalThis.__apl_tiledbg = false; + function _tiledbg(tag, msg) { + if (!globalThis.__apl_tiledbg) return; + try { console.warn(`[TILEDBG-JS:${tag}] ${msg}`); } catch (_) {} + } + // ── theme ──────────────────────────────────────────────────────────────── function _isDarkBg(node) { while (node && node !== document.body) { @@ -358,10 +369,44 @@ function render({ model, el }) { return state; } - // Parse the geom trait into the per-panel cache. + // Serialise a panel's VIEW state for the light `panel__json` trait, + // EXCLUDING the heavy geometry keys that `_applyGeom` splices in from the + // cache (`image_b64`, `colormap_data`, `*_b64`, and their binary `*_bytes` + // companions). The interaction handlers (wheel/pan/orbit) write the view + // state back on every mouse tick; without this strip they re-serialise — + // and re-transmit — the full frame each tick. For the BINARY path that is + // `JSON.stringify` of a Uint8Array → a `{"0":..,"1":..}` object with one + // key per byte (millions of entries), which stalls zoom on large images. + // Python's `_push` already keeps geometry off this trait; the echo-back + // listener re-splices geom from `_geomCache`, so dropping it here is safe + // (the geometry is never actually lost — only kept out of the light path). + function _viewStateJson(p2) { + const st = p2.state; + if (!st) return '{}'; + const geom = p2._geomCache; + if (!geom) return JSON.stringify(st); + // Fast path: shallow-clone without the cached geom keys (and their + // `_bytes` variants). Object.keys(geom) is exactly what _applyGeom + // merged in, so this auto-tracks any future geom key with no hardcoded + // list to drift out of sync with the Python-side _GEOM_KEYS. + const skip = new Set(Object.keys(geom)); + const view = {}; + for (const k in st) { if (!skip.has(k)) view[k] = st[k]; } + return JSON.stringify(view); + } + + // Parse the geom trait into the per-panel cache. PRESERVES any binary pixel + // bytes (`*_bytes`, delivered on the companion trait) — the slimmed geom JSON + // carries only the LUT/flags now, and it may arrive AFTER the binary frame, so + // replacing the cache wholesale would drop the pixels. Re-splice them. function _loadGeom(p2, raw, rev) { try { - p2._geomCache = JSON.parse(raw || '{}'); + const prev = p2._geomCache || {}; + const next = JSON.parse(raw || '{}'); + for (const k in prev) { + if (k.endsWith('_bytes') && next[k] === undefined) next[k] = prev[k]; + } + p2._geomCache = next; p2._geomRev = rev; } catch (_) {} } @@ -612,6 +657,37 @@ function render({ model, el }) { // ── per-panel state maps ────────────────────────────────────────────────── const panels = new Map(); let _suppressLayoutUpdate = false; // block re-entry during live resize + let _pixArrivalSeq = 0; // monotonic binary-pixel-frame arrival counter + + // Binary transport: raw pixel bytes ride a global side-table (`__apl_pixbytes`, + // keyed `panel__geom::`) because the model can't carry a + // Uint8Array. Splice ANY currently-present bytes for this panel's geom trait + // into `p2._geomCache` under `_bytes`, stamping a monotonic arrival + // sequence so _imageBytes gets a collision-proof cache key. Returns true if + // any bytes were spliced. Used by BOTH the per-key change listener AND the + // panel's INITIAL paint — the latter is load-bearing: a first pixel frame that + // arrived (via the awi_state_binary postMessage handler) BEFORE render() + // registered the change listener sits in the side-table with nothing to + // consume it; without this splice the panel stays permanently blank (the + // initial _loadGeom only parses the slimmed geom JSON, which carries the + // LUT/flags, NOT the bytes). Idempotent: re-splicing the same buffer keeps the + // same __aplSeq, so it does not force a spurious re-upload on the normal path. + function _spliceBinaryBytes(p2, geomTrait) { + const tbl = globalThis.__apl_pixbytes; + if (!tbl) return false; + let spliced = false; + for (const pixelKey of ['image_b64', 'overlay_mask_b64', 'detail_b64']) { + const b = tbl[`${geomTrait}::${pixelKey}`]; + if (!b) continue; + if (b.__aplSeq === undefined) { + try { b.__aplSeq = ++_pixArrivalSeq; } catch (_) {} + } + if (!p2._geomCache) p2._geomCache = {}; + p2._geomCache[pixelKey + '_bytes'] = b; + spliced = true; + } + return spliced; + } // ── layout application ─────────────────────────────────────────────────── function applyLayout() { @@ -666,7 +742,12 @@ function render({ model, el }) { } for (const [id, p] of panels) { - if (!seen.has(id)) { p.cell.remove(); panels.delete(id); } + if (!seen.has(id)) { + // Free GPU resources before dropping the panel (2D image textures + + // 3D geometry buffers), else they leak on re-layout / panel close. + try { _gpuDisposeImagePanel(p); _gpuDisposePanel(p); } catch (_) {} + p.cell.remove(); panels.delete(id); + } } // Update insetsContainer size and reposition all insets @@ -686,15 +767,28 @@ function render({ model, el }) { let cbCanvas=null, cbCtx=null, plotWrap=null, wrapNode=null; let titleCanvas=null; let stack3dGpuCanvas=null; // WebGPU geometry canvas (3D only) + let stack2dGpuCanvas=null; // WebGPU image canvas (2D large-image path) if (kind === '2d') { plotWrap = document.createElement('div'); plotWrap.style.cssText = `position:relative;display:inline-block;vertical-align:top;line-height:0;` + `width:${pw}px;height:${ph}px;overflow:visible;flex-shrink:0;`; + // gpuCanvas (WebGPU image) draws the image raster BELOW plotCanvas (via + // z-index 0) when GPU mode is active, while plotCanvas keeps drawing all + // decorations (axes/colorbar/scale-bar/mask/markers) over a now-transparent + // background. Hidden until/unless the GPU image path activates. It is + // appended AFTER plotCanvas (below) so DOM order keeps plotCanvas first — + // callers/tests that grab `querySelector('canvas')` still get the image + // canvas; z-index (not DOM order) controls the visual stacking. + var gpu2d = document.createElement('canvas'); + gpu2d.style.cssText = + `position:absolute;top:0;left:0;display:none;border-radius:2px;z-index:0;`; + stack2dGpuCanvas = gpu2d; + plotCanvas = document.createElement('canvas'); plotCanvas.style.cssText = - `position:absolute;display:block;border-radius:2px;background:${theme.bgCanvas};`; + `position:absolute;display:block;border-radius:2px;background:${theme.bgCanvas};z-index:1;`; overlayCanvas = document.createElement('canvas'); overlayCanvas.style.cssText = 'position:absolute;z-index:5;cursor:default;pointer-events:all;outline:none;touch-action:none;'; @@ -723,6 +817,7 @@ function render({ model, el }) { titleCanvas.style.cssText = `position:absolute;pointer-events:none;z-index:8;background:transparent;display:none;`; plotWrap.appendChild(plotCanvas); + plotWrap.appendChild(gpu2d); // below plotCanvas via z-index, after in DOM plotWrap.appendChild(overlayCanvas); plotWrap.appendChild(markersCanvas); plotWrap.appendChild(yAxisCanvas); @@ -795,7 +890,7 @@ function render({ model, el }) { return { plotCanvas, overlayCanvas, markersCanvas, statusBar, xAxisCanvas, yAxisCanvas, scaleBar, cbCanvas, cbCtx, plotWrap, wrapNode, titleCanvas, - gpuCanvas: stack3dGpuCanvas }; + gpuCanvas: stack3dGpuCanvas || stack2dGpuCanvas }; } function _createPanelDOM(id, kind, pw, ph, spec) { @@ -870,6 +965,20 @@ function render({ model, el }) { _loadGeom(p2, model.get(_geomTrait), rev); if (p2.state) { _applyGeom(p2, p2.state); _redrawPanel(p2); } }); + // Binary transport: raw pixel bytes for this panel arrive on a companion + // trait `panel__geom::` (a Uint8Array). Merge them into the + // geomCache under `_bytes` so _imageBytes uses them (no atob), and + // redraw. Fires INDEPENDENTLY of the geom JSON trait (which now carries only + // the small LUT/flags), so pixels + LUT converge in the cache. + for (const pixelKey of ['image_b64', 'overlay_mask_b64', 'detail_b64']) { + const slot = `${_geomTrait}::${pixelKey}`; + model.on(`change:${slot}`, () => { + const p2 = panels.get(id); + if (!p2) return; + _spliceBinaryBytes(p2, _geomTrait); + if (p2.state) { _applyGeom(p2, p2.state); _redrawPanel(p2); } + }); + } } model.on(`change:panel_${id}_json`, () => { @@ -890,7 +999,15 @@ function render({ model, el }) { _redrawPanel(p2); }); - if (_hasGeom) _loadGeom(p, model.get(_geomTrait), 1); + if (_hasGeom) { + _loadGeom(p, model.get(_geomTrait), 1); + // First-paint race fix: consume any binary pixel bytes that already + // arrived (before this render registered the change listeners above) so + // the INITIAL paint below shows them instead of leaving the panel blank + // until an organic second frame. No-op on the normal path (side-table + // empty until the first frame lands). + _spliceBinaryBytes(p, _geomTrait); + } try { p.state = JSON.parse(model.get(`panel_${id}_json`)); _applyGeom(p, p.state); @@ -1004,6 +1121,20 @@ function render({ model, el }) { _loadGeom(p2, model.get(_geomTrait), rev); if (p2.state) { _applyGeom(p2, p2.state); _redrawPanel(p2); } }); + // Binary transport: raw pixel bytes for this panel arrive on a companion + // trait `panel__geom::` (a Uint8Array). Merge them into the + // geomCache under `_bytes` so _imageBytes uses them (no atob), and + // redraw. Fires INDEPENDENTLY of the geom JSON trait (which now carries only + // the small LUT/flags), so pixels + LUT converge in the cache. + for (const pixelKey of ['image_b64', 'overlay_mask_b64', 'detail_b64']) { + const slot = `${_geomTrait}::${pixelKey}`; + model.on(`change:${slot}`, () => { + const p2 = panels.get(id); + if (!p2) return; + _spliceBinaryBytes(p2, _geomTrait); + if (p2.state) { _applyGeom(p2, p2.state); _redrawPanel(p2); } + }); + } } model.on(`change:panel_${id}_json`, () => { @@ -1024,7 +1155,12 @@ function render({ model, el }) { _redrawPanel(p2); }); - if (_hasGeom) _loadGeom(p, model.get(_geomTrait), 1); + if (_hasGeom) { + _loadGeom(p, model.get(_geomTrait), 1); + // First-paint race fix (see _createPanelDOM): splice any binary pixel + // bytes that arrived before render() registered the listeners above. + _spliceBinaryBytes(p, _geomTrait); + } try { p.state = JSON.parse(model.get(`panel_${id}_json`)); _applyGeom(p, p.state); @@ -1118,7 +1254,8 @@ function render({ model, el }) { function _sz(c, ctx, w, h) { c.style.width=w+'px'; c.style.height=h+'px'; c.width=w*dpr; c.height=h*dpr; - ctx.setTransform(dpr,0,0,dpr,0,0); + // ctx is null for a WebGPU canvas (no 2D context to transform). + if (ctx) ctx.setTransform(dpr,0,0,dpr,0,0); } if (p.kind === '2d') { @@ -1173,6 +1310,15 @@ function render({ model, el }) { p.plotCanvas.style.top = imgY + 'px'; _sz(p.plotCanvas, p.plotCtx, imgW, imgH); + // The 2D WebGPU image canvas (if present) sits under plotCanvas and matches + // the image area exactly. _sz sets CSS size + dpr backing; the WebGPU + // context reconfigures to this size on the next GPU draw. + if (p.gpuCanvas) { + p.gpuCanvas.style.left = imgX + 'px'; + p.gpuCanvas.style.top = imgY + 'px'; + _sz(p.gpuCanvas, null, imgW, imgH); + } + // Overlay and markers match the image canvas exactly p.overlayCanvas.style.left = imgX + 'px'; p.overlayCanvas.style.top = imgY + 'px'; @@ -1265,6 +1411,34 @@ function render({ model, el }) { return { x: (cw - fw) / 2, y: (ch - fh) / 2, w: fw, h: fh, s }; } + // On-screen rect occupied by image pixels at the current zoom. + // zoom>=1: full fit-rect; zoom<1: centred shrunken rect inside the fit-rect. + function _imgVisibleRect2d(st, pw, ph) { + const fr = _imgFitRect(st.image_width, st.image_height, pw, ph); + const z = st.zoom || 1.0; + if (z >= 1.0) return fr; + const w = fr.w * z, h = fr.h * z; + return { x: fr.x + (fr.w - w) / 2, y: fr.y + (fr.h - h) / 2, w, h, s: fr.s * z }; + } + + // Clamp a destination rect and its UV mapping to a clip rect. + // Returns null when fully outside the clip. + function _clipRectUv(dx, dy, dw, dh, u0, v0, u1, v1, cx, cy, cw, ch) { + if (!(dw > 0) || !(dh > 0) || !(cw > 0) || !(ch > 0)) return null; + const ox0 = dx, oy0 = dy, ox1 = dx + dw, oy1 = dy + dh; + const nx0 = Math.max(ox0, cx), ny0 = Math.max(oy0, cy); + const nx1 = Math.min(ox1, cx + cw), ny1 = Math.min(oy1, cy + ch); + if (nx1 <= nx0 || ny1 <= ny0) return null; + const tx0 = (nx0 - ox0) / dw, tx1 = (nx1 - ox0) / dw; + const ty0 = (ny0 - oy0) / dh, ty1 = (ny1 - oy0) / dh; + const du = u1 - u0, dv = v1 - v0; + return { + dx: nx0, dy: ny0, dw: nx1 - nx0, dh: ny1 - ny0, + u0: u0 + du * tx0, u1: u0 + du * tx1, + v0: v0 + dv * ty0, v1: v0 + dv * ty1, + }; + } + function _buildLut32(st) { const dMin=st.display_min, dMax=st.display_max; const hMin=st.raw_min!=null?st.raw_min:dMin; @@ -1319,7 +1493,42 @@ function render({ model, el }) { return _imgFitRect(st.image_width, st.image_height, pw, ph).s * st.zoom; } - function _blit2d(bitmap, st, pw, ph, ctx) { + // Build (and cache on the panel) the Canvas2D bitmap for the detail tile — + // LUT-colormapped like the base. Returns null if no tile. Cached by the tile's + // content key so it rebuilds only when the tile or LUT changes. + function _detailBitmap(p, st) { + if (st.is_rgb) return null; + const d = _detailBytes(st); + if (!d.bytes || !st.detail_width || !st.detail_height) return null; + const dw = st.detail_width, dh = st.detail_height; + const lk = _lutKey(st); + const cache = (p._detailBlit ||= {}); + if (cache.bitmap && cache.key === d.key && cache.lutKey === lk + && cache.w === dw && cache.h === dh) return cache.bitmap; + if (d.bytes.length < dw * dh) return null; + const imgData = new ImageData(dw, dh); + const lut = _buildLut32(st); + const out32 = new Uint32Array(imgData.data.buffer); + for (let i = 0; i < dw * dh; i++) out32[i] = lut[d.bytes[i]]; + const oc = new OffscreenCanvas(dw, dh); + oc.getContext('2d').putImageData(imgData, 0, 0); + cache.bitmap = oc; cache.key = d.key; cache.lutKey = lk; + cache.w = dw; cache.h = dh; + return oc; + } + + // Brief alpha ramp when detail-overlay mode changes (none/partial/full). + const _DETAIL_BLEND_MS = 90; + + function _detailBlendAlpha(p, mode) { + const now = performance.now(); + const b = (p._detailBlend ||= { mode: 'none', t0: now }); + if (b.mode !== mode) { b.mode = mode; b.t0 = now; } + if (mode === 'none') return 0; + return Math.max(0, Math.min(1, (now - b.t0) / _DETAIL_BLEND_MS)); + } + + function _blit2d(p, bitmap, st, pw, ph, ctx, detailBitmap) { const { x, y, w, h } = _imgFitRect(st.image_width, st.image_height, pw, ph); const zoom = st.zoom, cx = st.center_x, cy = st.center_y; const iw = st.image_width, ih = st.image_height; @@ -1327,18 +1536,58 @@ function render({ model, el }) { ctx.fillStyle = theme.bgCanvas; ctx.fillRect(0, 0, pw, ph); ctx.imageSmoothingEnabled = false; + // Base pass (overview or full frame) always draws first; detail layers blend over it. if (zoom >= 1.0) { - // Zoomed in: show a portion of the image filling the fit-rect. + // Zoomed in: show a portion of the image filling the fit-rect. The visible + // window is in LOGICAL px; scale it to the bitmap's OWN texels (the base may + // be a smaller overview: bitmap.width may be < image_width). + const bw = bitmap.width, bh = bitmap.height; + const kx = bw / iw, ky = bh / ih; // logical→bitmap texel scale const visW = iw / zoom, visH = ih / zoom; const srcX = Math.max(0, Math.min(iw - visW, cx * iw - visW / 2)); const srcY = Math.max(0, Math.min(ih - visH, cy * ih - visH / 2)); - ctx.drawImage(bitmap, srcX, srcY, visW, visH, x, y, w, h); + ctx.drawImage(bitmap, srcX * kx, srcY * ky, visW * kx, visH * ky, x, y, w, h); } else { - // Zoomed out: shrink the fit-rect proportionally, keep it centred. + // Zoomed out: shrink the fit-rect proportionally, keep it centred. Source is + // the whole bitmap (overview or full). const dstW = w * zoom, dstH = h * zoom; - ctx.drawImage(bitmap, 0, 0, iw, ih, + ctx.drawImage(bitmap, 0, 0, bitmap.width, bitmap.height, x + (w - dstW) / 2, y + (h - dstH) / 2, dstW, dstH); } + const det = detailBitmap ? _detailUV(st) : null; + const ov = (!det && detailBitmap) ? _detailOverlayRect(st, x, y, w, h) : null; + const mode = det ? 'full' : (ov ? 'partial' : 'none'); + const alpha = _detailBlendAlpha(p, mode); + if (mode !== 'none' && alpha < 1 && !p._detailBlendRAF) { + p._detailBlendRAF = requestAnimationFrame(() => { + p._detailBlendRAF = 0; + if (p.state) draw2d(p); + }); + } + if (!(alpha > 0) || !detailBitmap) return; + + if (det) { + const dw = detailBitmap.width, dh = detailBitmap.height; + const sx = det.u0 * dw, sy = det.v0 * dh; + const sw = (det.u1 - det.u0) * dw, sh = (det.v1 - det.v0) * dh; + ctx.save(); + ctx.globalAlpha = alpha; + ctx.drawImage(detailBitmap, sx, sy, sw, sh, x, y, w, h); + ctx.restore(); + return; + } + + const vr = _imgVisibleRect2d(st, pw, ph); + const cl = _clipRectUv(ov.dx, ov.dy, ov.dw, ov.dh, + 0, 0, 1, 1, vr.x, vr.y, vr.w, vr.h); + if (!cl) return; + const dw = detailBitmap.width, dh = detailBitmap.height; + ctx.save(); + ctx.globalAlpha = alpha; + ctx.drawImage(detailBitmap, + cl.u0 * dw, cl.v0 * dh, (cl.u1 - cl.u0) * dw, (cl.v1 - cl.v0) * dh, + cl.dx, cl.dy, cl.dw, cl.dh); + ctx.restore(); } function draw2d(p) { @@ -1353,25 +1602,71 @@ function render({ model, el }) { const imgW = p.imgW || Math.max(1, pw - PAD_L - PAD_R); const imgH = p.imgH || Math.max(1, ph - PAD_T - PAD_B); - // Decode base64 image bytes - const b64=st.image_b64||''; - const iw=st.image_width, ih=st.image_height; + // Image bytes: prefer the binary trait (image_bytes, no atob) over base64. + const _img = _imageBytes(st); + const b64 = _img.key; // cache identity (b64 string or bin token) + // The base bitmap is built at the OVERVIEW texel size (base_width/height in tile + // mode; else the full image). _blit2d then draws it over the fit-rect using the + // LOGICAL image_width/height for the zoom math, so the overview scales to the + // full extent (lower-res until a detail tile covers the view). + const iw=st.base_width||st.image_width, ih=st.base_height||st.image_height; - if(!b64||iw===0||ih===0){ctx.clearRect(0,0,imgW,imgH);return;} + if(!_img.bytes||iw===0||ih===0){ctx.clearRect(0,0,imgW,imgH);return;} const isRgb=!!st.is_rgb; // bytes are RGBA (4/px) — no LUT applies const lk=isRgb?'__rgb__':_lutKey(st); + + // ── WebGPU image path (large scalar images) ───────────────────────────── + // If the GPU path is active for this panel, draw the image raster on the + // gpuCanvas (shader-LUT colormap on a texture) and CLEAR the plotCanvas image + // area so the 2D decorations below (axes/colorbar/scale bar/mask/markers) + // composite over the GPU image. Any failure reverts to the Canvas2D blit for + // this frame (and the panel is marked unavailable on hard errors). + // Test/diagnostic hook: record what the GPU path decided this frame. + try { (globalThis.__apl_gpu2d ||= {})[p.id] = + { wanted: _gpuWanted2d(st), gpu: p._gpu, hasImg: !!p._gpuImg, + iw: st.image_width, ih: st.image_height, active: false }; } catch (_) {} + let _gpuPainted = false; + if (_gpuWanted2d(st) && p._gpu === 'active' && p._gpuImg) { + if (_gpuDraw2dImage(p, st, imgW, imgH)) { + if (p.gpuCanvas.style.display === 'none') p.gpuCanvas.style.display = 'block'; + // plotCanvas holds only decorations now → transparent, cleared each frame. + p.plotCanvas.style.background = 'transparent'; + ctx.clearRect(0, 0, imgW, imgH); + _gpuPainted = true; + try { globalThis.__apl_gpu2d[p.id].active = true; } catch (_) {} + } else { + _tiledbg('path', `panel=${p.id} GPU draw returned FALSE → Canvas2D fallback ` + + `(tile_enabled=${st.tile_enabled})`); + } + } else if (p.gpuCanvas && p.gpuCanvas.style.display !== 'none' + && (!_gpuWanted2d(st) || p._gpu !== 'active')) { + // GPU not painting this frame (shrank below threshold, RGB image, a + // zoom/pan, or device lost) → hide the GPU layer, restore the opaque plot + // canvas. gpu_active (capability) is echoed only on PERMANENT loss (the + // device-lost handler); a transient zoom/shrink is reversible so the flag + // stays as-is rather than flapping per frame. + p.gpuCanvas.style.display = 'none'; + p.plotCanvas.style.background = theme.bgCanvas; + try { globalThis.__apl_gpu2d[p.id].active = false; } catch (_) {} + } + + if (!_gpuPainted) { + if (globalThis.__apl_tiledbg && st.tile_enabled) { + const reg = st.detail_region || []; + const uv = _detailUV(st); + _tiledbg('canvas', `panel=${p.id} CANVAS2D blit (NOT gpu) ` + + `zoom=${(st.zoom||1).toFixed(2)} base[${st.base_width}x${st.base_height}] ` + + `image[${st.image_width}x${st.image_height}] detail_region=[${reg.join(',')}] ` + + `detailUV=${uv?'COVERS':'partial/none'} hasDetailBmp=${!!_detailBitmap(p, st)}`); + } const needRebuild = b64!==blitCache.bytesKey || lk!==blitCache.lutKey || !blitCache.bitmap || blitCache.w!==iw || blitCache.h!==ih; if(!needRebuild && blitCache.bitmap){ - _blit2d(blitCache.bitmap, st, imgW, imgH, ctx); + _blit2d(p, blitCache.bitmap, st, imgW, imgH, ctx, _detailBitmap(p, st)); } else { - let bytes; - try { - const bin=atob(b64); - bytes=new Uint8Array(bin.length); - for(let i=0;i { + // The panel may have been removed while the device promise was pending + // (guard like the 3D path) — don't init/report/draw on a detached panel. + if (!device || !panels.has(p.id)) { p._gpu = 'unavailable'; return; } + try { _gpuInitImagePanel(p, device); p._gpu = 'active'; _reportGpu(p); } + catch (e) { p._gpu = 'unavailable'; + console.warn('[anyplotlib] GPU image init failed:', e); } + _redrawPanel(p); + }); + } + + // Tile mode: once the panel has a real on-screen size, emit ONE view_changed so + // the Python tile consumer can size the overview / initial tile to the actual + // panel px (imgW/imgH are only known after the first layout). Fires immediately + // (not debounced) so the first crisp tile appears without a user gesture. + if (st.tile_enabled && !p._didInitialView && p.imgW > 0 && p.imgH > 0) { + p._didInitialView = true; + _emitViewChanged(p, true); } // ── Overlay mask compositing ───────────────────────────────────────────── @@ -1407,8 +1728,16 @@ function render({ model, el }) { mg=parseInt(mColor.slice(3,5),16); mb=parseInt(mColor.slice(5,7),16); } + // Binary transport ships the mask bytes on the companion trait (the + // geom then carries only a "\x00bin:" content token in mob64) — prefer + // them; atob is the base64-in-JSON fallback. let mBytes; - try{const bin=atob(mob64);mBytes=new Uint8Array(bin.length);for(let i=0;i function _gpuDevice() { @@ -1970,6 +2312,15 @@ function render({ model, el }) { if (p.plotCanvas) p.plotCanvas.style.background = theme.bgPlot; _gpuDisposePanel(p); _redrawPanel(p); + } else if (p.kind === '2d' && p._gpu === 'active') { + // 2-D image panel: revert to the Canvas2D blit path + free GPU + // resources, and echo the fallback so plot.gpu_active goes False. + p._gpu = 'unavailable'; + if (p.gpuCanvas) p.gpuCanvas.style.display = 'none'; + if (p.plotCanvas) p.plotCanvas.style.background = theme.bgCanvas; + _gpuDisposeImagePanel(p); + _reportGpu(p); + _redrawPanel(p); } } console.warn('[anyplotlib] WebGPU device lost — fell back to canvas:', @@ -1992,7 +2343,7 @@ function render({ model, el }) { } catch (_) {} } - // Should this panel try the GPU path for its current state? + // Should this 3-D panel try the GPU path for its current state? function _gpuWanted(st) { if (typeof navigator === 'undefined' || !navigator.gpu) return false; const mode = st.gpu_mode || 'auto'; @@ -2004,6 +2355,25 @@ function render({ model, el }) { return (st.vertices_count || 0) > thr; } + // Should this 2-D image panel take the WebGPU texture path? Scalar (LUT) images + // only — RGB(A) images stay on Canvas2D (they need no colormap and are rare/ + // small). Gated by megapixels above GPU_IMAGE_THRESHOLD unless gpu_mode forces. + function _gpuWanted2d(st) { + if (typeof navigator === 'undefined' || !navigator.gpu) return false; + const mode = st.gpu_mode || 'auto'; + if (mode === 'off') return false; + if (st.is_rgb) return false; // no LUT → nothing to accelerate + // Pixels present as base64 (image_b64) OR binary bytes (image_b64_bytes). + if ((!st.image_b64 && !st.image_b64_bytes) + || !st.image_width || !st.image_height) return false; + // ZOOM/PAN is now honoured by the shader (the quad's clip rect + uv sub-region + // mirror _blit2d, so the GPU image stays registered with the overlays and + // UPSAMPLES real texels on zoom-in), so we no longer fall back on zoom. See + // _imageDrawUniform. + if (mode === 'always') return true; + return (st.image_width * st.image_height) >= GPU_IMAGE_THRESHOLD; + } + const _GPU_POINT_WGSL = ` struct Uniforms { mvp : mat4x4, // clip-space transform (orthographic) @@ -2120,6 +2490,661 @@ fn fs(in : VsOut) -> @location(0) vec4 { } `; + // ── 2-D large-image WebGPU path ──────────────────────────────────────────── + // A fullscreen textured quad samples the normalized uint8 image (an R8 texture, + // the same bytes the Canvas2D path decodes) and maps each pixel through the + // 256-entry colormap LUT (a 256×1 RGBA texture). This replaces the 64-million- + // iteration JS atob+LUT loop with one GPU draw. + // + // CRITICAL: the LUT is built by _buildLut32, which ALREADY bakes the display + // window (clim) AND the scale_mode (linear/log/symlog) into a direct + // "pixel value → final colour" map: lut[raw] = final colour. The Canvas2D path + // uses it as a plain lookup (out32[i] = lut[bytes[i]]). So the shader must be an + // IDENTITY lookup — index the LUT by raw/255 and NOTHING else. (A previous + // version re-applied dmin/dmax in the shader on top of the already-windowed LUT, + // double-applying the contrast — correct only at full-range clim, wrong for any + // narrowed window or log/symlog. Do NOT reintroduce a clim uniform here.) + // + // The quad is fullscreen with NO zoom/pan: the GPU path is used only when the + // view is unzoomed/uncentred (see _gpuWanted2d); a zoomed/panned view falls + // back to Canvas2D so the base image stays registered with the axes/overlays. + const _GPU_IMAGE_WGSL = ` +// rect = the on-screen quad in CLIP space (x0,y0,x1,y1). For zoom<=1 this is the +// fit-rect shrunk by zoom (centred); for zoom>=1 it's the full fit-rect. +// uvrect = the texture sub-region to sample (u0,v0,u1,v1), in 0..1. For zoom>=1 +// this is the zoomed-in window of the image (a small region stretched over +// the fit-rect, nearest sampling UPSAMPLES real texels); for zoom<=1 it's +// the whole texture (0,0,1,1). Together they reproduce _blit2d's zoom/pan +// exactly, so the GPU image stays registered with the axes/overlays at any +// zoom. Outside the rect the clear colour shows as the letterbox bars. +struct U { rect : vec4, uvrect : vec4, }; +@group(0) @binding(0) var u : U; +@group(0) @binding(1) var img : texture_2d; +@group(0) @binding(2) var lut : texture_2d; +@group(0) @binding(3) var samp : sampler; + +struct VsOut { + @builtin(position) pos : vec4, + @location(0) uv : vec2, +}; + +// Unit quad in 0..1; mapped into the clip-space rect + the uv sub-region below. +const Q = array, 6>( + vec2(0.0,0.0), vec2(1.0,0.0), vec2(0.0,1.0), + vec2(0.0,1.0), vec2(1.0,0.0), vec2(1.0,1.0)); + +@vertex +fn vs(@builtin(vertex_index) vi : u32) -> VsOut { + var out : VsOut; + let q = Q[vi]; // 0..1 across the drawn quad + let x = mix(u.rect.x, u.rect.z, q.x); // lerp into clip-space rect + let y = mix(u.rect.y, u.rect.w, q.y); + out.pos = vec4(x, y, 0.0, 1.0); + let uu = mix(u.uvrect.x, u.uvrect.z, q.x); + // q.y=0 is the SCREEN BOTTOM (rect.y is the lower clip edge), which shows + // the LAST row of the visible window (v1); q.y=1 (screen top) shows the + // first (v0) — so v interpolates from v1 down to v0 WITHIN the window + // (imshow: texture row 0 = image top). Do NOT flip with a global (1 - v) + // after interpolating [v0,v1]: that samples the MIRRORED window + // [1-v1, 1-v0] — identical only when v0+v1 == 1 (full view / vertically + // centred), so it survived centred-zoom tests but inverted the pan-y + // direction and detached the image from markers/overlays on any + // vertically off-centre view (base AND detail passes). + let vv = mix(u.uvrect.w, u.uvrect.y, q.y); + out.uv = vec2(uu, vv); + return out; +} + +@fragment +fn fs(in : VsOut) -> @location(0) vec4 { + // R8 unorm (0..1) = the frame value already normalized to the 0..255 index the + // LUT is keyed on. Direct identity lookup — the LUT holds the final colour + // (clim + scale_mode baked in). Nearest sampling on img (no interpolation of + // raw values); the LUT is sampled at the texel centre. + let raw = textureSampleLevel(img, samp, in.uv, 0.0).r; // 0..1 == index/255 + return textureSampleLevel(lut, samp, vec2(raw, 0.5), 0.0); +} +`; + + function _gpuInitImagePanel(p, device) { + const fmt = navigator.gpu.getPreferredCanvasFormat(); + const ctx = p.gpuCanvas.getContext('webgpu'); + ctx.configure({ device, format: fmt, alphaMode: 'opaque' }); + const module = device.createShaderModule({ code: _GPU_IMAGE_WGSL }); + const pipeline = device.createRenderPipeline({ + layout: 'auto', + vertex: { module, entryPoint: 'vs' }, + fragment: { module, entryPoint: 'fs', targets: [{ format: fmt }] }, + primitive:{ topology: 'triangle-list' }, + }); + // NEAREST on both textures matches the Canvas2D path's imageSmoothingEnabled= + // false (no interpolation of raw values or between LUT colour entries), so the + // GPU output is a pixel-faithful match of the reference. + const samp = device.createSampler({ + magFilter: 'nearest', minFilter: 'nearest', mipmapFilter: 'nearest' }); + // Uniform holds the clip-space quad rect + the texture uv sub-region (two + // vec4 = 32 B) so the quad matches the fit-rect AND zoom/pan of the Canvas2D + // path + overlays. + const uniformBuf = device.createBuffer({ + size: 32, usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST }); + // Second uniform for the DETAIL overlay pass (its own screen rect + uv), so the + // crisp tile can be stitched OVER the upsampled overview base in one frame + // (base fills the margin, tile fills the padded FOV — no zoom-out flash). + const uniformBuf2 = device.createBuffer({ + size: 32, usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST }); + p._gpuImg = { device, ctx, fmt, pipeline, samp, uniformBuf, uniformBuf2, + tex: null, texW: 0, texH: 0, lutTex: null, lutKey: null, + bytesKey: null, bindGroup: null, + // Detail texture slot (kept resident alongside the base). + dtex: null, dtexW: 0, dtexH: 0, dbytesKey: null, dbindGroup: null }; + } + + // Raw single-channel pixel bytes for the current image state. Prefers the + // BINARY bytes `image_b64_bytes` (a Uint8Array spliced into the state from the + // geomCache by the binary transport — no atob) over `image_b64` (base64, needs + // decoding). Returns {bytes, key} where `key` is a cheap identity for the + // "unchanged → skip re-upload" check. + function _imageBytes(st) { + const raw = st.image_b64_bytes; + if (raw && (raw instanceof Uint8Array || raw.byteLength !== undefined)) { + const u8 = raw instanceof Uint8Array ? raw : new Uint8Array(raw); + // Cache identity, in preference order: + // 1. the content token in st.image_b64 ("\x00bin:", kept in the + // slimmed geom by _route_change), + // 2. the buffer's ARRIVAL sequence (stamped when the binary frame was + // spliced in — every received frame is new content), + // 3. a sampled fingerprint as a last resort. NB sampling is WEAK: it + // reads only 4 bytes, so frames differing anywhere else collide and + // the "unchanged → skip re-upload" check freezes the display — this + // branch must stay a fallback, never the primary key. + let key = st.image_b64; + if (!key && u8.__aplSeq !== undefined) key = `binseq:${u8.__aplSeq}`; + if (!key) { + const n = u8.length; + const s0 = u8[0] || 0, s1 = u8[(n >> 2) | 0] || 0; + const s2 = u8[(n >> 1) | 0] || 0, s3 = u8[n - 1] || 0; + key = `bin:${n}:${s0},${s1},${s2},${s3}`; + } + return { bytes: u8, key }; + } + const b64 = st.image_b64 || ''; + if (!b64) return { bytes: null, key: '' }; + try { + const bin = atob(b64); + const u8 = new Uint8Array(bin.length); + for (let i = 0; i < bin.length; i++) u8[i] = bin.charCodeAt(i); + return { bytes: u8, key: b64 }; + } catch (_) { return { bytes: null, key: '' }; } + } + + // Detail-tile bytes (binary `detail_b64_bytes` or base64 `detail_b64`), keyed so a + // new tile re-uploads. Prefix the key with 'detail:' so it never collides with a + // base-image key of the same length (they share the one GPU texture slot). + function _detailBytes(st) { + const raw = st.detail_b64_bytes; + if (raw && (raw instanceof Uint8Array || raw.byteLength !== undefined)) { + const u8 = raw instanceof Uint8Array ? raw : new Uint8Array(raw); + const reg = st.detail_region || []; + // Key on detail_seq (a monotonic counter bumped by Python's set_detail on EVERY + // push), NOT just length+region. A live movie scrub while zoomed in re-samples + // the SAME region every frame → identical length + region, so a length/region + // key would match every frame and the GPU/Canvas dedup would SKIP the re-upload + // — the display freezes on the first frame ("won't update when zoomed in"). + // detail_seq guarantees a distinct key per pushed tile. + const seq = st.detail_seq || 0; + return { bytes: u8, key: `detail:${seq}:${u8.length}:${reg.join(',')}` }; + } + const b64 = st.detail_b64 || ''; + if (!b64) return { bytes: null, key: '' }; + try { + const bin = atob(b64); + const u8 = new Uint8Array(bin.length); + for (let i = 0; i < bin.length; i++) u8[i] = bin.charCodeAt(i); + return { bytes: u8, key: `detail:${b64}` }; + } catch (_) { return { bytes: null, key: '' }; } + } + + // Write single-channel R8 bytes into a device texture, (re)creating it if the size + // changed. Returns the (possibly new) texture, or null if bytes are missing/short. + function _gpuWriteR8(device, tex, texWH, iw, ih, bytes) { + if (!bytes) return null; + if (bytes.length < iw * ih) return null; + if (!tex || texWH.w !== iw || texWH.h !== ih) { + if (tex) tex.destroy(); + tex = device.createTexture({ + size: [iw, ih, 1], format: 'r8unorm', + usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | + GPUTextureUsage.RENDER_ATTACHMENT }); + texWH.w = iw; texWH.h = ih; + } + // bytesPerRow must be a multiple of 256 for writeTexture; R8 = 1 B/px, so pad + // each row up to the next 256 boundary. + const bpr = Math.ceil(iw / 256) * 256; + let src = bytes; + if (bpr !== iw) { + src = new Uint8Array(bpr * ih); + for (let r = 0; r < ih; r++) src.set(bytes.subarray(r * iw, r * iw + iw), r * bpr); + } + device.queue.writeTexture( + { texture: tex }, src, { bytesPerRow: bpr, rowsPerImage: ih }, [iw, ih, 1]); + return tex; + } + + // Ensure the shared LUT texture is current for st. Returns false only if it can't + // be built. Sets g.lutTex; invalidates both bind groups when the LUT changes. + function _gpuEnsureLut(g, st) { + const device = g.device; + const lutKey = _lutKey(st); + if (lutKey === g.lutKey && g.lutTex) return true; + const lut = _buildLut32(st); // Uint32Array(256), 0xAABBGGRR + const rgba = new Uint8Array(256 * 4); + for (let i = 0; i < 256; i++) { + const v = lut[i]; + rgba[i*4] = v & 0xff; + rgba[i*4+1] = (v >> 8) & 0xff; + rgba[i*4+2] = (v >> 16) & 0xff; + rgba[i*4+3] = 255; + } + if (!g.lutTex) { + g.lutTex = device.createTexture({ + size: [256, 1, 1], format: 'rgba8unorm', + usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST }); + } + device.queue.writeTexture( + { texture: g.lutTex }, rgba, { bytesPerRow: 256 * 4, rowsPerImage: 1 }, + [256, 1, 1]); + g.lutKey = lutKey; + g.bindGroup = null; g.dbindGroup = null; // LUT changed → both groups rebuild + return true; + } + + // Upload the BASE (overview/full) image bytes + the LUT. Returns false if the base + // bytes couldn't be decoded (caller falls back to Canvas2D). The detail tile is + // uploaded separately (_gpuUploadDetail) into its own slot for the overlay pass — + // base and detail are STITCHED in one frame, not swapped, so a zoom-out shows the + // crisp padded tile over an upsampled-overview margin (no flash). + function _gpuUploadImage(p, st) { + const g = p._gpuImg, device = g.device; + // The BASE texture is an OVERVIEW in tile mode: its real texel dims are + // base_width/height (0 → full image_width/height). The shader samples UV as a + // fraction 0..1 of the texture, so a smaller overview still maps over the full + // logical extent — lower-res until the detail tile covers the view. + const iw = st.base_width || st.image_width; + const ih = st.base_height || st.image_height; + const img = _imageBytes(st); + if (img.key !== g.bytesKey || !g.tex || g.texW !== iw || g.texH !== ih) { + const texWH = { w: g.texW, h: g.texH }; + const tex = _gpuWriteR8(device, g.tex, texWH, iw, ih, img.bytes); + if (!tex) return false; + g.tex = tex; g.texW = texWH.w; g.texH = texWH.h; + g.bytesKey = img.key; + g.bindGroup = null; // texture changed → rebuild bind group + } + if (!_gpuEnsureLut(g, st)) return false; + if (!g.bindGroup) { + g.bindGroup = device.createBindGroup({ + layout: g.pipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: { buffer: g.uniformBuf } }, + { binding: 1, resource: g.tex.createView() }, + { binding: 2, resource: g.lutTex.createView() }, + { binding: 3, resource: g.samp }, + ], + }); + } + return true; + } + + // Upload the DETAIL tile into its own slot (g.dtex) + build g.dbindGroup bound to + // uniformBuf2. Returns true if a detail tile is resident & ready to draw, false if + // there's none (or its bytes couldn't be decoded → skip the overlay pass). Assumes + // _gpuUploadImage already ran this frame (LUT is current). + function _gpuUploadDetail(p, st) { + const g = p._gpuImg, device = g.device; + const reg = st.detail_region; + if (!_hasDetail(st) || !reg || reg.length !== 4 + || !st.detail_width || !st.detail_height) return false; + const iw = st.detail_width, ih = st.detail_height; + const img = _detailBytes(st); + if (img.key !== g.dbytesKey || !g.dtex || g.dtexW !== iw || g.dtexH !== ih) { + const texWH = { w: g.dtexW, h: g.dtexH }; + const tex = _gpuWriteR8(device, g.dtex, texWH, iw, ih, img.bytes); + if (!tex) return false; + g.dtex = tex; g.dtexW = texWH.w; g.dtexH = texWH.h; + g.dbytesKey = img.key; + g.dbindGroup = null; + } + if (!g.dtex) return false; + if (!g.dbindGroup) { + g.dbindGroup = device.createBindGroup({ + layout: g.pipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: { buffer: g.uniformBuf2 } }, + { binding: 1, resource: g.dtex.createView() }, + { binding: 2, resource: g.lutTex.createView() }, + { binding: 3, resource: g.samp }, + ], + }); + } + return true; + } + + // Compute the GPU uniform: the on-screen quad in CLIP space + the texture uv + // sub-region to sample, honouring zoom/center EXACTLY like the Canvas2D _blit2d + // (so the GPU image stays registered with the axes/overlays at any zoom, and a + // zoom-in UPSAMPLES real texels via nearest sampling). Returns 8 floats: + // [rx0,ry0,rx1,ry1, u0,v0,u1,v1]. imgW/imgH = the gpuCanvas area in CSS px. + // The visible image-pixel window [srcX, srcX+visW]×[srcY, srcY+visH] for the + // current zoom/center, in BASE image pixels (zoom>=1). Mirrors _imageDrawUniform. + function _visibleWindow(st) { + const iw = st.image_width, ih = st.image_height; + const zoom = st.zoom || 1.0; + if (zoom < 1.0) return { srcX: 0, srcY: 0, visW: iw, visH: ih }; + const cx = (st.center_x == null ? 0.5 : st.center_x); + const cy = (st.center_y == null ? 0.5 : st.center_y); + const visW = iw / zoom, visH = ih / zoom; + const srcX = Math.max(0, Math.min(iw - visW, cx * iw - visW / 2)); + const srcY = Math.max(0, Math.min(ih - visH, cy * ih - visH / 2)); + return { srcX, srcY, visW, visH }; + } + + // Is the detail tile active AND does it fully cover the current visible window? + // If so, return the tile-relative UV of that window (so the shader samples the + // hi-res tile); otherwise null (sample the base texture). Requires the whole + // visible window inside detail_region so a zoom-in shows only crisp tile texels. + // True when a detail tile is present, regardless of transport: base64 path leaves + // the string in st.detail_b64; the BINARY path strips detail_b64 out of the geom + // and ships bytes on the companion trait (spliced into st.detail_b64_bytes). Gating + // on detail_b64 alone made the binary path NEVER show the detail tile — only the + // blurry overview — because the string is absent under binary transport. + function _hasDetail(st) { + if (st.detail_b64) return true; + const b = st.detail_b64_bytes; + return !!(b && (b instanceof Uint8Array || b.byteLength !== undefined) && b.length); + } + + function _detailUV(st) { + const reg = st.detail_region; + if (!_hasDetail(st) || !reg || reg.length !== 4) return null; + const [tx0, tx1, ty0, ty1] = reg; + const tw = tx1 - tx0, th = ty1 - ty0; + if (!(tw > 0) || !(th > 0)) return null; + const w = _visibleWindow(st); + if (w.srcX < tx0 - 0.5 || w.srcX + w.visW > tx1 + 0.5 || + w.srcY < ty0 - 0.5 || w.srcY + w.visH > ty1 + 0.5) return null; + return { + u0: (w.srcX - tx0) / tw, u1: (w.srcX + w.visW - tx0) / tw, + v0: (w.srcY - ty0) / th, v1: (w.srcY + w.visH - ty0) / th, + }; + } + + // The screen rect (within the fit-rect fx,fy,fw,fh) where the detail tile's + // logical region maps at the current zoom/center — for compositing the still- + // valid crisp tile OVER the base when it no longer FULLY covers the view (a + // zoom-out/pan that outgrew the region). Returns null when: no tile; the tile + // already fully covers (the _detailUV branch draws it full-screen instead); or + // the region is entirely off-screen (nothing to overlay). This is what removes + // the zoom-out flash — the centre stays crisp until the wider tile lands. + function _detailOverlayRect(st, fx, fy, fw, fh) { + const reg = st.detail_region; + if (!_hasDetail(st) || !reg || reg.length !== 4) return null; + if (_detailUV(st)) return null; // fully covered → drawn elsewhere + const zoom = st.zoom || 1.0; + // At/below zoom 1 the whole image is visible and the overview base is + // authoritative — don't float a stale crisp patch over it. The overlay is a + // zoom-IN / transition aid (it prevents the zoom-OUT flash WHILE still + // magnified); the live loop clears the tile below VIEW_ZOOM_MIN anyway, so this + // only additionally guards a manually-set tile at zoom<=1. + if (zoom <= 1.0) return null; + const [tx0, tx1, ty0, ty1] = reg; + if (!(tx1 > tx0) || !(ty1 > ty0)) return null; + const iw = st.image_width, ih = st.image_height; + // Logical→screen mapping identical to the zoom>=1 base blit (windowed), so the + // overlaid tile registers exactly over the base. + const cx = (st.center_x == null ? 0.5 : st.center_x); + const cy = (st.center_y == null ? 0.5 : st.center_y); + const visW = iw / zoom, visH = ih / zoom; + const srcX = Math.max(0, Math.min(iw - visW, cx * iw - visW / 2)); + const srcY = Math.max(0, Math.min(ih - visH, cy * ih - visH / 2)); + const mapX = (lx) => fx + ((lx - srcX) / visW) * fw; + const mapY = (ly) => fy + ((ly - srcY) / visH) * fh; + const dx = mapX(tx0), dx1 = mapX(tx1); + const dy = mapY(ty0), dy1 = mapY(ty1); + const dw = dx1 - dx, dh = dy1 - dy; + if (!(dw > 0.5) || !(dh > 0.5)) return null; + // Fully outside the fit-rect → nothing to show. + if (dx1 <= fx || dx >= fx + fw || dy1 <= fy || dy >= fy + fh) return null; + return { dx, dy, dw, dh }; + } + + // Pack a screen dest rect (px, y-down) + uv sub-region (0..1) into the 8-float + // clip-space uniform the shader expects. Shared by the base + detail passes. + function _drawUniform(dx, dy, dw, dh, u0, v0, u1, v1, imgW, imgH) { + const rx0 = (dx / imgW) * 2 - 1; + const rx1 = ((dx + dw) / imgW) * 2 - 1; + const ry0 = 1 - ((dy + dh) / imgH) * 2; // bottom edge (lower clip y) + const ry1 = 1 - (dy / imgH) * 2; // top edge (upper clip y) + return new Float32Array([rx0, ry0, rx1, ry1, u0, v0, u1, v1]); + } + + // Uniform for the BASE pass: the overview/full frame over the fit-rect honouring + // zoom/pan (NOT the detail — that's a separate overlay pass now, so the base + // always fills the whole view and the crisp tile stitches on top). + function _imageDrawUniform(st, imgW, imgH) { + const fr = _imgFitRect(st.image_width, st.image_height, imgW, imgH); + const iw = st.image_width, ih = st.image_height; + const zoom = st.zoom || 1.0; + const cx = (st.center_x == null ? 0.5 : st.center_x); + const cy = (st.center_y == null ? 0.5 : st.center_y); + let dx = fr.x, dy = fr.y, dw = fr.w, dh = fr.h; // dest rect in px + let u0 = 0, v0 = 0, u1 = 1, v1 = 1; // full texture + if (zoom >= 1.0) { + // Zoomed in: sample a window of the image over the full fit-rect. + const visW = iw / zoom, visH = ih / zoom; + const srcX = Math.max(0, Math.min(iw - visW, cx * iw - visW / 2)); + const srcY = Math.max(0, Math.min(ih - visH, cy * ih - visH / 2)); + u0 = srcX / iw; u1 = (srcX + visW) / iw; + v0 = srcY / ih; v1 = (srcY + visH) / ih; + } else { + // Zoomed out: shrink the dest rect proportionally, keep centred. + dw = fr.w * zoom; dh = fr.h * zoom; + dx = fr.x + (fr.w - dw) / 2; dy = fr.y + (fr.h - dh) / 2; + } + return _drawUniform(dx, dy, dw, dh, u0, v0, u1, v1, imgW, imgH); + } + + // Uniform for the DETAIL overlay pass. Two cases: + // • fully covers (_detailUV): draw the tile over the WHOLE fit-rect, sampling the + // tile-relative UV of the visible window (crisp zoom-in) — hides the base. + // • partial (_detailOverlayRect): draw the tile at its own screen rect over the + // base (crisp centre + overview margin on a zoom-out, no flash). + // Returns null when there's no detail tile to overlay. + function _detailDrawUniform(st, imgW, imgH) { + const fr = _imgFitRect(st.image_width, st.image_height, imgW, imgH); + const vr = _imgVisibleRect2d(st, imgW, imgH); + const det = _detailUV(st); + if (det) { + return _drawUniform(vr.x, vr.y, vr.w, vr.h, + det.u0, det.v0, det.u1, det.v1, imgW, imgH); + } + const ov = _detailOverlayRect(st, fr.x, fr.y, fr.w, fr.h); + if (ov) { + const cl = _clipRectUv(ov.dx, ov.dy, ov.dw, ov.dh, + 0, 0, 1, 1, vr.x, vr.y, vr.w, vr.h); + if (!cl) return null; + return _drawUniform(cl.dx, cl.dy, cl.dw, cl.dh, + cl.u0, cl.v0, cl.u1, cl.v1, imgW, imgH); + } + return null; + } + + // Draw the image on the GPU canvas. Returns false on any failure so the caller + // reverts to Canvas2D for this frame. No clim uniform: the LUT already bakes in + // the display window + scale_mode (see _GPU_IMAGE_WGSL) — the shader is a plain + // identity lookup. + function _gpuDraw2dImage(p, st, imgW, imgH) { + const g = p._gpuImg; + try { + if (!_gpuUploadImage(p, st)) { _tiledbg('gpu', 'base upload FAILED → canvas fallback'); return false; } + const device = g.device; + // BASE pass: the overview/full frame over the fit-rect, honouring zoom/pan, so + // it lines up with the Canvas2D blit + overlays. Always drawn — the crisp tile + // stitches on top, and the base shows through any margin the tile doesn't cover. + device.queue.writeBuffer(g.uniformBuf, 0, _imageDrawUniform(st, imgW, imgH)); + // DETAIL pass (optional): the hi-res tile over its screen rect (full fit-rect + // when it covers the view; its own padded-FOV rect on a zoom-out). Prepared + // BEFORE the encoder so a decode failure just skips the overlay (base still + // draws) rather than aborting the whole frame to the Canvas2D fallback. + let drawDetail = false; + const du = _detailDrawUniform(st, imgW, imgH); + const _detUpload = du ? _gpuUploadDetail(p, st) : false; + if (du && _detUpload) { + device.queue.writeBuffer(g.uniformBuf2, 0, du); + drawDetail = true; + } + if (globalThis.__apl_tiledbg) { + const reg = st.detail_region || []; + const uv = _detailUV(st); + _tiledbg('gpu', `DRAW zoom=${(st.zoom||1).toFixed(2)} ` + + `center=(${(st.center_x||0.5).toFixed(3)},${(st.center_y||0.5).toFixed(3)}) ` + + `base[${st.base_width}x${st.base_height}] image[${st.image_width}x${st.image_height}] ` + + `detail_region=[${reg.join(',')}] detailUV=${uv?'COVERS(full)':'partial/none'} ` + + `drawDetail=${drawDetail} detUpload=${_detUpload} ` + + `display=(${st.display_min},${st.display_max})`); + } + // Clear to the canvas background so the letterbox bars match the Canvas2D + // path (which leaves the plotCanvas bg showing outside the fit-rect). + const bg = _clearRgba(theme.bgCanvas); + const enc = device.createCommandEncoder(); + const view = g.ctx.getCurrentTexture().createView(); + const pass = enc.beginRenderPass({ + colorAttachments: [{ view, loadOp: 'clear', storeOp: 'store', + clearValue: bg }] }); + pass.setPipeline(g.pipeline); + pass.setBindGroup(0, g.bindGroup); + pass.draw(6); + if (drawDetail) { + // Second pass on the SAME target, loadOp:'load' to keep the base underneath. + pass.end(); + const pass2 = enc.beginRenderPass({ + colorAttachments: [{ view, loadOp: 'load', storeOp: 'store' }] }); + pass2.setPipeline(g.pipeline); + pass2.setBindGroup(0, g.dbindGroup); + pass2.draw(6); + pass2.end(); + } else { + pass.end(); + } + device.queue.submit([enc.finish()]); + return true; + } catch (e) { + console.warn('[anyplotlib] GPU image draw failed — canvas fallback:', e); + return false; + } + } + + // '#rrggbb' → {r,g,b,a} floats 0..1 for a WebGPU clearValue. + function _clearRgba(hex) { + const h = (hex || '#000000').replace('#', ''); + const n = parseInt(h.length === 3 + ? h.split('').map(c => c + c).join('') : h, 16); + return { r: ((n >> 16) & 255) / 255, g: ((n >> 8) & 255) / 255, + b: (n & 255) / 255, a: 1 }; + } + + // Test hook: render an active 2-D image panel into an OFFSCREEN RGBA texture + // (not the live swapchain, which reads black under automation) and copy it back + // to CPU. Returns {w,h,px:[[r,g,b],...]} on an NxN sample grid so a test can + // verify the shader-LUT actually produced the expected colormapped output. + async function _gpuReadbackImage(panelId, N) { + N = N || 24; + const p = panels.get(panelId); + if (!p || p._gpu !== 'active' || !p._gpuImg) return null; + const g = p._gpuImg, device = g.device, st = p.state; + if (!_gpuUploadImage(p, st)) return null; + // Fill the whole NxN readback target (rect = full clip space) sampling the + // FULL texture (uv 0..1) so the readback reads the entire image regardless of + // aspect or the live zoom — the shader is a plain LUT identity lookup (clim + // baked into LUT), so this reads the true colormapped output. + device.queue.writeBuffer(g.uniformBuf, 0, + new Float32Array([-1, -1, 1, 1, 0, 0, 1, 1])); + // Offscreen target at NxN (a coarse but faithful downscale via the sampler). + const tex = device.createTexture({ + size: [N, N, 1], format: g.fmt, + usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.COPY_SRC }); + const enc = device.createCommandEncoder(); + const pass = enc.beginRenderPass({ + colorAttachments: [{ view: tex.createView(), loadOp: 'clear', + storeOp: 'store', clearValue: { r: 0, g: 0, b: 0, a: 1 } }] }); + pass.setPipeline(g.pipeline); pass.setBindGroup(0, g.bindGroup); pass.draw(6); pass.end(); + const bpr = Math.ceil(N * 4 / 256) * 256; + const buf = device.createBuffer({ + size: bpr * N, usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ }); + enc.copyTextureToBuffer({ texture: tex }, { buffer: buf, bytesPerRow: bpr }, [N, N, 1]); + device.queue.submit([enc.finish()]); + await buf.mapAsync(GPUMapMode.READ); + const data = new Uint8Array(buf.getMappedRange()).slice(); + buf.unmap(); buf.destroy(); tex.destroy(); + const isBgra = g.fmt.startsWith('bgra'); + const px = []; + for (let r = 0; r < N; r++) for (let c = 0; c < N; c++) { + const o = r * bpr + c * 4; + px.push(isBgra ? [data[o+2], data[o+1], data[o]] + : [data[o], data[o+1], data[o+2]]); + } + return { w: N, h: N, px }; + } + try { globalThis.__apl_gpuReadback = _gpuReadbackImage; } catch (_) {} + + // Test hook: set zoom/center on a 2-D image panel and redraw, so an automated + // test can verify the GPU path stays active + upsamples when zoomed (mirrors the + // wheel handler, without a synthetic wheel event). + try { + globalThis.__apl_setZoom = function (panelId, zoom, cx, cy) { + const p = panels.get(panelId); + if (!p || !p.state) return false; + p.state.zoom = zoom; + if (cx != null) p.state.center_x = cx; + if (cy != null) p.state.center_y = cy; + const _t0 = performance.now(); + try { draw2d(p); } catch (_) {} + const _tDraw = performance.now() - _t0; + // Compare the wheel handler's post-draw serialise: the OLD cost + // (JSON.stringify of the whole geom-polluted state) vs the NEW cost + // (_viewStateJson, geom stripped). On the binary path the old cost is a + // Uint8Array → {"0":..} stringify (millions of keys); the new one is tiny. + const _t1 = performance.now(); + let _slen = 0; + try { _slen = JSON.stringify(p.state).length; } catch (_) { _slen = -1; } + const _tStringify = performance.now() - _t1; + const _t2 = performance.now(); + let _vlen = 0; + try { _vlen = _viewStateJson(p).length; } catch (_) { _vlen = -1; } + const _tView = performance.now() - _t2; + globalThis.__apl_zoom_dbg = { draw: _tDraw, + stringify: _tStringify, len: _slen, // OLD (full state) + viewStringify: _tView, viewLen: _vlen, // NEW (geom stripped) + hasBytes: !!(p.state && p.state.image_b64_bytes), + b64len: (p.state && p.state.image_b64 && p.state.image_b64.length) || 0 }; + return _tDraw; // draw2d ms (a zoom-only redraw) + }; + // Test hook: return the EXACT string the interaction handlers now write to + // the light `panel__json` trait (geometry stripped). Lets a test assert + // the heavy geom keys never leak into the per-tick view write. + globalThis.__apl_viewStateJson = function (panelId) { + const p = panels.get(panelId); + if (!p) return null; + try { return _viewStateJson(p); } catch (_) { return null; } + }; + // Test hook: the image-px → canvas-px transform for a 2-D panel at its + // CURRENT zoom/center (the same _imgToCanvas2d the markers/widgets use), + // so a test can aim the mouse at a widget/feature under any view. + globalThis.__apl_imgToCanvas = function (panelId, ix, iy) { + const p = panels.get(panelId); + if (!p || !p.state || p.kind !== '2d') return null; + const imgW = p.imgW || Math.max(1, p.pw - PAD_L - PAD_R); + const imgH = p.imgH || Math.max(1, p.ph - PAD_T - PAD_B); + try { return _imgToCanvas2d(ix, iy, p.state, imgW, imgH); } + catch (_) { return null; } + }; + // Test hook: pixel-freshness diagnostic for a 2-D panel — which bytes does + // the state hold vs which bitmap/texture is cached? Distinguishes "stale + // bytes arrived" from "fresh bytes, stale render cache". + globalThis.__apl_panelDiag = function (panelId) { + const p = panels.get(panelId); + if (!p || !p.state || p.kind !== '2d') return null; + const st = p.state; + const img = _imageBytes(st); + const fp = (u8) => { + if (!u8) return null; + const n = u8.length; + let s = 0; + for (let i = 0; i < n; i += Math.max(1, n >> 6)) s = (s * 31 + u8[i]) >>> 0; + return `${n}:${s}`; + }; + return { + token: String(st.image_b64 || '').slice(0, 24), + hasBytes: !!st.image_b64_bytes, + bytesFp: fp(img.bytes), + base: [st.base_width, st.base_height], + logical: [st.image_width, st.image_height], + detail_region: st.detail_region || [], + detail_seq: st.detail_seq || 0, + geomRev: st._geom_rev, + blitKey: p.blitCache && String(p.blitCache.bytesKey || '').slice(0, 24), + blitWH: p.blitCache && [p.blitCache.w, p.blitCache.h], + gpuBytesKey: p._gpuImg && String(p._gpuImg.bytesKey || '').slice(0, 24), + gpu: p._gpu, + }; + }; + // Tile debug: __apl_tileDebug(true) then pan/zoom, then __apl_tileDump() to read + // the ring buffer of detail-tile decisions (detailUV vs FALLBACK->base) + the + // window/region/uv so we can see exactly where a pan snaps or inverts. + globalThis.__apl_tileDebug = function (on) { globalThis.__APL_TILE_DEBUG = !!on; + if (on) globalThis.__apl_tileDbg = []; return globalThis.__APL_TILE_DEBUG; }; + globalThis.__apl_tileDump = function () { return globalThis.__apl_tileDbg || []; }; + } catch (_) {} + function _gpuInitPanel(p, device, geom) { const fmt = navigator.gpu.getPreferredCanvasFormat(); const ctx = p.gpuCanvas.getContext('webgpu'); @@ -2379,6 +3404,22 @@ fn fs(in : VsOut) -> @location(0) vec4 { p._gpuObj = null; } + // Free the 2-D image GPU resources (R8 image texture can be tens of MB). Call on + // panel removal / figure dispose / device loss so repeatedly opening + closing + // large-image panels (an in-situ movie viewer) doesn't leak GPU memory. + function _gpuDisposeImagePanel(p) { + const g = p._gpuImg; + if (!g) return; + try { + g.tex && g.tex.destroy(); + g.dtex && g.dtex.destroy(); + g.lutTex && g.lutTex.destroy(); + g.uniformBuf && g.uniformBuf.destroy(); + g.uniformBuf2 && g.uniformBuf2.destroy(); + } catch (_) {} + p._gpuImg = null; + } + function draw3d(p, _retry) { const st = p.state; if (!st) return; if (!_retry) p._gpuFellBack = false; // per-render re-entry guard reset @@ -3015,6 +4056,30 @@ fn fs(in : VsOut) -> @location(0) vec4 { model.save_changes(); } + // Debounced 'view_changed' event: fires to Python after zoom/pan SETTLES (~90 ms + // idle) with the current view rect, so a viewport-LOD consumer can fetch a hi-res + // detail tile for exactly the visible region (see Plot2D.set_detail). Debounced so + // a fast wheel/drag doesn't flood Python — only the final resting view is sent. + const _viewChangeTimers = {}; + function _emitViewChanged(p, immediate) { + const st = p.state; if (!st) return; + const send = () => { + _viewChangeTimers[p.id] = null; + const s = p.state; if (!s) return; + const dpr = window.devicePixelRatio || 1; + _emitEvent(p.id, 'view_changed', null, { + zoom: s.zoom, center_x: s.center_x, center_y: s.center_y, + image_width: s.image_width, image_height: s.image_height, + // Panel device px → the tile consumer sizes the tile to what fits on screen. + display_width: Math.round((p.imgW || 0) * dpr), + display_height: Math.round((p.imgH || 0) * dpr), + }); + }; + if (_viewChangeTimers[p.id]) clearTimeout(_viewChangeTimers[p.id]); + if (immediate) { send(); return; } + _viewChangeTimers[p.id] = setTimeout(send, 90); + } + function _modifiers(e) { const mods = []; if (e.ctrlKey) mods.push("ctrl"); @@ -3044,7 +4109,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { // panel listener echo — see the _selfWrite guard in _createPanelDOM. function _writeState() { p._selfWrite = true; - try { model.set(`panel_${p.id}_json`, JSON.stringify(p.state)); } + try { model.set(`panel_${p.id}_json`, _viewStateJson(p)); } finally { p._selfWrite = false; } } @@ -3919,23 +4984,39 @@ fn fs(in : VsOut) -> @location(0) vec4 { function _markerHitTest2d(mx, my, st, pw, ph) { const sets = st.markers || []; const scale = _imgScale2d(st, pw, ph); + const vr = _imgVisibleRect2d(st, pw, ph); for (let si = sets.length-1; si >= 0; si--) { const ms = sets[si]; const type = ms.type || 'circles'; + const tfm = ms.transform || 'data'; + const clipSet = tfm !== 'display' || ms.clip_display !== false; + if (clipSet && (mx < vr.x || mx > vr.x + vr.w || my < vr.y || my > vr.y + vr.h)) { + continue; + } const collLabel = ms.label != null ? String(ms.label) : null; const perLabels = Array.isArray(ms.labels) ? ms.labels : null; + let _tc; + if(tfm==='axes'){ + const fr=_imgFitRect(st.image_width,st.image_height,pw,ph); + _tc=(fx,fy)=>[fr.x+fx*fr.w, fr.y+(1-fy)*fr.h]; + } else if(tfm==='display'){ + _tc=(ix,iy)=>[ix,iy]; + } else { + _tc=(ix,iy)=>_imgToCanvas2d(ix,iy,st,pw,ph); + } + const scl = tfm==='data' ? scale : 1; if (type === 'circles') { for (let i=0;i<(ms.offsets||[]).length;i++) { - const [cx,cy]=_imgToCanvas2d(ms.offsets[i][0],ms.offsets[i][1],st,pw,ph); - const r=Math.max(1,(ms.sizes[i]!=null?ms.sizes[i]:ms.sizes[0]||5)*scale); + const [cx,cy]=_tc(ms.offsets[i][0],ms.offsets[i][1]); + const r=Math.max(1,(ms.sizes[i]!=null?ms.sizes[i]:ms.sizes[0]||5)*scl); if(Math.sqrt((mx-cx)**2+(my-cy)**2)<=r+MARKER_HIT) return{si,i,collectionLabel:collLabel,markerLabel:perLabels?String(perLabels[i]??''):null}; } } else if (type === 'ellipses') { for (let i=0;i<(ms.offsets||[]).length;i++) { - const [cx,cy]=_imgToCanvas2d(ms.offsets[i][0],ms.offsets[i][1],st,pw,ph); - const rw=(ms.widths[i]||ms.widths[0]||10)*scale/2+MARKER_HIT; - const rh=(ms.heights[i]||ms.heights[0]||10)*scale/2+MARKER_HIT; + const [cx,cy]=_tc(ms.offsets[i][0],ms.offsets[i][1]); + const rw=(ms.widths[i]||ms.widths[0]||10)*scl/2+MARKER_HIT; + const rh=(ms.heights[i]||ms.heights[0]||10)*scl/2+MARKER_HIT; const dx=(mx-cx)/Math.max(1,rw), dy=(my-cy)/Math.max(1,rh); if(dx*dx+dy*dy<=1.0) return{si,i,collectionLabel:collLabel,markerLabel:perLabels?String(perLabels[i]??''):null}; @@ -3943,24 +5024,24 @@ fn fs(in : VsOut) -> @location(0) vec4 { } else if (type === 'rectangles' || type === 'squares') { const heights = type==='squares' ? ms.widths : ms.heights; for (let i=0;i<(ms.offsets||[]).length;i++) { - const [cx,cy]=_imgToCanvas2d(ms.offsets[i][0],ms.offsets[i][1],st,pw,ph); - const hw=(ms.widths[i]||ms.widths[0]||20)*scale/2+MARKER_HIT; - const hh=((heights[i]||heights[0]||20))*scale/2+MARKER_HIT; + const [cx,cy]=_tc(ms.offsets[i][0],ms.offsets[i][1]); + const hw=(ms.widths[i]||ms.widths[0]||20)*scl/2+MARKER_HIT; + const hh=((heights[i]||heights[0]||20))*scl/2+MARKER_HIT; if(Math.abs(mx-cx)<=hw&&Math.abs(my-cy)<=hh) return{si,i,collectionLabel:collLabel,markerLabel:perLabels?String(perLabels[i]??''):null}; } } else if (type === 'arrows') { for (let i=0;i<(ms.offsets||[]).length;i++) { - const [x1,y1]=_imgToCanvas2d(ms.offsets[i][0],ms.offsets[i][1],st,pw,ph); - const mx2=x1+(ms.U[i]||0)*scale/2, my2=y1+(ms.V[i]||0)*scale/2; + const [x1,y1]=_tc(ms.offsets[i][0],ms.offsets[i][1]); + const mx2=x1+(ms.U[i]||0)*scl/2, my2=y1+(ms.V[i]||0)*scl/2; if(Math.sqrt((mx-mx2)**2+(my-my2)**2)<=MARKER_HIT*2) return{si,i,collectionLabel:collLabel,markerLabel:perLabels?String(perLabels[i]??''):null}; } } else if (type === 'lines') { for (let i=0;i<(ms.segments||[]).length;i++) { const seg=ms.segments[i]; - const [x1,y1]=_imgToCanvas2d(seg[0][0],seg[0][1],st,pw,ph); - const [x2,y2]=_imgToCanvas2d(seg[1][0],seg[1][1],st,pw,ph); + const [x1,y1]=_tc(seg[0][0],seg[0][1]); + const [x2,y2]=_tc(seg[1][0],seg[1][1]); const dx=x2-x1,dy=y2-y1,len2=dx*dx+dy*dy; const t=len2>0?Math.max(0,Math.min(1,((mx-x1)*dx+(my-y1)*dy)/len2)):0; if(Math.sqrt((mx-(x1+t*dx))**2+(my-(y1+t*dy))**2)<=MARKER_HIT) @@ -3969,13 +5050,13 @@ fn fs(in : VsOut) -> @location(0) vec4 { } else if (type === 'polygons') { for (let i=0;i<(ms.vertices_list||[]).length;i++) { const verts=ms.vertices_list[i]; if(!verts||!verts.length) continue; - const [cx,cy]=_imgToCanvas2d(verts[0][0],verts[0][1],st,pw,ph); + const [cx,cy]=_tc(verts[0][0],verts[0][1]); if(Math.sqrt((mx-cx)**2+(my-cy)**2)<=MARKER_HIT*1.5) return{si,i,collectionLabel:collLabel,markerLabel:perLabels?String(perLabels[i]??''):null}; } } else if (type === 'texts') { for (let i=0;i<(ms.offsets||[]).length;i++) { - const [cx,cy]=_imgToCanvas2d(ms.offsets[i][0],ms.offsets[i][1],st,pw,ph); + const [cx,cy]=_tc(ms.offsets[i][0],ms.offsets[i][1]); if(Math.sqrt((mx-cx)**2+(my-cy)**2)<=MARKER_HIT*1.5) return{si,i,collectionLabel:collLabel,markerLabel:perLabels?String(perLabels[i]??''):null}; } @@ -4192,22 +5273,32 @@ fn fs(in : VsOut) -> @location(0) vec4 { e.preventDefault(); const st=p.state; if(!st) return; const imgW=p.imgW||Math.max(1,p.pw-PAD_L-PAD_R), imgH=p.imgH||Math.max(1,p.ph-PAD_T-PAD_B); - const {mx,my}=_clientPos(e,overlayCanvas,imgW,imgH); - // Image point under cursor before zoom change - const [anchorX,anchorY]=_canvasToImg2d(mx,my,st,imgW,imgH); - const curZ=st.zoom, newZ=Math.max(0.75,Math.min(100,curZ*(e.deltaY>0?0.9:1.1))); - st.zoom=newZ; - // Reposition center so the same image point stays under the cursor - const iw=st.image_width, ih=st.image_height; - const fr=_imgFitRect(iw,ih,imgW,imgH); - const newVisW=iw/newZ, newVisH=ih/newZ; - const newSrcX=anchorX-(mx-fr.x)/fr.w*newVisW; - const newSrcY=anchorY-(my-fr.y)/fr.h*newVisH; - st.center_x=Math.max(0,Math.min(1,(newSrcX+newVisW/2)/iw)); - st.center_y=Math.max(0,Math.min(1,(newSrcY+newVisH/2)/ih)); + const newZ=Math.max(0.75,Math.min(100,st.zoom*(e.deltaY>0?0.9:1.1))); + if(!st.tile_enabled){ + // NON-tiled image: anchor the zoom on the cursor (the image point under the + // cursor stays put) — there's no detail-tile fetch to disagree with it. + const {mx,my}=_clientPos(e,overlayCanvas,imgW,imgH); + const [anchorX,anchorY]=_canvasToImg2d(mx,my,st,imgW,imgH); + st.zoom=newZ; + const iw=st.image_width, ih=st.image_height; + const fr=_imgFitRect(iw,ih,imgW,imgH); + const newVisW=iw/newZ, newVisH=ih/newZ; + const newSrcX=anchorX-(mx-fr.x)/fr.w*newVisW; + const newSrcY=anchorY-(my-fr.y)/fr.h*newVisH; + st.center_x=Math.max(0,Math.min(1,(newSrcX+newVisW/2)/iw)); + st.center_y=Math.max(0,Math.min(1,(newSrcY+newVisH/2)/ih)); + } else { + // TILE mode: zoom about the CURRENT CENTER, not the cursor. Cursor-anchored + // zoom looked right DURING the wheel (base blit re-anchored to the cursor), + // but when the debounced detail-tile fetch landed it re-derived the visible + // window from center_x/center_y — a different anchor — so the image visibly + // JUMPED. Center-zoom makes the base blit + detail tile agree at every zoom. + st.zoom=newZ; + } draw2d(p); _propagateZoom2d(p); - model.set(`panel_${p.id}_json`, JSON.stringify(p.state)); + model.set(`panel_${p.id}_json`, _viewStateJson(p)); + _emitViewChanged(p); // debounced → fetch a hi-res detail tile on zoom settle _scheduleCommit(); },{passive:false}); @@ -4256,7 +5347,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { st.center_y=Math.max(0,Math.min(1,panStart.cy-(cmy-panStart.my)/fr.h/z)); draw2d(p); _propagateZoom2d(p); - model.set(`panel_${p.id}_json`, JSON.stringify(p.state)); + model.set(`panel_${p.id}_json`, _viewStateJson(p)); _scheduleCommit(); e.preventDefault(); }); document.addEventListener('mouseup',(e)=>{ @@ -4266,7 +5357,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { const _dw=(p.state.overlay_widgets||[])[_idx]||{}; const _did=_dw.id||null; p.ovDrag2d=null; overlayCanvas.style.cursor='default'; - model.set(`panel_${p.id}_json`, JSON.stringify(p.state)); + model.set(`panel_${p.id}_json`, _viewStateJson(p)); _emitEvent(p.id,'pointer_up',_did,{..._dw,..._pointerFields(e),button:e.button}); return; } @@ -4306,8 +5397,9 @@ fn fs(in : VsOut) -> @location(0) vec4 { // ── Normal pan settle ─────────────────────────────────────────────────── st.center_x=Math.max(0,Math.min(1,panStart.cx-(cmx-panStart.mx)/fr.w/st.zoom)); st.center_y=Math.max(0,Math.min(1,panStart.cy-(cmy-panStart.my)/fr.h/st.zoom)); - model.set(`panel_${p.id}_json`, JSON.stringify(p.state)); + model.set(`panel_${p.id}_json`, _viewStateJson(p)); _emitEvent(p.id,'pointer_up',null,{center_x:st.center_x,center_y:st.center_y,zoom:st.zoom,..._pointerFields(e),button:e.button}); + _emitViewChanged(p); // pan settled → refresh the detail tile for the new view model.save_changes(); }); @@ -4421,20 +5513,20 @@ fn fs(in : VsOut) -> @location(0) vec4 { const key=e.key.toLowerCase(); if(key==='r'){ st.zoom=1; st.center_x=0.5; st.center_y=0.5; - draw2d(p); model.set(`panel_${p.id}_json`,JSON.stringify(st)); model.save_changes(); + draw2d(p); model.set(`panel_${p.id}_json`,_viewStateJson(p)); model.save_changes(); e.stopPropagation(); e.preventDefault(); } else if(key==='c'){ st.show_colorbar=!st.show_colorbar; draw2d(p); - model.set(`panel_${p.id}_json`,JSON.stringify(st)); model.save_changes(); + model.set(`panel_${p.id}_json`,_viewStateJson(p)); model.save_changes(); e.stopPropagation(); e.preventDefault(); } else if(key==='l'){ st.scale_mode=st.scale_mode==='log'?'linear':'log'; - draw2d(p); model.set(`panel_${p.id}_json`,JSON.stringify(st)); model.save_changes(); + draw2d(p); model.set(`panel_${p.id}_json`,_viewStateJson(p)); model.save_changes(); e.stopPropagation(); e.preventDefault(); } else if(key==='s'){ st.scale_mode=st.scale_mode==='symlog'?'linear':'symlog'; - draw2d(p); model.set(`panel_${p.id}_json`,JSON.stringify(st)); model.save_changes(); + draw2d(p); model.set(`panel_${p.id}_json`,_viewStateJson(p)); model.save_changes(); e.stopPropagation(); e.preventDefault(); } }); @@ -4475,7 +5567,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { st.view_x0=nx0;st.view_x1=nx1; draw1d(p); _propagateView1d(p); - model.set(`panel_${p.id}_json`,JSON.stringify(st)); + model.set(`panel_${p.id}_json`,_viewStateJson(p)); _scheduleCommit(); },{passive:false}); @@ -4512,7 +5604,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { if(nx0<0){nx0=0;nx1=span;}if(nx1>1){nx1=1;nx0=1-span;} st.view_x0=nx0;st.view_x1=nx1; draw1d(p);_propagateView1d(p); - model.set(`panel_${p.id}_json`,JSON.stringify(st));_scheduleCommit();e.preventDefault(); + model.set(`panel_${p.id}_json`,_viewStateJson(p));_scheduleCommit();e.preventDefault(); }); document.addEventListener('mouseup',(e)=>{ settled.clear(); @@ -4523,7 +5615,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { const _dw=(p.state.overlay_widgets||[])[_idx]||{}; const _did=_dw.id||null; p.ovDrag=null; overlayCanvas.style.cursor='crosshair'; - model.set(`panel_${p.id}_json`,JSON.stringify(p.state)); + model.set(`panel_${p.id}_json`,_viewStateJson(p)); _emitEvent(p.id,'pointer_up',_did,{..._dw,..._pointerFields(e),button:e.button}); } if(p.isPanning){ @@ -4562,7 +5654,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { x:p.mouseX ?? 0, y:p.mouseY ?? 0, xdata:physX, }); - if(e.key.toLowerCase()==='r'){st.view_x0=0;st.view_x1=1;draw1d(p);model.set(`panel_${p.id}_json`,JSON.stringify(st));model.save_changes();e.stopPropagation();e.preventDefault();} + if(e.key.toLowerCase()==='r'){st.view_x0=0;st.view_x1=1;draw1d(p);model.set(`panel_${p.id}_json`,_viewStateJson(p));model.save_changes();e.stopPropagation();e.preventDefault();} }); overlayCanvas.addEventListener('keyup',(e)=>{ _emitEvent(p.id,'key_up',null,{ @@ -4817,7 +5909,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { } drawOverlay2d(p); - model.set(`panel_${p.id}_json`, JSON.stringify(st)); + model.set(`panel_${p.id}_json`, _viewStateJson(p)); model.save_changes(); e.preventDefault(); } @@ -4902,7 +5994,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { w.y=st.data_max-((clampY-r.y)/(r.h||1))*(st.data_max-st.data_min); } drawOverlay1d(p); - model.set(`panel_${p.id}_json`,JSON.stringify(st));model.save_changes(); + model.set(`panel_${p.id}_json`,_viewStateJson(p));model.save_changes(); e.preventDefault(); } @@ -4923,7 +6015,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { if(axis==='x'||axis==='both'){tp.state.zoom=st.zoom;tp.state.center_x=st.center_x;} if(axis==='y'||axis==='both'){tp.state.center_y=st.center_y;} _redrawPanel(tp); - model.set(`panel_${pid}_json`,JSON.stringify(tp.state)); + model.set(`panel_${pid}_json`,_viewStateJson(tp)); } } } @@ -4941,7 +6033,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { const tp=panels.get(pid); if(!tp||!tp.state) continue; tp.state.view_x0=st.view_x0; tp.state.view_x1=st.view_x1; _redrawPanel(tp); - model.set(`panel_${pid}_json`,JSON.stringify(tp.state)); + model.set(`panel_${pid}_json`,_viewStateJson(tp)); } } } @@ -5664,7 +6756,7 @@ fn fs(in : VsOut) -> @location(0) vec4 { const _did = _dw.id || null; p.ovDrag = null; overlayCanvas.style.cursor = 'default'; - model.set(`panel_${p.id}_json`, JSON.stringify(p.state)); + model.set(`panel_${p.id}_json`, _viewStateJson(p)); _emitEvent(p.id, 'pointer_up', _did, {..._dw, ..._pointerFields(e), button: e.button}); _scheduleCommit(); }); @@ -5969,6 +7061,8 @@ export function createLocalModel(initialState) { // opts.onEvent(ev) — parsed interaction events (pointer/key/wheel …) // opts.onSync(key, value) — raw outbound model writes, for a Python bridge export function mount(el, state, opts) { + // Diagnostic marker: proves THIS (WebGPU-2D) build of figure_esm.js is loaded. + try { globalThis.__apl_build = 'webgpu-2d'; } catch (_) {} const o = opts || {}; const model = createLocalModel(state); if (o.onSync) model._setSyncHandler(o.onSync); @@ -6000,7 +7094,13 @@ export function mount(el, state, opts) { // Remove the figure's DOM. (Window-level listeners registered by the // renderer are inert once the DOM is gone; for complete cleanup, discard // the containing element or iframe.) - dispose() { model.off(); el.replaceChildren(); }, + dispose() { + // Free GPU resources for every panel before tearing down the DOM. + try { for (const p of panels.values()) { + _gpuDisposeImagePanel(p); _gpuDisposePanel(p); + } } catch (_) {} + model.off(); el.replaceChildren(); + }, }; } diff --git a/anyplotlib/markers.py b/anyplotlib/markers.py index 3f6cc799..330c5992 100644 --- a/anyplotlib/markers.py +++ b/anyplotlib/markers.py @@ -111,6 +111,8 @@ def __init__(self, marker_type: str, name: str, kwargs: dict, push_fn, raise ValueError( f"transform must be one of {sorted(_VALID_TRANSFORMS)}, got {tfm!r}" ) + if "clip_display" in kwargs and not isinstance(kwargs["clip_display"], bool): + raise ValueError("clip_display must be a bool") self._data: dict = dict(kwargs) self._push_fn = push_fn self._parent: "MarkerTypeDict | None" = parent @@ -130,6 +132,8 @@ def set(self, **kwargs) -> None: f"transform must be one of {sorted(_VALID_TRANSFORMS)}, " f"got {kwargs['transform']!r}" ) + if "clip_display" in kwargs and not isinstance(kwargs["clip_display"], bool): + raise ValueError("clip_display must be a bool") self._data.update(kwargs) self._push_fn() @@ -361,6 +365,9 @@ def to_wire(self, group_id: str) -> dict: # ── coordinate transform (always emitted; defaults to "data") ────── wire["transform"] = d.get("transform", "data") + # Display-space markers are clipped to the visible image rect by default; + # this flag allows opting out for HUD-style annotations. + wire["clip_display"] = bool(d.get("clip_display", True)) # ── common optional fields ────────────────────────────────────────── label = d.get("label") diff --git a/anyplotlib/plot1d/_plot1d.py b/anyplotlib/plot1d/_plot1d.py index a1f411a3..2928471c 100644 --- a/anyplotlib/plot1d/_plot1d.py +++ b/anyplotlib/plot1d/_plot1d.py @@ -882,7 +882,8 @@ def add_circles(self, offsets, name=None, *, radius=5, linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add circle markers at explicit (x, y) positions. On 1-D panels circles are rendered as filled/stroked discs; *radius* @@ -924,14 +925,16 @@ def add_circles(self, offsets, name=None, *, radius=5, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_points(self, offsets, name=None, *, sizes=5, color="#ff0000", facecolors=None, linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add point markers at (x, y) positions in data coordinates. Parameters @@ -967,13 +970,15 @@ def add_points(self, offsets, name=None, *, sizes=5, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_hlines(self, y_values, name=None, *, color="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add static horizontal lines spanning the full x range. Parameters @@ -1001,13 +1006,15 @@ def add_hlines(self, y_values, name=None, *, color=color, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_vlines(self, x_values, name=None, *, color="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add static vertical lines spanning the full y range. Parameters @@ -1035,13 +1042,15 @@ def add_vlines(self, x_values, name=None, *, color=color, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_arrows(self, offsets, U, V, name=None, *, edgecolors="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add arrow markers at explicit (x, y) positions. Parameters @@ -1071,14 +1080,16 @@ def add_arrows(self, offsets, U, V, name=None, *, edgecolors=edgecolors, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_ellipses(self, offsets, widths, heights, name=None, *, angles=0, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add ellipse markers at explicit (x, y) positions. Parameters @@ -1117,13 +1128,15 @@ def add_ellipses(self, offsets, widths, heights, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_lines(self, segments, name=None, *, edgecolors="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add line-segment markers (static, not draggable). Parameters @@ -1151,14 +1164,16 @@ def add_lines(self, segments, name=None, *, edgecolors=edgecolors, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_rectangles(self, offsets, widths, heights, name=None, *, angles=0, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add rectangle markers at explicit (x, y) positions. Parameters @@ -1197,14 +1212,16 @@ def add_rectangles(self, offsets, widths, heights, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_squares(self, offsets, widths, name=None, *, angles=0, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add square markers at explicit (x, y) positions. Parameters @@ -1243,14 +1260,16 @@ def add_squares(self, offsets, widths, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_polygons(self, vertices_list, name=None, *, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, clip_path=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add polygon markers defined by explicit vertex lists. Parameters @@ -1289,11 +1308,13 @@ def add_polygons(self, vertices_list, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, clip_path=clip_path, - transform=transform) + transform=transform, + clip_display=clip_display) def add_raster(self, rgba, *, extent, name=None, clip_path=None, smooth: bool = False, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add an RGBA image drawn between data-coordinate ``extent`` corners. A single-``drawImage`` raster — the fast path for a dense regular grid @@ -1332,13 +1353,15 @@ def add_raster(self, rgba, *, extent, name=None, clip_path=None, image_width=int(w), image_height=int(h), extent=tuple(float(v) for v in extent), clip_path=clip_path, smooth=bool(smooth), - transform=transform) + transform=transform, + clip_display=clip_display) def add_texts(self, offsets, texts, name=None, *, color="#ff0000", fontsize=12, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add text annotations at explicit (x, y) positions. Parameters @@ -1368,7 +1391,8 @@ def add_texts(self, offsets, texts, name=None, *, color=color, fontsize=fontsize, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def __repr__(self) -> str: n = len(self._state.get("data", [])) diff --git a/anyplotlib/plot2d/_plot2d.py b/anyplotlib/plot2d/_plot2d.py index cd970f43..b0f7e399 100644 --- a/anyplotlib/plot2d/_plot2d.py +++ b/anyplotlib/plot2d/_plot2d.py @@ -6,9 +6,20 @@ from __future__ import annotations +import logging +import os import numpy as np from typing import Callable +# Tile-mode diagnostics. WARNING level so it reaches stderr / the SpyDE Log panel +# without turning on DEBUG. Grep the app log for "[TILEDBG]". +_TLOG = logging.getLogger("anyplotlib.tile") + +# Set APL_TILE_DEBUG=1 to log each tiled zoom/pan decision (region fetched) to the +# backend logger — reads to the terminal so a snap/inverted-pan can be diagnosed +# without the browser console. Off by default (zero cost). +_TILE_DEBUG = os.environ.get("APL_TILE_DEBUG") == "1" + from anyplotlib._base_plot import _BasePlot, _PanelMixin, _MarkerMixin from anyplotlib.markers import MarkerRegistry from anyplotlib.callbacks import CallbackRegistry @@ -20,6 +31,18 @@ from anyplotlib._utils import _normalize_image, _build_colormap_lut, _to_rgba_u8 +def _binary_transport_active() -> bool: + """True when the Electron binary pixel transport is enabled. + + Reads ``APL_BINARY_TRANSPORT`` from the ENVIRONMENT fresh each call (not a + cached module global) so it tracks the live setting — and so a test that + toggles the env var is honoured without a stale reloaded-module flag leaking + across tests. Only SpyDE / the Electron host sets it; every other host + (Jupyter / Pyodide / standalone / ``save_html``) leaves it unset → base64.""" + import os + return os.environ.get("APL_BINARY_TRANSPORT") == "1" + + class Plot2D(_BasePlot, _PanelMixin, _MarkerMixin): """2-D image plot panel. @@ -32,15 +55,37 @@ class Plot2D(_BasePlot, _PanelMixin, _MarkerMixin): """ #: Heavy state keys routed to the geometry channel (see Figure._push). - #: ``colormap_data`` is large and only changes on set_colormap. - _GEOM_KEYS = frozenset({"image_b64", "colormap_data", "overlay_mask_b64"}) - - def __init__(self, data: np.ndarray, + #: ``colormap_data`` is large and only changes on set_colormap. ``detail_b64`` + #: (the zoom detail tile) MUST be here: it's a large pixel blob, and only geom- + #: channel keys are eligible for the PLOTBIN binary route in _route_change — if it + #: rode the light view trait instead, its "\x00bin:" token would never be resolved + #: to bytes and the crisp zoom tile would never render (only the overview shows). + _GEOM_KEYS = frozenset({"image_b64", "colormap_data", "overlay_mask_b64", + "detail_b64"}) + + # Logical image bigger than this (either side) uses the tile backend under + # tile="auto": it sends a downsampled OVERVIEW as the base + streams a hi-res + # detail tile of the visible region on zoom/pan, instead of the whole frame. + TILE_THRESHOLD = 1024 + OVERVIEW_MAX = 1024 # initial overview edge (refined to panel px on first view) + VIEW_OVERFETCH = 2.0 # sample 2× the visible area (a full extra viewport of + # padding) so casual zoom-out/pan stays on the crisp tile + # and the overview only shows past that padded FOV + VIEW_ZOOM_MIN = 1.05 # below this the overview base is enough (no tile) + RANGE_SAMPLE_MAX = 2048 # subsample edge for the tile display-range probe (native + # pixels → true extremes, so a zoom tile doesn't blow out) + + def __init__(self, data, x_axis=None, y_axis=None, units: str = "px", cmap: str | None = None, vmin: float | None = None, vmax: float | None = None, - origin: str = "upper"): + origin: str = "upper", + gpu: "str | bool" = "auto", + tile: "str | bool" = "auto", + integration_method: str = "mean", + overview_method: str = "mean", + tile_backend=None): self._id: str = "" # assigned by Axes._attach self._fig: object = None # assigned by Axes._attach @@ -51,6 +96,54 @@ def __init__(self, data: np.ndarray, ) self._origin: str = origin + # ── Tile backend resolution ────────────────────────────────────────────── + # A backend OWNS the source + sampling; a bare ndarray is wrapped. Decide + # tile mode from the LOGICAL shape (so we never materialise a huge array just + # to measure it). When tiled, `data` below becomes a downsampled OVERVIEW and + # the logical full size is remembered for the coordinate system. + from anyplotlib.plot2d._tile_backend import as_tile_backend, TileBackend + self._integration_method = integration_method + # Sampling method used for the always-visible base OVERVIEW texture. + # Defaults to "mean" so sparse images are integration-consistent between + # overview and detail tiles — a "subsample" overview would show very + # different intensities on a sparse image compared to a "mean" detail + # tile, producing a visible shift on zoom-in. Pass + # overview_method="subsample" to restore the old fast path when a full- + # frame area-mean is too expensive (e.g. 16 MP+ frames on every scrub). + self._overview_method: str = overview_method + self._tile_backend = None + self._detail_pending = None # (id) latest requested tile — latest-wins + src = tile_backend if tile_backend is not None else data + is_backend = isinstance(src, TileBackend) and not isinstance(src, np.ndarray) + if is_backend: + logical_h, logical_w = src.full_shape + else: + _arr = np.asarray(src) + logical_h, logical_w = _arr.shape[:2] if _arr.ndim >= 2 else (0, 0) + _rgb_src = (not is_backend) and np.asarray(src).ndim == 3 + tile_on = (not _rgb_src) and ( + tile is True or (tile == "auto" and max(logical_h, logical_w) > self.TILE_THRESHOLD)) + + # Remember the tile PREFERENCE so a later set_data can auto-enable tiling when + # a large frame arrives on a plot that started small (the live-navigator case: + # imshow a tiny placeholder, then set_data the real 4k frames). tile=False + # stays off forever; "auto"/True honour the threshold per frame. + self._tile_pref = tile + self._tile_on = tile_on + self._logical_w = logical_w + self._logical_h = logical_h + if tile_on: + self._tile_backend = as_tile_backend( + src, origin=origin) if not is_backend else src + # Base texture = a downsampled OVERVIEW (row 0 at top, already oriented); + # the logical full size drives image_width/height + the coordinate system. + data = self._make_overview(self._tile_backend) + # The overview is already display-oriented; don't let the origin='lower' + # flip below double-flip it (the y_axis still reverses for correct ticks). + self._overview_pre_oriented = True + else: + self._overview_pre_oriented = False + data = np.asarray(data) # (H, W, 3|4) arrays render as true-colour RGB(A); anything else 2-D. self._is_rgb: bool = data.ndim == 3 and data.shape[2] in (3, 4) @@ -63,12 +156,15 @@ def __init__(self, data: np.ndarray, f"got {data.shape}") h, w = data.shape[:2] + # In tile mode `data` is the overview; the LOGICAL size drives the axes/extent. + axis_w = self._logical_w if self._tile_on else w + axis_h = self._logical_h if self._tile_on else h # origin='lower' — row 0 at the bottom, matching matplotlib's matrix # convention. Flip the data so our renderer (which always draws row 0 # at the top) shows the correct orientation, and reverse the y-axis so - # tick values increase upward. - if origin == "lower": + # tick values increase upward. (A tile overview is already oriented.) + if origin == "lower" and not self._overview_pre_oriented: data = np.flipud(data) self._data: np.ndarray = data.astype(float) @@ -76,21 +172,34 @@ def __init__(self, data: np.ndarray, x_axis_given = x_axis is not None y_axis_given = y_axis is not None if x_axis is None: - x_axis = np.arange(w, dtype=float) + x_axis = np.arange(axis_w, dtype=float) if y_axis is None: - y_axis = np.arange(h, dtype=float) + y_axis = np.arange(axis_h, dtype=float) x_axis = np.asarray(x_axis, dtype=float) y_axis = np.asarray(y_axis, dtype=float) if origin == "lower": y_axis = y_axis[::-1] + # Tile mode with no explicit clim: derive the display range from the FULL-RES + # data (a native-pixel subsample), NOT the averaged overview. The overview's + # min/max are pulled toward the mean, so quantising a near-native zoom DETAIL + # tile over that narrow range clips the true extremes → the region goes white + # on zoom-in. Using the full-res range keeps base + detail on one honest range. + tile_clim = None + if (self._tile_on and not self._is_rgb + and vmin is None and vmax is None): + tile_clim = self._backend_display_range(self._tile_backend) + if self._is_rgb: # True-colour path: bytes go to JS as RGBA; no LUT applies. img_u8 = _to_rgba_u8(data) raw_vmin, raw_vmax = 0.0, 255.0 else: - img_u8, raw_vmin, raw_vmax = _normalize_image(data) + # Quantise the overview base over the full-res range (tile_clim) so its + # codes line up with the detail tile's; falls back to the overview's own + # min/max when the probe couldn't run. + img_u8, raw_vmin, raw_vmax = _normalize_image(data, clim=tile_clim) self._raw_u8 = img_u8 self._raw_vmin = raw_vmin self._raw_vmax = raw_vmax @@ -102,18 +211,33 @@ def __init__(self, data: np.ndarray, disp_min = float(vmin) if vmin is not None else raw_vmin disp_max = float(vmax) if vmax is not None else raw_vmax - # Compute physical pixel scale (data-units per pixel) from axis arrays - scale_x = float(abs(x_axis[-1] - x_axis[0]) / max(w - 1, 1)) if len(x_axis) >= 2 else 1.0 - scale_y = float(abs(y_axis[-1] - y_axis[0]) / max(h - 1, 1)) if len(y_axis) >= 2 else 1.0 + # Compute physical pixel scale (data-units per pixel) from axis arrays over + # the LOGICAL image size (the axes span the full image, not the overview). + scale_x = float(abs(x_axis[-1] - x_axis[0]) / max(axis_w - 1, 1)) if len(x_axis) >= 2 else 1.0 + scale_y = float(abs(y_axis[-1] - y_axis[0]) / max(axis_h - 1, 1)) if len(y_axis) >= 2 else 1.0 + + # WebGPU image path: "auto" (GPU above ~1 Mpx), True (force attempt), + # False/"off" (never). Maps to the JS gpu_mode gate. + _gpu_mode = ("always" if gpu is True + else "off" if gpu in (False, "off") + else "auto") self._state: dict = { "kind": "2d", "is_mesh": False, "is_rgb": self._is_rgb, + "gpu_mode": _gpu_mode, + "gpu_active": False, "has_axes": x_axis_given or y_axis_given, "image_b64": self._encode_bytes(img_u8), - "image_width": w, - "image_height": h, + # LOGICAL full-image size (zoom math + detail_region are in these px). In + # tile mode the base texture (image_b64) is a smaller OVERVIEW whose real + # pixel dims are base_width/height (0 → base == image size). + "image_width": axis_w, + "image_height": axis_h, + "base_width": (w if self._tile_on else 0), + "base_height": (h if self._tile_on else 0), + "tile_enabled": self._tile_on, "x_axis": x_axis.tolist(), "y_axis": y_axis.tolist(), "units": units, @@ -130,6 +254,17 @@ def __init__(self, data: np.ndarray, "zoom": 1.0, "center_x": 0.5, "center_y": 0.5, + # Detail tile: a HIGHER-RES texture for a logical sub-region of the + # image, sampled by the shader when the current zoom window is inside + # detail_region — so a zoom-in shows true native pixels for the visible + # area WITHOUT transferring the whole full-res frame. Set via set_detail; + # "" / [] clears it (revert to the base texture). See set_detail. + "detail_b64": "", + "detail_region": [], # [x0, x1, y0, y1] image-pixel rect of the base + "detail_width": 0, + "detail_height": 0, + "detail_seq": 0, # bumped per set_detail so the renderer re-uploads + # a re-sampled tile even at the same size/region "overlay_widgets": [], "markers": [], "pointer_settled_ms": 0, @@ -157,20 +292,414 @@ def __init__(self, data: np.ndarray, self.markers = MarkerRegistry(self._push_markers, allowed=MarkerRegistry._KNOWN_2D) self.callbacks = CallbackRegistry() + # Tile mode: anyplotlib itself reacts to view_changed (zoom/pan) → sample a + # hi-res detail tile of the visible region from the backend. The consumer + # does nothing; it can still add its OWN view_changed handler (they coexist). + if self._tile_on: + self.callbacks.connect("view_changed", self._on_view_changed_internal) self._widgets: dict[str, Widget] = {} + # Set True once the JS side reports the WebGPU image path activated for + # this panel (via the gpu_status event → _set_gpu_active). Reflects the + # actual render path, not just the requested gpu_mode. + self._gpu_active: bool = False + + @property + def gpu_active(self) -> bool: + """True when this image is being rendered by the WebGPU path (reported by + JS after the device resolves and the panel flips to GPU). ``False`` on the + Canvas2D path (small image, gpu=False, no GPU, or a GPU failure/fallback).""" + return self._gpu_active + + def _set_gpu_active(self, active: bool) -> None: + """Internal: called from the Figure's gpu_status event dispatch.""" + self._gpu_active = bool(active) + # Keep the state echo in sync for save_html snapshots / introspection. + self._state["gpu_active"] = self._gpu_active + + # ── Tile mode ────────────────────────────────────────────────────────────── + + def _make_overview(self, backend) -> np.ndarray: + """Downsampled overview of the WHOLE image for the base texture — fit to + ~OVERVIEW_MAX px on the long edge (refined to the real panel size on the + first view_changed). Row 0 at top (display orientation). + + Uses ``self._overview_method`` (default ``"mean"``) so the base overview is + integration-consistent with detail tiles — important for sparse images where a + ``"subsample"`` overview shows very different intensities from a ``"mean"`` + detail tile, producing a visible shift on zoom-in. Pass + ``overview_method="subsample"`` to ``imshow``/``enable_tile`` when a full-frame + area-mean is too expensive (e.g. 16 MP+ frames scrubbed on every tick).""" + h, w = backend.full_shape + scale = max(1.0, max(h, w) / float(self.OVERVIEW_MAX)) + ov_w = max(1, int(round(w / scale))) + ov_h = max(1, int(round(h / scale))) + ov = backend.sample(0, w, 0, h, ov_w, ov_h, self._overview_method) + if backend.origin == "lower": + ov = np.flipud(ov) # normalise to row-0-top for the base texture + return np.asarray(ov) + + def _backend_display_range(self, backend): + """A display (vmin, vmax) for tile mode derived from the FULL-RES data, NOT + the overview. The overview is a box-MEAN downsample, so its min/max are pulled + toward the mean → a NARROWER range than the native pixels. Quantising the base + over that narrow range is fine (the base is itself averaged), but when a zoom + samples a near-native DETAIL tile and quantises it over the same narrow range, + the true extremes clip to black/white — the region visibly BLOWS OUT (goes + white) on zoom-in. Deriving the range from a SUBSAMPLE (native pixels, no + averaging) of the full image keeps the extremes, so base and detail share one + honest range and the contrast is stable from zoomed-out to zoomed-in. + + Returns (vmin, vmax) or None if it can't be computed (caller falls back to the + overview range). Subsample is capped to ~RANGE_SAMPLE_MAX px/edge so this stays + cheap even on an 8k+ frame.""" + try: + h, w = backend.full_shape + n = self.RANGE_SAMPLE_MAX + sw = min(w, n); sh = min(h, n) + samp = backend.sample(0, w, 0, h, sw, sh, "subsample") + samp = np.asarray(samp) + finite = samp[np.isfinite(samp)] + if finite.size == 0: + return None + vmin = float(finite.min()); vmax = float(finite.max()) + if vmax <= vmin: + return None + return vmin, vmax + except Exception: + return None + + def _visible_region(self, zoom, cx, cy): + """The visible LOGICAL image-pixel rect for the current zoom/center, + expanded by VIEW_OVERFETCH (clamped) so a small pan stays inside the tile.""" + iw, ih = self._logical_w, self._logical_h + vis_w = iw / max(zoom, 1e-9) + vis_h = ih / max(zoom, 1e-9) + # over-fetch around the visible window + ex_w = min(iw, vis_w * self.VIEW_OVERFETCH) + ex_h = min(ih, vis_h * self.VIEW_OVERFETCH) + x0 = int(max(0, min(iw - ex_w, cx * iw - ex_w / 2))) + y0 = int(max(0, min(ih - ex_h, cy * ih - ex_h / 2))) + x1 = int(min(iw, x0 + int(round(ex_w)))) + y1 = int(min(ih, y0 + int(round(ex_h)))) + return x0, x1, y0, y1 + + def enable_tile(self, backend=None, integration_method: str = "mean", + overview_method: str | None = None) -> None: + """Turn tile mode ON (or reconfigure it) AFTER construction — for a consumer + whose large frame isn't known at ``imshow`` time (e.g. a live navigator whose + signal frame arrives later and changes). Registers the internal view→tile loop + (once), sets the logical size from the backend, and paints the overview base. + Call ``update_tile_source(new_frame)`` on each subsequent data change. + + ``backend``: a TileBackend (or an ndarray, wrapped). ``None`` keeps the + current backend (just re-enable / change method). + + ``overview_method``: sampling method for the base overview texture + (``"mean"|"subsample"|"max"``). ``None`` keeps the current setting + (default ``"mean"`` from construction). Use ``"subsample"`` to restore the + old fast nearest-neighbour path when a full-frame area-mean is too expensive.""" + from anyplotlib.plot2d._tile_backend import as_tile_backend + if backend is not None: + self._tile_backend = as_tile_backend(backend, origin=self._origin) + if self._tile_backend is None: + _TLOG.warning("[TILEDBG] enable_tile ABORT: no backend") + return + self._integration_method = integration_method + # Only overwrite when caller explicitly passes a value so that internal + # re-enables (_set_data_tiled, _enable_tile_from_frame) which omit this + # arg preserve whatever the user set at construction / last enable_tile. + if overview_method is not None: + self._overview_method = overview_method + h, w = self._tile_backend.full_shape + self._logical_w, self._logical_h = int(w), int(h) + self._state["image_width"] = int(w) + self._state["image_height"] = int(h) + self._state["tile_enabled"] = True + # Fixed QUANTISATION band for the tile bytes: the overview + detail tiles are + # quantised to uint8 over raw_min/raw_max (the full-res data range), NOT the + # display window. That lets a CONTRAST change re-window purely in the LUT + # (raw_min/raw_max fixed, display_min/display_max move) with NO pixel re-encode + # or re-transfer — set_clim just moves the display window (see set_clim). Set + # once from the full-res range; keep any existing band on a re-enable. + if (self._state.get("raw_min") is None + or not (self._state.get("raw_max", 0) > self._state.get("raw_min", 0))): + rng = self._backend_display_range(self._tile_backend) + if rng is not None: + self._state["raw_min"], self._state["raw_max"] = rng + _was_on = self._tile_on + if not self._tile_on: + self._tile_on = True + self.callbacks.connect("view_changed", self._on_view_changed_internal) + _TLOG.debug( + "[TILEDBG] enable_tile logical=%s was_on=%s method=%s overview_method=%s " + "display=(%s,%s) → will build overview", + (h, w), _was_on, integration_method, self._overview_method, + self._state.get("display_min"), self._state.get("display_max")) + # Paint the overview base + refresh any active tile from the (new) backend. + self.update_tile_source() + + def _disable_tile(self) -> None: + """Leave tile mode (a consumer sent tile=False). Disconnect the internal + view→tile handler, drop the backend, and clear the tile state fields so a + following plain set_data sets image_width to the real frame size with + base_width=0 and no stale detail tile.""" + if self._tile_on: + try: + self.callbacks.disconnect_fn(self._on_view_changed_internal, + "view_changed") + except Exception: + pass + self._tile_on = False + self._tile_backend = None + self._state["tile_enabled"] = False + self._state["base_width"] = 0 + self._state["base_height"] = 0 + self._state.update({"detail_b64": "", "detail_region": [], + "detail_width": 0, "detail_height": 0}) + _TLOG.debug("[TILEDBG] _disable_tile: tile mode OFF (forced plain)") + + def update_tile_source(self, array=None) -> None: + """The backing DATA changed (e.g. a movie navigator advanced a frame) — keep + the current zoom/subselection but refresh the pixels. Re-samples the overview + base AND (if a detail tile is currently shown) the SAME detail region from the + new data, so a live source updates in place without a view change. + + ``array``: swap the numpy backend's source first (convenience for the common + ndarray case). For a custom backend that already mutated its own source, call + ``update_tile_source()`` with no argument to just re-sample.""" + if not self._tile_on or self._tile_backend is None: + _TLOG.debug("[TILEDBG] update_tile_source SKIP: tile_on=%s backend=%s", + self._tile_on, self._tile_backend is not None) + return + if array is not None: + setter = getattr(self._tile_backend, "set_array", None) + if setter is not None: + setter(array) + reg = self._state.get("detail_region") or [] + has_detail = len(reg) == 4 + # Refresh the overview base ONLY when no detail tile is shown (zoomed out) — + # when zoomed in, the overview isn't visible, so re-pushing it every frame is + # wasted work + a texture swap that can flicker. The skipped overview is + # marked STALE and refreshed once on the next view settle (see + # _on_view_changed_internal) so a zoom-out after a zoomed-in scrub never + # shows the pre-scrub frame. + if not has_detail: + self._refresh_overview() + self._push() + return + # Zoomed in: re-sample only the CURRENT detail tile from the new data. + try: + x0, x1, y0, y1 = reg + out_h = self._state.get("detail_height") or (y1 - y0) + out_w = self._state.get("detail_width") or (x1 - x0) + tile = self._tile_backend.sample( + x0, x1, y0, y1, out_w, out_h, self._integration_method) + if self._tile_backend.origin == "lower": + tile = np.flipud(tile) + # The base overview now shows OLD data (only the tile was refreshed). + # Any zoom-out / partial-cover pan would reveal it — refresh once on + # the next view settle. + self._overview_stale = True + self.set_detail(np.ascontiguousarray(tile), x0, x1, y0, y1) + _TLOG.debug( + "[TILEDBG] update_tile_source DETAIL-path (zoomed in): region=%s " + "resampled tile=%s from new frame (overview marked stale)", + reg, (out_h, out_w)) + except Exception as e: + _TLOG.warning("[TILEDBG] detail refresh FAILED: %s", e) + self._push() + + def _refresh_overview(self) -> None: + """Re-sample the overview base from the backend into the state (image_b64 + + base_width/height) and clear the staleness flag. Does NOT push — callers + bundle it with their own push so the fresh base rides the same frame as the + detail/clear that exposed it.""" + try: + ov = self._make_overview(self._tile_backend) + oh, ow = ov.shape + img_u8, _vmin, _vmax = self._normalize_for_base(ov) + self._state["image_b64"] = self._encode_pixels("image_b64", img_u8) + self._state["base_width"] = int(ow) + self._state["base_height"] = int(oh) + self._overview_stale = False + _TLOG.debug( + "[TILEDBG] overview refresh: overview=%s u8[min=%d max=%d] " + "display=(%s,%s) base_wh=(%d,%d) image_wh=(%s,%s)", (oh, ow), + int(img_u8.min()), int(img_u8.max()), + self._state.get("display_min"), self._state.get("display_max"), + ow, oh, self._state.get("image_width"), + self._state.get("image_height")) + except Exception as e: + _TLOG.warning("[TILEDBG] overview refresh FAILED: %s", e) + + def _tile_quant_clim(self): + """The FIXED quantisation band (raw_min/raw_max) the tile bytes are encoded + over, so a contrast change re-windows in the LUT without re-encoding pixels. + Falls back to the display window if no band is set.""" + lo, hi = self._state.get("raw_min"), self._state.get("raw_max") + if lo is None or hi is None or not (hi > lo): + lo, hi = self._state.get("display_min"), self._state.get("display_max") + if lo is None or hi is None or not (hi > lo): + return None + return (lo, hi) + + def _normalize_for_base(self, arr): + """Quantise a base/overview array to uint8 over the fixed raw_min/raw_max band + (NOT the display window) so contrast re-windows via the LUT alone.""" + return _normalize_image(arr, clim=self._tile_quant_clim()) + + def _on_view_changed_internal(self, event) -> None: + """Zoom/pan settled → sample a hi-res detail tile of the (over-fetched) + visible region from the backend and upload it. Zoomed out → clear the tile + (the overview base is enough). The consumer never wires this up.""" + if not self._tile_on or self._tile_backend is None: + _TLOG.debug("[TILEDBG] view_changed IGNORED: tile_on=%s backend=%s", + self._tile_on, self._tile_backend is not None) + return + try: + zoom = float(getattr(event, "zoom", 1.0) or 1.0) + # A zoomed-in scrub refreshed only the detail tile (update_tile_source), + # leaving the base overview on the pre-scrub frame. Any view where the + # base can peek out (zoom-out below; the margin around a partial tile + # here) must therefore re-sample it ONCE on settle — bundled into the + # same push as the detail/clear so the fresh base lands atomically. + stale = getattr(self, "_overview_stale", False) + if zoom < self.VIEW_ZOOM_MIN: + _TLOG.debug( + "[TILEDBG] view_changed zoom=%.3f < MIN=%.2f → CLEAR detail " + "(show overview base only%s)", zoom, self.VIEW_ZOOM_MIN, + ", refresh stale overview" if stale else "") + if stale: + self._refresh_overview() + had_detail = bool(self._state.get("detail_b64")) + self.set_detail(None) # pushes iff a tile was set + if stale and not had_detail: + self._push() # fresh overview still must ship + return + if stale: + self._refresh_overview() # rides the set_detail push below + cx = float(getattr(event, "center_x", 0.5) or 0.5) + cy = float(getattr(event, "center_y", 0.5) or 0.5) + # Panel device px (JS carries it): the tile is ALWAYS output at the panel + # resolution (matched to the region's aspect), NOT min(region, panel). Why: + # the tile fills the same on-screen fit-rect regardless of zoom, so its + # displayed texel density must be CONSTANT — clamping the output to the + # (shrinking) region on a deep zoom made every frame a different-res + # texture stretched over the panel, so the image visibly SNAPPED as the + # scale jumped. A region SMALLER than the panel is upsampled from real + # native pixels (crisp, nearest); a bigger one is area-meaned down. + dw = int(getattr(event, "display_width", 0) or 0) or self.OVERVIEW_MAX + dh = int(getattr(event, "display_height", 0) or 0) or self.OVERVIEW_MAX + x0, x1, y0, y1 = self._visible_region(zoom, cx, cy) + rw, rh = x1 - x0, y1 - y0 + if rw < 2 or rh < 2: + return + # Fit the panel box (dw×dh) to the region's aspect so the tile isn't + # distorted (the fit-rect on screen already has the region's aspect). + aspect = rw / rh + if aspect >= dw / dh: + out_w, out_h = dw, max(1, int(round(dw / aspect))) + else: + out_h, out_w = dh, max(1, int(round(dh * aspect))) + b = self._tile_backend + # DIAGNOSTIC: the raw native region straight from the backend, BEFORE + # sampling — its shape + distinct-value count tells us whether the backend + # actually holds native pixels. If region is 82×82 but the raw crop has + # only ~100 distinct values, the backend source is a downsample (bug). + _raw = np.asarray(b._a[y0:y1, x0:x1]) if hasattr(b, "_a") else None + tile = self._tile_backend.sample( + x0, x1, y0, y1, out_w, out_h, self._integration_method) + if b.origin == "lower": + tile = np.flipud(tile) + self.set_detail(np.ascontiguousarray(tile), x0, x1, y0, y1) + _bshape = b.full_shape if hasattr(b, "full_shape") else "?" + _rawinfo = ("raw_crop=%s distinct=%d" % ( + _raw.shape, int(np.unique(_raw).size))) if _raw is not None else "raw=?" + _tileinfo = "tile_distinct=%d" % int(np.unique(np.asarray(tile)).size) + _TLOG.debug( + "[TILEDBG] view_changed FETCH zoom=%.2f region=[x %d:%d y %d:%d] " + "(%dx%d logical) BACKEND_shape=%s %s → tile=%dx%d %s " + "u8[min=%d max=%d]", + zoom, x0, x1, y0, y1, rw, rh, _bshape, _rawinfo, out_w, out_h, + _tileinfo, int(np.asarray(tile).min()), int(np.asarray(tile).max())) + except Exception as e: + _TLOG.warning("[TILEDBG] view_changed tile update FAILED: %s", e) @staticmethod def _encode_bytes(arr: np.ndarray) -> str: import base64 return base64.b64encode(arr.tobytes()).decode("ascii") + def _encode_pixels(self, key: str, arr: np.ndarray) -> str: + """Return the value to store under a pixel geom key (e.g. ``image_b64``). + + Fast path (Electron BINARY transport active + attached to a Figure): + stash the RAW uint8 bytes on the Figure's ``_raw_pixels`` side-table and + return only a tiny change-token string. ``_electron._route_change`` then + ships those bytes directly as a PLOTBIN frame — skipping the ~20 ms + base64 encode here, the megabyte ``json.dumps`` in ``Figure._push``, and + the ~17 ms base64 decode that the old binary path paid in + ``_route_change`` (it re-decoded the string we had just encoded). + + Fallback (Jupyter / Pyodide / standalone / ``save_html``, or no Figure): + base64-encode into the geom JSON exactly as before — those hosts have no + binary channel, so the pixels must ride in the trait string. + + The returned token must CHANGE whenever the pixels change, because + ``Figure._push`` re-sends the geom channel only when its dict differs + from the last one sent (``_geom_last``). The token is a cheap CONTENT + checksum (adler32, ~2 ms on a 4 MP frame) of the raw bytes, so it also + preserves the "unchanged frame → skip re-send" optimisation exactly like + the old base64-string comparison did (identical pixels → identical + token → ``_push`` skips), while costing a fraction of a base64 encode. + """ + fig = getattr(self, "_fig", None) + store = getattr(fig, "_raw_pixels", None) if fig is not None else None + if store is not None and _binary_transport_active(): + import zlib + raw = arr.tobytes() + store[(self._id, key)] = raw + # Content token: same bytes → same token (skip); changed → re-send. + return f"\x00bin:{zlib.adler32(raw) & 0xFFFFFFFF}" + # No binary channel — pixels must travel as base64 in the geom JSON. + return self._encode_bytes(arr) + def to_state_dict(self) -> dict: - """Return a JSON-serialisable copy of the current state.""" + """Return a JSON-serialisable copy of the current state. + + On the binary-transport path ``_state["image_b64"]`` may hold a tiny + ``"\\x00bin:"`` change-token instead of a base64 string — the + real pixels ride the PLOTBIN channel (``_electron._route_change`` reads + them from the Figure's ``_raw_pixels`` side-table). The token is the + correct WIRE representation and is passed through untouched here so the + hot ``Figure._push`` path does NOT re-encode base64 every frame. + + A COLD consumer with no binary channel (``save_html`` / standalone / + Jupyter) must instead materialise real base64; it calls + :meth:`resolve_pixel_tokens` on the returned dict (``Figure._push`` does + this for the standalone trait when binary transport is off).""" d = dict(self._state) d["overlay_widgets"] = [w.to_dict() for w in self._widgets.values()] d["markers"] = self.markers.to_wire_list() return d + def resolve_pixel_tokens(self, d: dict) -> dict: + """Replace any ``"\\x00bin:…"`` pixel change-token in *d* with the real + base64 (materialised from the Figure's ``_raw_pixels`` side-table), in + place, and return *d*. Used by the COLD paths (save_html / standalone) + that have no PLOTBIN channel and need the pixels inline. A no-op for a + normal base64 string. See :meth:`to_state_dict`.""" + import base64 + fig = getattr(self, "_fig", None) + raw_tbl = getattr(fig, "_raw_pixels", None) + for key in ("image_b64", "overlay_mask_b64"): + val = d.get(key) + if not (isinstance(val, str) and val.startswith("\x00bin:")): + continue + raw = raw_tbl.get((self._id, key)) if raw_tbl is not None else None + d[key] = base64.b64encode(raw).decode("ascii") if raw else "" + return d + # ------------------------------------------------------------------ # Data # ------------------------------------------------------------------ @@ -180,19 +709,34 @@ def data(self) -> np.ndarray: Returns a float64 copy with ``writeable=False``. To replace the data call :meth:`set_data`. + + The float64 cast happens HERE, lazily, not in ``set_data`` — a scrub + pushes a new frame every tick but rarely reads ``.data`` back, and a + full float64 copy of a 4k frame is ~12 ms. ``set_data`` keeps the + frame in its source dtype (``self._data``); we cast + copy only on the + (rare) read. """ - arr = np.flipud(self._data).copy() if self._origin == "lower" else self._data.copy() + src = np.flipud(self._data) if self._origin == "lower" else self._data + arr = np.asarray(src, dtype=float).copy() # cast + independent copy arr.flags.writeable = False return arr def set_data(self, data: np.ndarray, x_axis=None, y_axis=None, units: str | None = None, - clim: tuple | None = None) -> None: + clim: tuple | None = None, tile: "str | bool | None" = None) -> None: """Replace the image data. The ``origin`` supplied at construction is automatically re-applied so the new data is displayed with the same orientation. + ``tile`` — per-call override of the tile decision (default ``None`` uses the + plot's construction preference). ``False`` forces a PLAIN full-frame push even + for a large frame (the caller manages its own decimation); ``True`` forces tile + mode. A live consumer that only wants tiling on some frames (e.g. only when its + GPU is active, sending the NATIVE frame then, but a pre-decimated frame + otherwise) passes ``tile=False`` on the decimated frames so they don't get + auto-tiled at the wrong logical size. + ``clim`` — optional ``(vmin, vmax)`` display range applied in the SAME push as the new data. Without it the caller must follow with a separate ``set_clim``, which pushes a SECOND time: the first push shows the image @@ -210,16 +754,76 @@ def set_data(self, data: np.ndarray, raise ValueError(f"data must be 2-D or (H x W x 3|4), got {data.shape}") h, w = data.shape[:2] + # ── Tile mode: a live consumer (e.g. a movie navigator) calls set_data with + # each new frame. Route it through the tile pipeline instead of clobbering the + # tile state. Writing the plain base frame here would corrupt tiling: it sets + # image_width to THIS frame's width while base_width still names the OLD + # overview (so JS misreads the base texture — the "shrinks to overview size" + # bug), and it clears the active detail tile (so a zoom-in snaps back to the + # blurry overview until the next view_changed re-fetches — the flash). The + # tile path swaps the backend source + re-samples the overview/detail in place, + # keeping image_width, base_width, zoom/center, and the detail region intact. + # RGB frames don't tile — fall through to the plain path. + # Per-call `tile` override wins; else the construction preference decides. + eff_pref = self._tile_pref if tile is None else tile + want_tile = (not is_rgb) and ( + eff_pref is True + or (eff_pref == "auto" and max(h, w) > self.TILE_THRESHOLD)) + # tile=False FORCES the plain path even on an already-tiled plot (the caller is + # sending a pre-decimated frame it wants shown as-is) → tear tiling down first. + force_plain = (tile is False) + _TLOG.debug( + "[TILEDBG] set_data ROUTE frame=%s dtype=%s clim=%s is_rgb=%s " + "tile_arg=%r eff_pref=%r want_tile=%s tile_on=%s backend=%s force_plain=%s " + "THRESH=%s → %s", + (h, w), data.dtype, clim, is_rgb, tile, eff_pref, want_tile, + self._tile_on, self._tile_backend is not None, force_plain, + self.TILE_THRESHOLD, + "PLAIN(forced)" if force_plain + else "TILED-SWAP" if (self._tile_on and self._tile_backend is not None + and not is_rgb) + else "AUTO-ENABLE" if (want_tile and not self._tile_on) + else "PLAIN") + if force_plain and self._tile_on: + # Leaving tile mode: drop the internal view handler + backend so the plain + # push below sets image_width to the real frame size with base_width=0. + self._disable_tile() + elif self._tile_on and self._tile_backend is not None and not is_rgb: + self._set_data_tiled(data, clim) + return + # A large scalar frame arriving on a plot that started small (tiny placeholder + # → real 4k frames): auto-ENABLE tiling now, exactly like imshow(huge) would, + # so the consumer just calls set_data and never hand-rolls a backend. This is + # the seam the SpyDE movie viewer uses. + elif want_tile and not self._tile_on: + self._enable_tile_from_frame(data, clim) + return + if self._origin == "lower": data = np.flipud(data) - self._data = data.astype(float) + # Keep the frame in its SOURCE dtype (no float64 copy on the hot path); + # the `.data` property casts to float lazily on the rare read-back. + # np.array(copy=True) so a later caller mutation can't alias our frame + # (the old `.astype(float)` also copied) — cheap vs float64 (same-dtype). + self._data = np.array(data, copy=True) self._is_rgb = is_rgb self._state["is_rgb"] = is_rgb + # Parse the caller's display range up front so quantisation can spend all + # 256 codes on it (clipping a hot pixel / zero beam) instead of stretching + # them across the raw min/max — see _normalize_image. + parsed_clim = None + if clim is not None and not is_rgb: + try: + parsed_clim = (float(clim[0]), float(clim[1])) + if not (parsed_clim[1] > parsed_clim[0]): + parsed_clim = None # degenerate range → fall back to raw min/max + except (TypeError, ValueError, IndexError): + parsed_clim = None if is_rgb: img_u8, vmin, vmax = _to_rgba_u8(data), 0.0, 255.0 else: - img_u8, vmin, vmax = _normalize_image(data) + img_u8, vmin, vmax = _normalize_image(data, clim=parsed_clim) self._raw_u8, self._raw_vmin, self._raw_vmax = img_u8, vmin, vmax if x_axis is not None: @@ -236,15 +840,13 @@ def set_data(self, data: np.ndarray, if units is not None: self._state["units"] = units - # Apply a caller-supplied display range in the SAME push (no flash). + # Display range for the SAME push (no flash). When a valid clim was used, + # the codes were quantised over it, so raw_min/raw_max already ARE the clim + # and the display window is that same range (identity, t = code/255). Only a + # degenerate/absent clim leaves the display range at the raw min/max. disp_min, disp_max = vmin, vmax - if clim is not None and not is_rgb: - try: - disp_min, disp_max = float(clim[0]), float(clim[1]) - except (TypeError, ValueError, IndexError): - disp_min, disp_max = vmin, vmax fields = { - "image_b64": self._encode_bytes(img_u8), + "image_b64": self._encode_pixels("image_b64", img_u8), "image_width": w, "image_height": h, "display_min": disp_min, @@ -252,6 +854,11 @@ def set_data(self, data: np.ndarray, "raw_min": vmin, "raw_max": vmax, } + # A new base frame invalidates any detail tile of the OLD frame — clear it + # so the shader doesn't sample a stale hi-res crop over the new image. + if self._state.get("detail_b64"): + fields.update({"detail_b64": "", "detail_region": [], + "detail_width": 0, "detail_height": 0}) # RGB images never use the colormap LUT — skip the (costly) rebuild and # leave the existing entry untouched. Only recompute for scalar data. if not is_rgb: @@ -259,6 +866,98 @@ def set_data(self, data: np.ndarray, self._state.update(fields) self._push() + def _set_data_tiled(self, data: np.ndarray, clim: tuple | None) -> None: + """set_data for a plot already in tile mode: swap the tile source + refresh + the overview/detail from it (via update_tile_source) instead of pushing a + plain base frame. Applies ``clim`` first so the re-sampled overview/detail + quantise over the new display range in the SAME push. Keeps the logical + image_width/height, base_width/height, the active detail region, and the + frontend zoom/center — a live frame update, not a view reset. + + The new frame may differ in shape from the current tile source (e.g. a signal + whose signal axes changed); update_tile_source → enable_tile re-derives the + logical size + overview, so image_width/height stay correct for the new frame. + """ + # Apply the display range up front (if given + valid) so the overview and any + # detail tile re-sampled below quantise over it — otherwise the tile path uses + # the stale display_min/display_max and the contrast lags a frame. + if clim is not None: + try: + vmin, vmax = float(clim[0]), float(clim[1]) + if vmax > vmin: + self._state["display_min"] = vmin + self._state["display_max"] = vmax + except (TypeError, ValueError, IndexError): + pass + # If the frame's shape changed, re-enable tiling so image_width/height and the + # overview are re-derived for the new logical size; otherwise just swap + re- + # sample in place (the common live-navigator case: same-size frames). + from anyplotlib.plot2d._tile_backend import as_tile_backend + h, w = data.shape[:2] + same_shape = (int(w) == int(self._logical_w) + and int(h) == int(self._logical_h)) + # ascontiguousarray is a no-op (returns the SAME array) when already C- + # contiguous — the common movie-frame case — so this is free. It replaces the + # old np.array(copy=True) which unconditionally copied all 16 MP every frame + # (~4.5 ms) just to stash self._data for a rare .data read-back. The backend + # holds the frame now; expose it lazily for read-back below. + arr = np.ascontiguousarray(data) + self._data = arr # reference, not a copy — read-back reads the frame + setter = getattr(self._tile_backend, "set_array", None) + _TLOG.debug( + "[TILEDBG] _set_data_tiled frame=%s logical=(%s,%s) same_shape=%s " + "has_set_array=%s display=(%s,%s) → %s", (h, w), + self._logical_h, self._logical_w, same_shape, setter is not None, + self._state.get("display_min"), self._state.get("display_max"), + "swap+update_tile_source" if (same_shape and setter is not None) + else "rebuild+enable_tile") + if same_shape and setter is not None: + setter(arr) + self.update_tile_source() + else: + # Shape changed (or a custom backend without set_array): rebuild the + # backend around the new frame and re-enable, preserving zoom/center. + self._tile_backend = as_tile_backend(arr, origin=self._origin) + self.enable_tile(self._tile_backend, self._integration_method) + + def _enable_tile_from_frame(self, data: np.ndarray, clim: tuple | None) -> None: + """First large frame on a not-yet-tiled plot → turn tile mode ON around it, + exactly like ``imshow(huge)``: wrap the frame in a NumpyTileBackend, set the + display range (the caller's ``clim`` if given, else the FULL-RES range so a + zoom detail tile doesn't blow out — see _backend_display_range), then + enable_tile (which pushes the overview base). This is the seam a live consumer + uses: imshow a small placeholder, then set_data the real frames; no need to + hand-roll a backend or call enable_tile directly.""" + from anyplotlib.plot2d._tile_backend import as_tile_backend + self._data = np.array(data, copy=True) + arr = np.ascontiguousarray(data) + self._tile_backend = as_tile_backend(arr, origin=self._origin) + # Display range: caller's clim wins; else derive from the full-res frame. + rng = None + if clim is not None: + try: + lo, hi = float(clim[0]), float(clim[1]) + if hi > lo: + rng = (lo, hi) + except (TypeError, ValueError, IndexError): + rng = None + _clim_src = "caller-clim" if rng is not None else None + if rng is None: + rng = self._backend_display_range(self._tile_backend) + _clim_src = "full-res-probe" + if rng is not None: + self._state["display_min"], self._state["display_max"] = rng + else: + _clim_src = "NONE(kept-stale)" + _TLOG.debug( + "[TILEDBG] _enable_tile_from_frame frame=%s dtype=%s data[min=%.4g " + "max=%.4g] → display=(%s,%s) via %s", data.shape, data.dtype, + float(np.nanmin(data)) if data.size else 0.0, + float(np.nanmax(data)) if data.size else 0.0, + self._state.get("display_min"), self._state.get("display_max"), + _clim_src) + self.enable_tile(self._tile_backend, self._integration_method) + def set_overlay_mask(self, mask: "np.ndarray | None", color: str = "#ff4444", alpha: float = 0.4) -> None: @@ -320,12 +1019,103 @@ def set_colormap(self, name: str) -> None: self._push() def set_clim(self, vmin=None, vmax=None) -> None: + """Set the display range. Because scalar frames are quantised to uint8 over + their clim (so a hot pixel / zero beam can't crush the signal — see + _normalize_image), a new range is honoured by RE-QUANTISING from the cached + raw frame, not merely re-windowing the existing codes (which are saturated + outside the previous band and so couldn't widen past it). For an RGB frame + (no scalar quantisation) or when no raw frame is cached, fall back to a pure + display-window update.""" + new_min = float(vmin) if vmin is not None else self._state.get("display_min") + new_max = float(vmax) if vmax is not None else self._state.get("display_max") + + # ── TILE MODE: window via the LUT only — do NOT re-quantise/re-push pixels. + # A tiled plot's base is the OVERVIEW and its pixels stay resident on the GPU; + # the shader re-windows contrast purely from the 256-entry LUT (rebuilt in JS + # from display_min/display_max over the fixed raw_min/raw_max band). Re- + # quantising self._data here would re-encode the FULL 16 MP native frame AND + # ship it into the overview slot every drag tick — the "retransfers the whole + # image" lag. So just move the display window (a tiny push). Contrast resolves + # within the base's original quantisation band, which spans the frame's range. + if self._tile_on: + if vmin is not None: + self._state["display_min"] = float(vmin) + if vmax is not None: + self._state["display_max"] = float(vmax) + self._push() + return + + raw = getattr(self, "_data", None) + if (not self._is_rgb and raw is not None + and new_min is not None and new_max is not None + and new_max > new_min): + img_u8, qmin, qmax = _normalize_image(raw, clim=(new_min, new_max)) + self._raw_u8, self._raw_vmin, self._raw_vmax = img_u8, qmin, qmax + self._state.update({ + "image_b64": self._encode_pixels("image_b64", img_u8), + "display_min": qmin, + "display_max": qmax, + "raw_min": qmin, + "raw_max": qmax, + }) + self._push() + return + # Fallback: RGB / no cached frame → just move the display window. if vmin is not None: self._state["display_min"] = float(vmin) if vmax is not None: self._state["display_max"] = float(vmax) self._push() + def set_detail(self, tile=None, x0=None, x1=None, y0=None, y1=None) -> None: + """Upload a HIGH-RES detail tile covering the LOGICAL image-pixel rectangle + ``[x0:x1, y0:y1]`` of the base image (in the SAME orientation as the frame + passed to ``set_data`` — already ``origin``-applied), so a zoom-in shows true + native pixels for the visible region WITHOUT transferring the whole full-res + frame. The shader samples this tile instead of the base whenever the current + zoom window lies inside ``[x0:x1, y0:y1]``; otherwise it falls back to the + base texture. ``set_detail(None)`` clears it (revert to base). + + ``tile`` is quantised to uint8 over the SAME display range as the base + (``display_min``/``display_max``) so its contrast matches seamlessly — a + zoom crop must look identical to the base, just sharper. Only meaningful for + scalar (non-RGB) images.""" + if tile is None or self._is_rgb: + if self._state.get("detail_b64"): + self._state.update({ + "detail_b64": "", "detail_region": [], + "detail_width": 0, "detail_height": 0, + }) + self._push() + return + tile = np.asarray(tile) + if tile.ndim != 2: + raise ValueError(f"detail tile must be 2-D, got {tile.shape}") + # In tile mode quantise over the FIXED raw_min/raw_max band (so contrast re- + # windows via the LUT with no re-encode); otherwise (manual set_detail on a + # non-tiled plot) match the display window as before. + if self._tile_on: + clim = self._tile_quant_clim() + else: + clim = (self._state.get("display_min"), self._state.get("display_max")) + if clim[0] is None or clim[1] is None or not (clim[1] > clim[0]): + clim = None + img_u8, _vmin, _vmax = _normalize_image(tile, clim=clim) + th, tw = tile.shape + # Monotonic sequence so the renderer's dedup key CHANGES on every pushed tile + # even when length + region are identical (a live movie scrub re-samples the + # SAME region every frame — without this the JS skips the re-upload and the + # zoomed-in view freezes on the first frame). See _detailBytes in figure_esm.js. + self._detail_seq = getattr(self, "_detail_seq", 0) + 1 + self._state.update({ + "detail_b64": self._encode_pixels("detail_b64", img_u8), + "detail_region": [int(x0), int(x1), int(y0), int(y1)], + "detail_width": int(tw), + "detail_height": int(th), + "detail_seq": self._detail_seq, + }) + self._push() + def set_scale_mode(self, mode: str) -> None: valid = ("linear", "log", "symlog") if mode not in valid: @@ -378,7 +1168,15 @@ def get_xbound(self) -> tuple: xarr = np.asarray(self._state["x_axis"]) return (float(xarr.min()), float(xarr.max())) - def set_extent(self, x_axis, y_axis) -> None: + def set_extent(self, x_axis, y_axis, units: str | None = None) -> None: + """Recalibrate the image axes to the given coordinate arrays. + + Sets ``has_axes`` so the front-end draws physical tick gutters + the + scale bar — the same gate ``set_data(x_axis=, y_axis=)`` sets. Without + this a tiled image (which is calibrated ONLY through ``set_extent`` / + ``enable_tile``, never through ``set_data`` with axis args) would show no + ticks and no scale bar (the ``has_axes`` gate stayed False). Pass + ``units`` to update the scale-bar unit label in the same push.""" x_axis = np.asarray(x_axis, dtype=float) y_axis = np.asarray(y_axis, dtype=float) w = self._state["image_width"] @@ -389,6 +1187,9 @@ def set_extent(self, x_axis, y_axis) -> None: self._state["y_axis"] = y_axis.tolist() self._state["scale_x"] = scale_x self._state["scale_y"] = scale_y + self._state["has_axes"] = len(x_axis) >= 2 and len(y_axis) >= 2 + if units is not None: + self._state["units"] = units self._push() def set_colorbar_label(self, label: str, fontsize: float | None = None) -> None: @@ -578,7 +1379,8 @@ def add_circles(self, offsets, name=None, *, radius=5, linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add circle markers at (x, y) positions in data coordinates.""" return self._add_marker("circles", name, offsets=offsets, radius=radius, facecolors=facecolors, edgecolors=edgecolors, @@ -586,14 +1388,16 @@ def add_circles(self, offsets, name=None, *, radius=5, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_points(self, offsets, name=None, *, sizes=5, color="#ff0000", facecolors=None, linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add point markers at (x, y) positions in data coordinates.""" return self._add_marker("circles", name, offsets=offsets, radius=sizes, edgecolors=color, facecolors=facecolors, @@ -601,49 +1405,57 @@ def add_points(self, offsets, name=None, *, sizes=5, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_hlines(self, y_values, name=None, *, color="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add static horizontal lines at the given y positions.""" return self._add_marker("hlines", name, offsets=y_values, color=color, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_vlines(self, x_values, name=None, *, color="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 """Add static vertical lines at the given x positions.""" return self._add_marker("vlines", name, offsets=x_values, color=color, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_arrows(self, offsets, U, V, name=None, *, edgecolors="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("arrows", name, offsets=offsets, U=U, V=V, edgecolors=edgecolors, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_ellipses(self, offsets, widths, heights, name=None, *, angles=0, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("ellipses", name, offsets=offsets, widths=widths, heights=heights, angles=angles, facecolors=facecolors, edgecolors=edgecolors, @@ -651,25 +1463,29 @@ def add_ellipses(self, offsets, widths, heights, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_lines(self, segments, name=None, *, edgecolors="#ff0000", linewidths=1.5, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("lines", name, segments=segments, edgecolors=edgecolors, linewidths=linewidths, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_rectangles(self, offsets, widths, heights, name=None, *, angles=0, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("rectangles", name, offsets=offsets, widths=widths, heights=heights, angles=angles, facecolors=facecolors, edgecolors=edgecolors, @@ -677,14 +1493,16 @@ def add_rectangles(self, offsets, widths, heights, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_squares(self, offsets, widths, name=None, *, angles=0, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("squares", name, offsets=offsets, widths=widths, angles=angles, facecolors=facecolors, edgecolors=edgecolors, @@ -692,32 +1510,37 @@ def add_squares(self, offsets, widths, name=None, *, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_polygons(self, vertices_list, name=None, *, facecolors=None, edgecolors="#ff0000", linewidths=1.5, alpha=0.3, hover_edgecolors=None, hover_facecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("polygons", name, vertices_list=vertices_list, facecolors=facecolors, edgecolors=edgecolors, linewidths=linewidths, alpha=alpha, hover_edgecolors=hover_edgecolors, hover_facecolors=hover_facecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def add_texts(self, offsets, texts, name=None, *, color="#ff0000", fontsize=12, hover_edgecolors=None, labels=None, label=None, - transform: str = "data") -> "MarkerGroup": # noqa: F821 + transform: str = "data", + clip_display: bool = True) -> "MarkerGroup": # noqa: F821 return self._add_marker("texts", name, offsets=offsets, texts=texts, color=color, fontsize=fontsize, hover_edgecolors=hover_edgecolors, labels=labels, label=label, - transform=transform) + transform=transform, + clip_display=clip_display) def __repr__(self) -> str: w = self._state.get("image_width", "?") diff --git a/anyplotlib/plot2d/_plotmesh.py b/anyplotlib/plot2d/_plotmesh.py index 0a8184c9..3210b493 100644 --- a/anyplotlib/plot2d/_plotmesh.py +++ b/anyplotlib/plot2d/_plotmesh.py @@ -92,7 +92,7 @@ def set_data(self, data: np.ndarray, self._raw_u8, self._raw_vmin, self._raw_vmax = img_u8, vmin, vmax self._state.update({ - "image_b64": self._encode_bytes(img_u8), + "image_b64": self._encode_pixels("image_b64", img_u8), "image_width": cols, "image_height": rows, "x_axis": xe.tolist(), diff --git a/anyplotlib/plot2d/_tile_backend.py b/anyplotlib/plot2d/_tile_backend.py new file mode 100644 index 00000000..be68a0fb --- /dev/null +++ b/anyplotlib/plot2d/_tile_backend.py @@ -0,0 +1,181 @@ +"""Pluggable tile backend for large-image display. + +A ``TileBackend`` OWNS the source data (an ndarray, a lazy/dask array, a GPU-resident +tensor, a remote store …) and answers two things: + + * the LOGICAL geometry of the full image — ``full_shape`` (H, W), ``dtype``, + ``origin``, and a data-space ``extent`` — so anyplotlib can set up the axes/aspect + WITHOUT ever holding the whole array; + * ``sample(x0, x1, y0, y1, out_w, out_h, method)`` — downsample (or upsample) the + logical region ``[y0:y1, x0:x1]`` to an ``(out_h, out_w)`` tile. + +anyplotlib owns the zoom/pan → re-tile lifecycle and calls ``sample`` for the visible +region at the panel's resolution. Swap the backend to change WHERE/HOW tiles are +computed (GPU+torch for a fast mean, dask/zarr for out-of-core, an app's chunk cache) +without touching that lifecycle. + +The default :class:`NumpyTileBackend` covers an in-memory ndarray. anyplotlib keeps no +hard dependency on torch/dask — a GPU/lazy backend is a separate class implementing the +same ``Protocol`` that the consumer (or an optional extra) provides. +""" +from __future__ import annotations + +import math +from typing import Protocol, runtime_checkable + +import numpy as np + + +@runtime_checkable +class TileBackend(Protocol): + """Source of tiled large-image data. Owns the array; anyplotlib never holds it.""" + + @property + def full_shape(self) -> "tuple[int, int]": + """(H, W) of the full LOGICAL image in pixels.""" + ... + + @property + def dtype(self) -> np.dtype: + """Source element dtype (anyplotlib quantises tiles to uint8 for display).""" + ... + + @property + def origin(self) -> str: + """``"upper"`` (row 0 at top) or ``"lower"`` (row 0 at bottom).""" + ... + + def extent(self) -> "tuple[float, float, float, float] | None": + """Data-space ``(x0, x1, y0, y1)`` the image spans, or ``None`` for pixel + coordinates (``(0, W, 0, H)``).""" + ... + + def sample(self, x0: int, x1: int, y0: int, y1: int, + out_w: int, out_h: int, method: str = "mean") -> np.ndarray: + """Return the logical region ``[y0:y1, x0:x1]`` resampled to ``(out_h, + out_w)`` via ``method`` (``"mean"|"subsample"|"max"``) as a 2-D ndarray.""" + ... + + +# ── Integer-friendly area reductions (shared by the numpy backend) ───────────── + +def _box_reduce(region: np.ndarray, out_h: int, out_w: int, op: str) -> np.ndarray: + """Reduce ``region`` (2-D) to ``(out_h, out_w)`` by a block ``op`` ("mean"|"max"). + + Fast path (region divisible by the stride): a single VECTORISED reshape-reduce in + a wide integer accumulator (no full float cast of the source — the cast of a 16 MP + uint16 frame alone is ~34 ms). Ragged path (non-divisible): a strided-accumulate + box filter with a per-cell count, so the last partial block is reduced over only + its valid pixels. Either way the grid is nearest-resized to the exact (out_h, + out_w) the caller asked for.""" + h, w = region.shape + sy = max(1, h // out_h) + sx = max(1, w // out_w) + is_int = np.issubdtype(region.dtype, np.integer) + + if h % sy == 0 and w % sx == 0: + # Divisible → one reshape-reduce (fast, vectorised). Integer sum stays uint32 + # (uint16 × up-to-64 block fits) so there's no giant float cast. + gh, gw = h // sy, w // sx + blk = region.reshape(gh, sy, gw, sx) + if op == "max": + out = blk.max(axis=(1, 3)) + else: + out = (blk.sum(axis=(1, 3), dtype=np.uint32 if is_int else np.float64) + .astype(np.float32) / (sy * sx)) + return _nearest_resize(out, out_h, out_w) + + # Ragged → strided accumulate with a per-cell count (handles the partial block). + gh = math.ceil(h / sy) + gw = math.ceil(w / sx) + if op == "max": + acc = None + for dy in range(sy): + for dx in range(sx): + sub = region[dy::sy, dx::sx] + sh, sw = sub.shape + if acc is None: + acc = np.zeros((gh, gw), region.dtype) + acc[:sh, :sw] = sub + else: + np.maximum(acc[:sh, :sw], sub, out=acc[:sh, :sw]) + out = acc + else: + acc = np.zeros((gh, gw), np.uint32 if is_int else np.float64) + cnt = np.zeros((gh, gw), np.uint32) + for dy in range(sy): + for dx in range(sx): + sub = region[dy::sy, dx::sx] + sh, sw = sub.shape + acc[:sh, :sw] += sub + cnt[:sh, :sw] += 1 + out = acc.astype(np.float32) / cnt + return _nearest_resize(out, out_h, out_w) + + +def _nearest_resize(a: np.ndarray, out_h: int, out_w: int) -> np.ndarray: + """Nearest-neighbour resize a 2-D array to ``(out_h, out_w)`` (up or down).""" + h, w = a.shape + if (h, w) == (out_h, out_w): + return a + yi = (np.arange(out_h) * h // max(1, out_h)).clip(0, h - 1) + xi = (np.arange(out_w) * w // max(1, out_w)).clip(0, w - 1) + return a[yi][:, xi] + + +class NumpyTileBackend: + """Default :class:`TileBackend` over an in-memory ndarray.""" + + def __init__(self, array: np.ndarray, extent=None, origin: str = "upper") -> None: + a = np.asarray(array) + if a.ndim != 2: + raise ValueError(f"NumpyTileBackend needs a 2-D array, got {a.shape}") + self._a = a + self._extent = tuple(float(v) for v in extent) if extent is not None else None + self._origin = origin + + # The tiling lifecycle can swap the frame in place (e.g. a movie navigator) so a + # zoom re-tiles from the current frame without rebuilding the plot. + def set_array(self, array: np.ndarray) -> None: + a = np.asarray(array) + if a.ndim != 2: + raise ValueError(f"NumpyTileBackend needs a 2-D array, got {a.shape}") + self._a = a + + @property + def full_shape(self): + return self._a.shape + + @property + def dtype(self): + return self._a.dtype + + @property + def origin(self): + return self._origin + + def extent(self): + return self._extent + + def sample(self, x0, x1, y0, y1, out_w, out_h, method="mean"): + h, w = self._a.shape + x0 = int(max(0, min(w, x0))); x1 = int(max(x0 + 1, min(w, x1))) + y0 = int(max(0, min(h, y0))); y1 = int(max(y0 + 1, min(h, y1))) + out_w = int(max(1, out_w)); out_h = int(max(1, out_h)) + region = self._a[y0:y1, x0:x1] + if method == "subsample": + sy = max(1, (y1 - y0) // out_h) + sx = max(1, (x1 - x0) // out_w) + return _nearest_resize(region[::sy, ::sx], out_h, out_w) + if method == "max": + return _box_reduce(region, out_h, out_w, "max") + # "mean" (default) + return _box_reduce(region, out_h, out_w, "mean") + + +def as_tile_backend(source, *, extent=None, origin="upper") -> TileBackend: + """Coerce ``source`` to a TileBackend: pass a backend through; wrap an ndarray in + a :class:`NumpyTileBackend`.""" + if isinstance(source, TileBackend) and not isinstance(source, np.ndarray): + return source + return NumpyTileBackend(np.asarray(source), extent=extent, origin=origin) diff --git a/anyplotlib/tests/_gpu_clim_check.cjs b/anyplotlib/tests/_gpu_clim_check.cjs new file mode 100644 index 00000000..c14eedd0 --- /dev/null +++ b/anyplotlib/tests/_gpu_clim_check.cjs @@ -0,0 +1,51 @@ +/** + * _gpu_clim_check.cjs — verify the GPU shader honors a NARROWED clim correctly + * (regression guard for the "clim applied twice" critical bug). Runs in real + * Electron (WebGPU on the GPU). Loads a standalone anyplotlib page with a known + * ramp image + a narrow vmin/vmax, reads back the GPU output via + * __apl_gpuReadback, and compares to the expected windowed-colormap values passed + * in as argv[3] (a JSON grid computed in numpy with the SAME clim). + * + * Prints CLIM_RESULT {meanDiff, maxDiff, gpuActive} and exits 0 iff GPU matches. + */ +const { app, BrowserWindow } = require('electron') +app.commandLine.appendSwitch('enable-unsafe-webgpu') + +const htmlPath = process.argv[2] +const expected = JSON.parse(process.argv[3]) // [[r,g,b], ...] length N*N, row-major +const panelId = process.argv[4] +const N = Math.round(Math.sqrt(expected.length)) + +app.whenReady().then(async () => { + const win = new BrowserWindow({ width: 900, height: 900, show: false, + webPreferences: { offscreen: false, webSecurity: false } }) + try { + await win.loadFile(htmlPath) + // Wait for mount + async GPU device + activation + a paint. + await new Promise(r => setTimeout(r, 3500)) + const res = await win.webContents.executeJavaScript(`(async () => { + const g = globalThis.__apl_gpu2d || {}; + const pid = ${JSON.stringify(panelId)} || Object.keys(g)[0]; + const active = !!(g[pid] && g[pid].active); + let px = null; + if (typeof globalThis.__apl_gpuReadback === 'function' && pid) + px = await globalThis.__apl_gpuReadback(pid, ${N}); + return { active, px: px && px.px }; + })()`, true) + if (!res.px) { console.log('CLIM_RESULT', JSON.stringify({ err: 'no readback', ...res })); app.exit(1); return } + let sum = 0, maxd = 0, n = Math.min(res.px.length, expected.length) + for (let i = 0; i < n; i++) + for (let c = 0; c < 3; c++) { + const d = Math.abs(res.px[i][c] - expected[i][c]); sum += d + if (d > maxd) maxd = d + } + const meanDiff = sum / (n * 3) + console.log('CLIM_RESULT', JSON.stringify({ + gpuActive: res.active, meanDiff: +meanDiff.toFixed(2), maxDiff: maxd, n })) + // Small tolerance for the NxN downscale sampling + nearest-rounding. + app.exit(res.active && meanDiff < 8 ? 0 : 2) + } catch (e) { + console.log('CLIM_RESULT', JSON.stringify({ err: String(e).slice(0, 80) })) + app.exit(1) + } finally { try { win.destroy() } catch (_) {} } +}) diff --git a/anyplotlib/tests/_png_utils.py b/anyplotlib/tests/_png_utils.py index cfb2ae98..272a7c14 100644 --- a/anyplotlib/tests/_png_utils.py +++ b/anyplotlib/tests/_png_utils.py @@ -287,3 +287,35 @@ def compare_arrays( return True, f"ok — {n_bad}/{n_total} pixels ({frac:.2%}) differ by >{tol}" + +def compare_arrays_exact(actual: np.ndarray, expected: np.ndarray) -> tuple[bool, str]: + """Compare two arrays with strict byte-for-byte equality. + + Returns ``(True, msg)`` only when *all* values are identical. + """ + if actual.shape != expected.shape: + return False, ( + f"shape mismatch: actual {actual.shape} vs expected {expected.shape}" + ) + + neq = actual != expected + if not np.any(neq): + return True, f"ok — all {actual.size} values are identical" + + # For image arrays (H, W, C), count per-pixel mismatches (any channel differs). + if actual.ndim >= 3: + bad = np.any(neq, axis=-1) + else: + bad = neq + n_bad = int(bad.sum()) + n_total = bad.size + + first = tuple(int(i) for i in np.argwhere(neq)[0]) + aval = int(actual[first]) + eval_ = int(expected[first]) + return False, ( + f"{n_bad}/{n_total} elements differ; " + f"first mismatch at {first}: actual={aval}, expected={eval_}" + ) + + diff --git a/anyplotlib/tests/conftest.py b/anyplotlib/tests/conftest.py index b1b3faa7..65e34343 100644 --- a/anyplotlib/tests/conftest.py +++ b/anyplotlib/tests/conftest.py @@ -282,6 +282,155 @@ def _open(widget): path.unlink(missing_ok=True) +# --------------------------------------------------------------------------- +# WebGPU browser + interaction fixture +# --------------------------------------------------------------------------- + +@pytest.fixture(scope="session") +def _pw_gpu_browser(_pw_browser): + """Full-build Chromium with WebGPU enabled, or skip. + + Playwright's default headless browser is the stripped "chromium headless + shell", which has NO ``navigator.gpu`` at all. The full build + (``channel="chromium"``, new headless mode) exposes the machine's real + WebGPU adapter when launched with ``--enable-unsafe-webgpu``. Tests that + depend on this fixture are skipped when the channel isn't installed + (``playwright install chromium`` installs both) or when the machine has + no usable adapter (e.g. a GPU-less CI runner). + + Launched through the EXISTING sync-playwright context (via + ``_pw_browser.browser_type``) — only one ``sync_playwright()`` may be + active per thread, so a second context manager here would raise + "Sync API inside the asyncio loop" whenever both fixtures are used in + one run. + + NB ``navigator.gpu`` only exists in secure contexts: probe/test pages + must be loaded via ``file://`` (fine), not ``about:blank``. + """ + try: + browser = _pw_browser.browser_type.launch( + headless=True, channel="chromium", + args=["--enable-unsafe-webgpu"], + ) + except Exception as e: # channel not installed + pytest.skip(f"full-chromium channel unavailable: {e}") + # Probe for a real adapter once per session (file:// for secure context). + with tempfile.NamedTemporaryFile( + suffix=".html", mode="w", encoding="utf-8", delete=False + ) as fh: + fh.write("") + probe_path = pathlib.Path(fh.name) + page = browser.new_page() + try: + page.goto(probe_path.as_uri()) + # Probe the FULL render path, not just requestAdapter(): on GPU-less CI + # runners (GitHub ubuntu) a SwiftShader adapter resolves but device + # creation / the canvas 'webgpu' context / an actual submit fails, so an + # adapter-only probe passes and every test then times out waiting for + # __apl_gpu2d activation instead of skipping. Mirror what the figure ESM + # does: adapter → device → canvas context → configure → one cleared + # render pass, each step raced against a timeout. + adapter = page.evaluate( + """async () => { + if (!navigator.gpu) return null; + const timeout = (ms) => + new Promise((r) => setTimeout(() => r('__timeout__'), ms)); + try { + const a = await Promise.race( + [navigator.gpu.requestAdapter(), timeout(10000)]); + if (!a || a === '__timeout__') return null; + const d = await Promise.race( + [a.requestDevice(), timeout(10000)]); + if (!d || d === '__timeout__') return null; + const c = document.createElement('canvas'); + c.width = 4; c.height = 4; + const ctx = c.getContext('webgpu'); + if (!ctx) return null; + ctx.configure({ + device: d, + format: navigator.gpu.getPreferredCanvasFormat(), + alphaMode: 'premultiplied', + }); + const tex = ctx.getCurrentTexture(); + if (!tex) return null; + const enc = d.createCommandEncoder(); + const pass = enc.beginRenderPass({ colorAttachments: [{ + view: tex.createView(), loadOp: 'clear', storeOp: 'store', + clearValue: { r: 0, g: 0, b: 0, a: 1 }, + }]}); + pass.end(); + d.queue.submit([enc.finish()]); + const done = await Promise.race( + [d.queue.onSubmittedWorkDone(), timeout(10000)]); + if (done === '__timeout__') return null; + return (a.info && a.info.vendor) || 'unknown'; + } catch (_) { return null; } + }""" + ) + finally: + page.close() + probe_path.unlink(missing_ok=True) + if adapter is None: + browser.close() + pytest.skip("no usable WebGPU render path in headless Chromium") + yield browser + browser.close() + + +@pytest.fixture +def gpu_interact_page(_pw_gpu_browser): + """Like ``interact_page`` but in the WebGPU-capable browser. + + Returns ``open_widget(fig, expect_gpu=False) → page``. With + ``expect_gpu=True`` it additionally waits until every 2-D image panel + reports the GPU path ACTIVE (the async device init + activation redraw + landed) via the ``__apl_gpu2d`` diagnostic — so a screenshot taken after + this is guaranteed to be the WebGPU rendering, not the first-frame + Canvas2D paint. CPU-reference figures (``gpu=False``) should be opened in + this SAME browser (``expect_gpu=False``) so GPU-vs-CPU comparisons see an + identical rendering environment apart from the image path under test. + """ + _pages: list = [] + _paths: list = [] + + def _open(widget, expect_gpu: bool = False): + html = _build_interact_html(widget) + with tempfile.NamedTemporaryFile( + suffix=".html", mode="w", encoding="utf-8", delete=False + ) as fh: + fh.write(html) + tmp = pathlib.Path(fh.name) + _paths.append(tmp) + + page = _pw_gpu_browser.new_page() + _pages.append(page) + page.goto(tmp.as_uri()) + page.wait_for_function("() => window._aplReady === true", timeout=15_000) + page.evaluate( + "() => new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)))" + ) + if expect_gpu: + page.wait_for_function( + """() => { + const d = globalThis.__apl_gpu2d || {}; + const v = Object.values(d); + return v.length > 0 && v.every(x => x.active); + }""", + timeout=20_000, + ) + return page + + yield _open + + for page in _pages: + try: + page.close() + except Exception: + pass + for path in _paths: + path.unlink(missing_ok=True) + + # --------------------------------------------------------------------------- # Benchmark fixtures + helper # --------------------------------------------------------------------------- diff --git a/anyplotlib/tests/test_electron/__init__.py b/anyplotlib/tests/test_electron/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/anyplotlib/tests/test_electron/test_binary_frame.py b/anyplotlib/tests/test_electron/test_binary_frame.py new file mode 100644 index 00000000..f97ff4c4 --- /dev/null +++ b/anyplotlib/tests/test_electron/test_binary_frame.py @@ -0,0 +1,245 @@ +""" +Binary transport wire format + Electron routing. + +The PLOTBIN framing (_binary_frame) is the single source of truth shared by the +Python producer and the JS host, so it's unit-tested here in isolation — no +Electron, no stdout — plus the _electron routing that decides base64 vs binary. +Runs in the normal CI matrix. +""" +from __future__ import annotations + +import base64 +import importlib +import json + +import numpy as np +import pytest + +from anyplotlib import _binary_frame as bf + + +class TestFrameRoundTrip: + def test_encode_decode_roundtrip(self): + payload = np.arange(256, dtype=np.uint8).tobytes() + frame = bf.encode_frame("fig1", "image_b64", + {"w": 16, "h": 16, "dtype": "uint8"}, payload) + header, out, n = frame, None, 0 + header, out, n = bf.decode_frame(frame) + assert header["fig_id"] == "fig1" + assert header["key"] == "image_b64" + assert header["w"] == 16 and header["h"] == 16 + assert out == payload + assert n == len(frame) + + def test_payload_bytes_are_raw_not_base64(self): + # The whole point: the payload on the wire is the raw bytes, not base64. + payload = bytes([0, 255, 128, 7, 200]) + frame = bf.encode_frame("f", "image_b64", {}, payload) + # The raw payload appears verbatim at the tail. + assert frame.endswith(payload) + # And it is NOT the base64 text of the payload. + assert base64.b64encode(payload) not in frame + + def test_marker_and_prefix(self): + payload = b"\x01\x02\x03\x04" + frame = bf.encode_frame("f", "k", {"a": 1}, payload) + assert frame.startswith(bf.MARKER) + nl = frame.find(b"\n") + hlen, plen = bf.parse_prefix(frame[:nl]) + assert plen == len(payload) + # header_len covers the merged fig_id/key/metadata JSON. + hdr = json.loads(frame[nl + 1:nl + 1 + hlen]) + assert hdr == {"a": 1, "fig_id": "f", "key": "k"} + + def test_binary_safe_payload(self): + # A payload containing the marker bytes / newlines must survive (length- + # prefixed, not delimiter-scanned). + payload = bf.MARKER + b"\n:123:\nPLOTAPP:{}" + bytes(range(256)) + frame = bf.encode_frame("f", "k", {}, payload) + header, out, n = bf.decode_frame(frame) + assert out == payload + + def test_truncated_frame_raises(self): + payload = b"abcdefgh" + frame = bf.encode_frame("f", "k", {}, payload) + with pytest.raises(ValueError): + bf.decode_frame(frame[:-2]) # payload cut short + + def test_two_frames_concatenated(self): + # A host reads frame 1, then continues at n_consumed for frame 2. + f1 = bf.encode_frame("a", "k", {}, b"first") + f2 = bf.encode_frame("b", "k", {}, b"second-longer") + buf = f1 + f2 + h1, p1, n1 = bf.decode_frame(buf) + assert p1 == b"first" and h1["fig_id"] == "a" + h2, p2, n2 = bf.decode_frame(buf[n1:]) + assert p2 == b"second-longer" and h2["fig_id"] == "b" + + +class TestElectronRouting: + def _reload_electron(self, monkeypatch, binary: bool): + monkeypatch.setenv("APL_BINARY_TRANSPORT", "1" if binary else "0") + import anyplotlib._electron as el + importlib.reload(el) + return el + + def _geom_value(self, pixels: np.ndarray) -> str: + # The real wire shape: image_b64 nested in a panel geom JSON string. + return json.dumps({ + "image_b64": base64.b64encode(pixels.tobytes()).decode(), + "colormap_data": [[0, 0, 0]], + }) + + def test_geom_pixels_go_binary_when_enabled(self, monkeypatch): + el = self._reload_electron(monkeypatch, binary=True) + captured = {"json": []} + monkeypatch.setattr(el, "emit_binary", + lambda *a: captured.setdefault("bin", a)) + monkeypatch.setattr(el, "emit", lambda o: captured["json"].append(o)) + + pixels = np.arange(64, dtype=np.uint8) + el._route_change("figX", "panel_p1_geom", self._geom_value(pixels)) + + # Pixels shipped as a binary frame… + assert "bin" in captured, "geom image_b64 should route to emit_binary" + fig_id, key, header, raw = captured["bin"] + assert key == "image_b64" + assert header["geom"] == "panel_p1_geom" + assert raw == pixels.tobytes() + # …and the SLIMMED geom (LUT kept) goes as one JSON update, with the + # pixel key replaced by a SMALL content token — NOT removed. The + # renderer's blit/texture caches key "unchanged → skip re-upload" on + # this string; with the key absent, JS fell back to a 4-sampled-byte + # fingerprint that collided across frames and froze the display. + assert len(captured["json"]) == 1 + slim = json.loads(captured["json"][0]["value"]) + tok = slim["image_b64"] + assert tok.startswith("\x00bin:") and len(tok) < 64 + assert slim["colormap_data"] == [[0, 0, 0]] + + def test_geom_stays_base64_when_disabled(self, monkeypatch): + el = self._reload_electron(monkeypatch, binary=False) + captured = {"json": []} + monkeypatch.setattr(el, "emit_binary", + lambda *a: captured.setdefault("bin", a)) + monkeypatch.setattr(el, "emit", lambda o: captured["json"].append(o)) + pixels = np.arange(64, dtype=np.uint8) + geom = self._geom_value(pixels) + el._route_change("figX", "panel_p1_geom", geom) + assert "bin" not in captured + assert captured["json"][0]["value"] == geom # unchanged, pixels included + + def test_non_geom_key_never_binary(self, monkeypatch): + el = self._reload_electron(monkeypatch, binary=True) + captured = {"json": []} + monkeypatch.setattr(el, "emit_binary", + lambda *a: captured.setdefault("bin", a)) + monkeypatch.setattr(el, "emit", lambda o: captured["json"].append(o)) + el._route_change("figX", "fig_width", 640) # a small scalar trait + assert "bin" not in captured + assert captured["json"][0]["value"] == 640 + + def test_detail_tile_goes_binary(self, monkeypatch): + # The zoom DETAIL TILE must ship over PLOTBIN too — else with binary + # transport on, its "\x00bin:" token stays in the geom JSON, the bytes are + # never sent, and the renderer can't decode it → the crisp zoom tile never + # displays (only the downsampled overview base shows). Regression for + # detail_b64 missing from _BINARY_KEYS. + el = self._reload_electron(monkeypatch, binary=True) + assert "detail_b64" in el._BINARY_KEYS # sanity: the key is registered + captured = {"json": [], "bin": []} + monkeypatch.setattr(el, "emit_binary", + lambda *a: captured["bin"].append(a)) + monkeypatch.setattr(el, "emit", lambda o: captured["json"].append(o)) + tile = np.arange(49, dtype=np.uint8) # a 7x7 detail tile + geom = json.dumps({ + "detail_b64": base64.b64encode(tile.tobytes()).decode(), + "detail_region": [0, 7, 0, 7], "detail_width": 7, "detail_height": 7, + "colormap_data": [[0, 0, 0]], + }) + el._route_change("figX", "panel_p1_geom", geom) + keys = [a[1] for a in captured["bin"]] + assert "detail_b64" in keys, f"detail tile did not go binary: {keys}" + # the slimmed geom keeps the small region metadata and carries a small + # content TOKEN under the pixel key (cache identity for the renderer). + slim = json.loads(captured["json"][0]["value"]) + assert slim["detail_b64"].startswith("\x00bin:") + assert len(slim["detail_b64"]) < 64 + assert slim["detail_region"] == [0, 7, 0, 7] + + +class TestRawPixelSideChannel: + """The zero-copy fast path: set_data stashes RAW uint8 bytes on the Figure's + _raw_pixels side-table + puts a tiny token in image_b64, and _route_change + ships those bytes straight to PLOTBIN — no base64 encode/decode round-trip.""" + + def _reload_electron(self, monkeypatch, binary: bool): + monkeypatch.setenv("APL_BINARY_TRANSPORT", "1" if binary else "0") + import anyplotlib._electron as el + importlib.reload(el) + return el + + def test_set_data_stashes_raw_bytes_and_tokenises(self, monkeypatch): + # Binary transport ON: set_data must NOT base64-encode the pixels into + # _state; it stashes raw bytes on the figure and leaves a token. + monkeypatch.setenv("APL_BINARY_TRANSPORT", "1") + import anyplotlib as apl + fig, ax = apl.subplots(1, 1) + img = np.arange(64, dtype=np.uint8).reshape(8, 8) + p = ax.imshow(img) + p.set_data(img.astype(float)) + + tok = p._state["image_b64"] + assert isinstance(tok, str) and tok.startswith("\x00bin:"), \ + "binary path should leave a change-token, not base64, in _state" + raw = fig._raw_pixels.get((p._id, "image_b64")) + assert raw is not None and len(raw) == 64, "raw bytes not stashed on figure" + # to_state_dict passes the token through (wire form); resolve_pixel_tokens + # materialises real base64 for a cold consumer (save_html / standalone). + assert p.to_state_dict()["image_b64"] == tok + st = p.resolve_pixel_tokens(p.to_state_dict()) + assert base64.b64decode(st["image_b64"]) == raw + + def test_route_change_ships_raw_bytes_no_reencode(self, monkeypatch): + # A geom whose image_b64 is a TOKEN → _route_change must pull the bytes + # from fig._raw_pixels (NOT base64-decode the token) and emit them raw. + el = self._reload_electron(monkeypatch, binary=True) + + class _FakeFig: + _raw_pixels = {("p1", "image_b64"): b"\x01\x02\x03\x04rawpixels"} + monkeypatch.setitem(el._figures, "figX", _FakeFig()) + + captured = {"json": []} + monkeypatch.setattr(el, "emit_binary", + lambda *a: captured.setdefault("bin", a)) + monkeypatch.setattr(el, "emit", lambda o: captured["json"].append(o)) + + geom = json.dumps({"image_b64": "\x00bin:12345", + "colormap_data": [[1, 2, 3]]}) + el._route_change("figX", "panel_p1_geom", geom) + + assert "bin" in captured, "token geom should route to emit_binary" + _fig_id, key, header, raw = captured["bin"] + assert key == "image_b64" + assert raw == b"\x01\x02\x03\x04rawpixels", "must ship the side-table bytes verbatim" + # slimmed geom keeps the LUT + the original content token under the + # pixel key (already small — passed through as the cache identity). + slim = json.loads(captured["json"][0]["value"]) + assert slim["image_b64"] == "\x00bin:12345" + assert slim["colormap_data"] == [[1, 2, 3]] + + def test_route_change_falls_back_to_base64_without_sidetable(self, monkeypatch): + # Robustness: a REAL base64 geom (initial frame / no side-table entry) + # still decodes correctly through the same path. + el = self._reload_electron(monkeypatch, binary=True) + captured = {"json": []} + monkeypatch.setattr(el, "emit_binary", + lambda *a: captured.setdefault("bin", a)) + monkeypatch.setattr(el, "emit", lambda o: captured["json"].append(o)) + + pixels = np.arange(16, dtype=np.uint8) + geom = json.dumps({"image_b64": base64.b64encode(pixels.tobytes()).decode(), + "colormap_data": [[0, 0, 0]]}) + el._route_change("figY", "panel_pz_geom", geom) # figY not in _figures + assert "bin" in captured + assert captured["bin"][3] == pixels.tobytes() diff --git a/anyplotlib/tests/test_interactive/test_tile_parity_playwright.py b/anyplotlib/tests/test_interactive/test_tile_parity_playwright.py new file mode 100644 index 00000000..44787f4c --- /dev/null +++ b/anyplotlib/tests/test_interactive/test_tile_parity_playwright.py @@ -0,0 +1,315 @@ +"""Playwright parity tests for tile vs non-tile interactive behaviour. + +These tests use the same interactions on paired figures and require strict +pixel equality so subtle overlay regressions are not hidden by tolerances. +""" +from __future__ import annotations + +import json + +import numpy as np + +import anyplotlib as apl +from anyplotlib.tests._png_utils import compare_arrays, compare_arrays_exact, decode_png +from anyplotlib.tests.test_interactive._event_test_utils import ( + _collect_events, + _get_events, + _plot_center_page, +) + +FIG_W, FIG_H = 400, 300 +PAD_L, PAD_R, PAD_T, PAD_B = 58, 12, 12, 42 + + +def _widget_png(page) -> np.ndarray: + return decode_png(page.locator("#widget-root").screenshot()) + + +def _markers_canvas_rgb(page) -> np.ndarray: + data = page.evaluate( + """() => { + const c = Array.from(document.querySelectorAll('canvas')) + .find(x => x.style && x.style.zIndex === '6'); + if (!c) throw new Error('markers canvas not found'); + const d = c.getContext('2d').getImageData(0, 0, c.width, c.height).data; + return { w: c.width, h: c.height, data: Array.from(d) }; + }""" + ) + arr = np.asarray(data["data"], dtype=np.uint8).reshape(data["h"], data["w"], 4) + return arr[:, :, :3] + + +def _assert_exact( + name: str, + actual: np.ndarray, + expected: np.ndarray, + *, + allow_tolerance: bool = False, +) -> None: + ok, msg = compare_arrays_exact(actual, expected) + if ok: + return + if allow_tolerance: + ok_tol, msg_tol = compare_arrays(actual, expected, tol=4, max_diff_frac=0.003) + assert ok_tol, f"{name}: exact mismatch ({msg}); tolerance mismatch ({msg_tol})" + return + assert ok, f"{name}: {msg}" + + +def _make_scene(tile: bool): + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + # 1024x1024 keeps tile mode active when requested but avoids overview decimation. + img = np.tile(np.linspace(0.0, 1.0, 1024, dtype=np.float32), (1024, 1)) + plot = ax.imshow(img, cmap="gray", vmin=0.0, vmax=1.0, tile=tile, gpu=False) + plot.add_circles( + np.array([[120.0, 180.0], [500.0, 500.0], [860.0, 260.0]], dtype=np.float32), + name="pts", + radius=7, + facecolors="#ff4d4d", + edgecolors="#ffffff", + ) + rect = plot.add_widget("rectangle", x=320.0, y=320.0, w=180.0, h=140.0) + return fig, plot, rect + + +def _rect_center_page(page, panel_id: str, widget_id: str) -> tuple[float, float]: + return tuple(page.evaluate( + """(args) => { + const [pid, wid, padL, padR, padT, padB] = args; + const st = JSON.parse(globalThis.__apl_viewStateJson(pid)); + const w = (st.overlay_widgets || []).find(v => v.id === wid); + if (!w) throw new Error('widget not found'); + const ov = Array.from(document.querySelectorAll('canvas')) + .find(c => c.style && c.style.zIndex === '5'); + if (!ov) throw new Error('overlay canvas not found'); + const r = ov.getBoundingClientRect(); + const availW = ov.width - padL - padR; + const availH = ov.height - padT - padB; + const iw = Math.max(1, st.image_width || 1); + const ih = Math.max(1, st.image_height || 1); + const s = Math.min(availW / iw, availH / ih); + const fitW = iw * s; + const fitH = ih * s; + const ox = padL + (availW - fitW) * 0.5; + const oy = padT + (availH - fitH) * 0.5; + const cx = w.x + w.w * 0.5; + const cy = w.y + w.h * 0.5; + return [r.left + ox + cx * s, r.top + oy + cy * s]; + }""", + [panel_id, widget_id, PAD_L, PAD_R, PAD_T, PAD_B], + )) + + +def _widget_rect(page, panel_id: str, widget_id: str) -> dict: + st = json.loads(page.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", panel_id)) + w = [x for x in st.get("overlay_widgets", []) if x.get("id") == widget_id] + assert w, "rectangle widget not found in state" + return w[0] + + +def _set_widget_visible(page, panel_id: str, widget_id: str, visible: bool) -> None: + page.evaluate( + """(args) => { + const [pid, wid, vis] = args; + const key = 'panel_' + pid + '_json'; + const st = JSON.parse(window._aplModel.get(key)); + const ws = st.overlay_widgets || []; + const w = ws.find(x => x.id === wid); + if (!w) throw new Error('widget not found: ' + wid); + w.visible = !!vis; + st.overlay_widgets = ws; + window._aplModel.set(key, JSON.stringify(st)); + }""", + [panel_id, widget_id, visible], + ) + + +def _set_circles_offsets(page, panel_id: str, offsets: list[list[float]]) -> None: + page.evaluate( + """(args) => { + const [pid, offsets] = args; + const key = 'panel_' + pid + '_json'; + const st = JSON.parse(window._aplModel.get(key)); + const markers = st.markers || []; + const g = markers.find(m => m.type === 'circles'); + if (!g) throw new Error('circles marker group not found'); + g.offsets = offsets; + if (Array.isArray(g.sizes)) { + const s = g.sizes.length > 0 ? g.sizes[0] : 7; + g.sizes = offsets.map(() => s); + } + st.markers = markers; + window._aplModel.set(key, JSON.stringify(st)); + }""", + [panel_id, offsets], + ) + + +def _make_rich_marker_scene(tile: bool): + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + x = np.linspace(0.0, 1.0, 1024, dtype=np.float32) + y = np.linspace(0.0, 1.0, 1024, dtype=np.float32) + img = (np.sin(6 * np.pi * x)[None, :] * np.cos(4 * np.pi * y)[:, None]).astype(np.float32) + plot = ax.imshow(img, cmap="gray", vmin=-1.0, vmax=1.0, tile=tile, gpu=False) + plot.add_circles( + np.array([[140.0, 220.0], [520.0, 520.0], [900.0, 300.0]], dtype=np.float32), + name="cir", + radius=7, + facecolors="#ff4d4d", + edgecolors="#ffffff", + ) + plot.add_lines( + [ + [[120.0, 120.0], [920.0, 760.0]], + [[180.0, 860.0], [860.0, 180.0]], + ], + name="ln", + edgecolors="#00e5ff", + linewidths=2.5, + ) + plot.add_polygons( + [[[250.0, 250.0], [330.0, 240.0], [300.0, 330.0]]], + name="poly", + facecolors="#ff9100", + edgecolors="#ff9100", + alpha=0.35, + ) + return fig, plot + + +class TestTileInteractiveParity: + def test_initial_marker_widget_overlay_matches(self, interact_page): + fig_plain, p_plain, _ = _make_scene(tile=False) + fig_tile, p_tile, _ = _make_scene(tile=True) + page_plain = interact_page(fig_plain) + page_tile = interact_page(fig_tile) + page_plain.wait_for_timeout(120) + page_tile.wait_for_timeout(120) + + # Sanity: this test should exercise tile mode on one side. + st_tile = json.loads(page_tile.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p_tile._id)) + assert st_tile.get("tile_enabled") is True + + arr_plain = _widget_png(page_plain) + arr_tile = _widget_png(page_tile) + _assert_exact("initial marker/widget overlay parity", arr_tile, arr_plain) + + def test_widget_drag_parity_events_and_pixels(self, interact_page): + fig_plain, p_plain, r_plain = _make_scene(tile=False) + fig_tile, p_tile, r_tile = _make_scene(tile=True) + page_plain = interact_page(fig_plain) + page_tile = interact_page(fig_tile) + _collect_events(page_plain) + _collect_events(page_tile) + + sx_plain, sy_plain = _rect_center_page(page_plain, p_plain._id, r_plain.id) + sx_tile, sy_tile = _rect_center_page(page_tile, p_tile._id, r_tile.id) + + for page, sx, sy in ( + (page_plain, sx_plain, sy_plain), + (page_tile, sx_tile, sy_tile), + ): + page.mouse.move(sx, sy) + page.mouse.down() + page.mouse.move(sx + 35, sy + 20, steps=10) + page.mouse.up() + page.wait_for_timeout(120) + + rect_plain = _widget_rect(page_plain, p_plain._id, r_plain.id) + rect_tile = _widget_rect(page_tile, p_tile._id, r_tile.id) + for key in ("x", "y", "w", "h"): + assert rect_tile[key] == rect_plain[key], ( + f"widget field {key} diverged: tile={rect_tile[key]} plain={rect_plain[key]}" + ) + + ev_plain = _get_events(page_plain) + ev_tile = _get_events(page_tile) + pm_plain = len([e for e in ev_plain if e.get("event_type") == "pointer_move"]) + pm_tile = len([e for e in ev_tile if e.get("event_type") == "pointer_move"]) + pu_plain = len([e for e in ev_plain if e.get("event_type") == "pointer_up"]) + pu_tile = len([e for e in ev_tile if e.get("event_type") == "pointer_up"]) + assert pm_plain > 0 and pu_plain > 0 + assert pm_tile == pm_plain and pu_tile == pu_plain + + arr_plain = _widget_png(page_plain) + arr_tile = _widget_png(page_tile) + _assert_exact("post-drag marker/widget overlay parity", arr_tile, arr_plain) + + def test_zoomed_marker_overlay_matches(self, interact_page): + fig_plain, p_plain, _ = _make_scene(tile=False) + fig_tile, p_tile, _ = _make_scene(tile=True) + page_plain = interact_page(fig_plain) + page_tile = interact_page(fig_tile) + + cx, cy = _plot_center_page(FIG_W, FIG_H) + for page in (page_plain, page_tile): + page.mouse.move(cx, cy) + page.mouse.wheel(0, -500) + page.wait_for_timeout(160) + + arr_plain = _markers_canvas_rgb(page_plain) + arr_tile = _markers_canvas_rgb(page_tile) + _assert_exact( + "zoomed marker-canvas parity", + arr_tile, + arr_plain, + allow_tolerance=True, + ) + + st_plain = json.loads(page_plain.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p_plain._id)) + st_tile = json.loads(page_tile.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p_tile._id)) + assert st_tile.get("zoom") == st_plain.get("zoom") + + def test_rich_marker_render_parity(self, interact_page): + fig_plain, p_plain = _make_rich_marker_scene(tile=False) + fig_tile, p_tile = _make_rich_marker_scene(tile=True) + page_plain = interact_page(fig_plain) + page_tile = interact_page(fig_tile) + page_plain.wait_for_timeout(120) + page_tile.wait_for_timeout(120) + + st_tile = json.loads(page_tile.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p_tile._id)) + assert st_tile.get("tile_enabled") is True + + arr_plain = _widget_png(page_plain) + arr_tile = _widget_png(page_tile) + _assert_exact("rich marker render parity", arr_tile, arr_plain) + + def test_marker_offsets_update_parity(self, interact_page): + fig_plain, p_plain, _ = _make_scene(tile=False) + fig_tile, p_tile, _ = _make_scene(tile=True) + page_plain = interact_page(fig_plain) + page_tile = interact_page(fig_tile) + + offsets = [[180.0, 180.0], [540.0, 600.0], [860.0, 260.0]] + _set_circles_offsets(page_plain, p_plain._id, offsets) + _set_circles_offsets(page_tile, p_tile._id, offsets) + page_plain.wait_for_timeout(80) + page_tile.wait_for_timeout(80) + + arr_plain = _markers_canvas_rgb(page_plain) + arr_tile = _markers_canvas_rgb(page_tile) + _assert_exact("marker offset update parity", arr_tile, arr_plain) + + def test_widget_visibility_toggle_parity(self, interact_page): + fig_plain, p_plain, _ = _make_scene(tile=False) + fig_tile, p_tile, _ = _make_scene(tile=True) + w_plain = p_plain.add_widget("crosshair", cx=700.0, cy=720.0) + w_tile = p_tile.add_widget("crosshair", cx=700.0, cy=720.0) + + page_plain = interact_page(fig_plain) + page_tile = interact_page(fig_tile) + + _set_widget_visible(page_plain, p_plain._id, w_plain.id, False) + _set_widget_visible(page_tile, p_tile._id, w_tile.id, False) + page_plain.wait_for_timeout(80) + page_tile.wait_for_timeout(80) + + arr_plain = _widget_png(page_plain) + arr_tile = _widget_png(page_tile) + _assert_exact("widget visibility toggle parity", arr_tile, arr_plain) + + + + + diff --git a/anyplotlib/tests/test_interactive/test_title.py b/anyplotlib/tests/test_interactive/test_title.py index b8d05101..5fe822e9 100644 --- a/anyplotlib/tests/test_interactive/test_title.py +++ b/anyplotlib/tests/test_interactive/test_title.py @@ -143,9 +143,9 @@ def test_title_above_image_not_overlapping(self, interact_page): page.wait_for_timeout(200) plot_canvas_top = page.evaluate("""() => { - // z-index auto = plotCanvas (the image canvas) - const canvases = Array.from(document.querySelectorAll('canvas')); - const pc = canvases.find(c => !c.style.zIndex && c.style.position === 'absolute'); + // plotCanvas is the FIRST canvas in DOM order (a WebGPU image canvas, + // if present, is appended after it and sits below via z-index). + const pc = document.querySelector('canvas'); return pc ? pc.style.top : null; }""") diff --git a/anyplotlib/tests/test_layouts/test_visual.py b/anyplotlib/tests/test_layouts/test_visual.py index 4487dc55..09bb000a 100644 --- a/anyplotlib/tests/test_layouts/test_visual.py +++ b/anyplotlib/tests/test_layouts/test_visual.py @@ -36,7 +36,12 @@ import pytest import anyplotlib as apl -from anyplotlib.tests._png_utils import decode_png, encode_png, compare_arrays +from anyplotlib.tests._png_utils import ( + decode_png, + encode_png, + compare_arrays, + compare_arrays_exact, +) BASELINES = pathlib.Path(__file__).parent.parent / "baselines" @@ -64,6 +69,37 @@ def _check(name: str, arr: np.ndarray, update: bool) -> None: assert ok, f"Visual regression [{name}]: {msg}" +def _check_exact_pair(name: str, a: np.ndarray, b: np.ndarray) -> None: + """Assert two screenshots are strictly identical (no tolerance).""" + ok, msg = compare_arrays_exact(a, b) + assert ok, f"Tile parity [{name}] failed: {msg}" + + +def _make_imshow_scene(scene: str, *, tile: bool): + """Build one of the existing imshow baseline scenes with tile mode control.""" + if scene == "gradient": + fig, ax = apl.subplots(1, 1, figsize=(320, 320)) + data = np.linspace(0.0, 1.0, 64 * 64, dtype=np.float32).reshape(64, 64) + ax.imshow(data, tile=tile, gpu=False) + return fig, "imshow_gradient" + + if scene == "checkerboard": + fig, ax = apl.subplots(1, 1, figsize=(256, 256)) + board = np.indices((32, 32)).sum(axis=0) % 2 + ax.imshow(board.astype(np.float32), tile=tile, gpu=False) + return fig, "imshow_checkerboard" + + if scene == "viridis": + fig, ax = apl.subplots(1, 1, figsize=(320, 256)) + rng = np.random.default_rng(0) + data = rng.uniform(0.0, 1.0, (48, 64)).astype(np.float32) + plot = ax.imshow(data, tile=tile, gpu=False) + plot.set_colormap("viridis") + return fig, "imshow_viridis" + + raise ValueError(f"Unknown scene: {scene}") + + # --------------------------------------------------------------------------- # Tests # --------------------------------------------------------------------------- @@ -341,3 +377,26 @@ def test_imshow_axis_off(self, take_screenshot, update_baselines): arr = take_screenshot(fig) _check("imshow_axis_off", arr, update_baselines) + +class TestTileParityVisual: + """Tile backend must match non-tile rendering for baseline imshow scenes.""" + + @pytest.mark.parametrize("scene", ["gradient", "checkerboard", "viridis"]) + def test_imshow_tile_matches_plain_and_baseline( + self, + scene, + take_screenshot, + update_baselines, + ): + fig_plain, baseline = _make_imshow_scene(scene, tile=False) + fig_tile, _ = _make_imshow_scene(scene, tile=True) + arr_plain = take_screenshot(fig_plain) + arr_tile = take_screenshot(fig_tile) + + # Strict parity check first: catches subtle backend divergences. + _check_exact_pair(f"{baseline}_tile_vs_plain", arr_tile, arr_plain) + + # Existing goldens remain the source of truth for scene correctness. + _check(baseline, arr_plain, update_baselines) + + diff --git a/anyplotlib/tests/test_markers/test_marker_transforms.py b/anyplotlib/tests/test_markers/test_marker_transforms.py index 25aeb63b..e68bfe97 100644 --- a/anyplotlib/tests/test_markers/test_marker_transforms.py +++ b/anyplotlib/tests/test_markers/test_marker_transforms.py @@ -56,6 +56,17 @@ def test_transform_display_round_trips(self): w = g.to_wire("gid") assert w["transform"] == "display" + def test_clip_display_defaults_true(self): + g = _group("circles", offsets=[[8.0, 8.0]], transform="display") + w = g.to_wire("gid") + assert w["clip_display"] is True + + def test_clip_display_round_trips_false(self): + g = _group("circles", offsets=[[8.0, 8.0]], transform="display", + clip_display=False) + w = g.to_wire("gid") + assert w["clip_display"] is False + def test_transform_data_explicit(self): g = _group("rectangles", offsets=[[0.0, 0.0]], widths=10, heights=10, transform="data") @@ -109,6 +120,15 @@ def test_valid_transforms_do_not_raise(self): for tfm in ("data", "axes", "display"): _group("circles", offsets=[[1, 2]], transform=tfm) # no error + def test_invalid_clip_display_raises(self): + with pytest.raises(ValueError, match="clip_display"): + _group("circles", offsets=[[1, 2]], clip_display="nope") + + def test_invalid_clip_display_raises_on_set(self): + g = _group("circles", offsets=[[1, 2]]) + with pytest.raises(ValueError, match="clip_display"): + g.set(clip_display=1) + # --------------------------------------------------------------------------- # set() preserves transform @@ -160,6 +180,12 @@ def test_add_rectangles_transform_display(self): wire = self.plot.markers.to_wire_list() assert wire[0]["transform"] == "display" + def test_add_rectangles_clip_display_false(self): + self.plot.add_rectangles([[5, 5]], widths=10, heights=10, name="r2", + transform="display", clip_display=False) + wire = self.plot.markers.to_wire_list() + assert wire[0]["clip_display"] is False + def test_add_arrows_transform_axes(self): g = self.plot.add_arrows([[5, 5]], U=1, V=1, name="a", transform="axes") wire = self.plot.markers.to_wire_list() @@ -218,6 +244,12 @@ def test_add_texts_transform_axes(self): wire = self.plot.markers.to_wire_list() assert wire[0]["transform"] == "axes" + def test_add_texts_clip_display_false(self): + self.plot.add_texts([[8.0, 8.0]], ["hud"], name="t2", + transform="display", clip_display=False) + wire = self.plot.markers.to_wire_list() + assert wire[0]["clip_display"] is False + def test_default_transform_is_data(self): self.plot.add_vlines([0.5], name="v2") wire = self.plot.markers.to_wire_list() diff --git a/anyplotlib/tests/test_plot2d/test_detail_tile.py b/anyplotlib/tests/test_plot2d/test_detail_tile.py new file mode 100644 index 00000000..bbf37a91 --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_detail_tile.py @@ -0,0 +1,419 @@ +"""Detail-tile viewport LOD — a hi-res tile for a logical sub-region is sampled by +the renderer when the zoom window is inside it, so a zoom-in shows crisp native +pixels WITHOUT transferring the whole full-res frame. + +Playwright's bundled Chromium has no WebGPU, so these exercise the Canvas2D path +(the GPU path mirrors the same _detailUV math and is verified in the app on real +hardware). We prove the tile is actually sampled by making the tile's content +DIFFERENT from the base in the same region: after zooming into the region the +visible pixels must reflect the TILE, not the base. +""" +from __future__ import annotations + +import json +import numpy as np + +import anyplotlib as apl + + +class TestDetailTileState: + def test_set_detail_populates_state(self): + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((64, 64), np.float32), vmin=0, vmax=1, gpu=False) + tile = np.ones((32, 32), np.float32) + p.set_detail(tile, 16, 48, 16, 48) + assert p._state["detail_region"] == [16, 48, 16, 48] + assert p._state["detail_width"] == 32 and p._state["detail_height"] == 32 + assert p._state["detail_b64"] # binary token or base64 + p.set_detail(None) + assert p._state["detail_b64"] == "" and p._state["detail_region"] == [] + + def test_set_data_clears_stale_detail(self): + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((64, 64), np.float32), vmin=0, vmax=1, gpu=False) + p.set_detail(np.ones((32, 32), np.float32), 16, 48, 16, 48) + assert p._state["detail_b64"] + p.set_data(np.zeros((64, 64), np.float32)) + assert p._state["detail_b64"] == "", "a new base frame must clear the detail" + + +class TestDetailSeqAndForcePlain: + """Regressions for the SpyDE movie viewer: (1) a live scrub while zoomed in must + bump detail_seq every push so the renderer re-uploads the re-sampled tile (else the + zoomed-in view freezes on the first frame); (2) tile=False forces a plain full-frame + push even for a large frame / an already-tiled plot (so a pre-decimated frame isn't + auto-tiled at the wrong logical size).""" + + def test_detail_seq_advances_per_push(self): + from anyplotlib.callbacks import Event + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10), np.float32)) + p.set_data(np.random.RandomState(0).rand(4096, 4096).astype(np.float32), + clim=(0, 1), tile=True) + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + reg = list(p._state["detail_region"]) + seqs = [p._state["detail_seq"]] + # Scrub frames: SAME region (zoom unchanged), new data each time. + for i in range(3): + p.set_data(np.random.RandomState(10 + i).rand(4096, 4096).astype(np.float32), + clim=(0, 1), tile=True) + assert list(p._state["detail_region"]) == reg # region unchanged + seqs.append(p._state["detail_seq"]) + assert seqs == sorted(seqs) and len(set(seqs)) == len(seqs), ( + f"detail_seq must advance per push (JS dedup key) — freeze bug: {seqs}") + + def test_tile_false_forces_plain_on_large_frame(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10), np.float32)) + # A large frame that WOULD auto-tile, but tile=False → plain full-res push. + p.set_data(np.random.RandomState(0).rand(1366, 1366).astype(np.float32), + clim=(0, 1), tile=False) + assert p._state["tile_enabled"] is False + assert p._state["image_width"] == 1366 and p._state["base_width"] == 0 + + def test_tile_false_tears_down_existing_tiling(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10), np.float32)) + p.set_data(np.random.RandomState(0).rand(4096, 4096).astype(np.float32), + clim=(0, 1), tile=True) + assert p._state["tile_enabled"] is True + # Now a tile=False frame must LEAVE tile mode (plain push, base_width=0). + p.set_data(np.random.RandomState(1).rand(800, 800).astype(np.float32), + clim=(0, 1), tile=False) + assert p._state["tile_enabled"] is False + assert p._state["image_width"] == 800 and p._state["base_width"] == 0 + + +class TestOverviewStaleness: + """A zoomed-in scrub refreshes ONLY the detail tile (the overview base isn't + visible, so re-encoding it per frame is wasted work). But the base then holds + the PRE-scrub frame — a zoom-out (or a pan past the tile edge) would flash old + data. Contract: the skipped overview is marked stale and re-sampled ONCE on + the next view settle, riding the same push as the detail/clear.""" + + def _zoomed_scrubbed_plot(self): + from anyplotlib.callbacks import Event + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10), np.float32)) + p.set_data(np.random.RandomState(0).rand(4096, 4096).astype(np.float32), + clim=(0, 1), tile=True) + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + assert list(p._state["detail_region"]) # zoomed in, tile shown + base_tok = p._state["image_b64"] + # Scrub while zoomed: only the detail refreshes; the base is untouched + # (the optimisation) but must now be flagged stale. + p.set_data(np.random.RandomState(7).rand(4096, 4096).astype(np.float32), + clim=(0, 1), tile=True) + assert p._state["image_b64"] == base_tok, "overview must NOT re-encode per scrub frame" + assert p._overview_stale is True + return p, base_tok + + def test_zoom_out_after_zoomed_scrub_refreshes_overview(self): + from anyplotlib.callbacks import Event + p, base_tok = self._zoomed_scrubbed_plot() + # Zoom out below the tile threshold: the tile clears AND the stale + # overview re-samples from the CURRENT frame (no pre-scrub flash). + p.callbacks.fire(Event("view_changed", zoom=1.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + assert p._state["detail_b64"] == "" and p._state["detail_region"] == [] + assert p._state["image_b64"] != base_tok, ( + "stale overview survived zoom-out — the view flashes the pre-scrub frame") + assert p._overview_stale is False + + def test_settle_while_zoomed_refreshes_stale_overview(self): + from anyplotlib.callbacks import Event + p, base_tok = self._zoomed_scrubbed_plot() + # A pan settle while STILL zoomed: the margin around a partially-covering + # tile shows the base, so the stale overview refreshes here too. + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.3, center_y=0.6, + display_width=1000, display_height=1000)) + assert p._state["image_b64"] != base_tok + assert p._overview_stale is False + assert list(p._state["detail_region"]) # new tile also landed + + def test_zoom_out_without_scrub_keeps_overview(self): + from anyplotlib.callbacks import Event + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10), np.float32)) + p.set_data(np.random.RandomState(0).rand(4096, 4096).astype(np.float32), + clim=(0, 1), tile=True) + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + base_tok = p._state["image_b64"] + # No scrub happened → zoom-out must NOT pay an overview re-encode. + p.callbacks.fire(Event("view_changed", zoom=1.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + assert p._state["image_b64"] == base_tok + assert p._state["detail_b64"] == "" + + +class TestDetailTileCanvasRender: + def test_zoom_out_strict_visible_rect_clips_data_markers(self, interact_page): + img = np.zeros((64, 64), np.float32) + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(img, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + # Large radius at the image corner would bleed into the zoom-out margin + # without strict clipping to the shrunken visible rect. + p.add_circles([[0.0, 0.0]], name="edge", radius=18, + edgecolors="#ffffff", facecolors="#ffffff") + page = interact_page(fig) + page.wait_for_timeout(250) + page.evaluate("(pid) => globalThis.__apl_setZoom(pid, 0.75, 0.5, 0.5)", p._id) + page.wait_for_timeout(120) + + outside_alpha = page.evaluate( + """(pid) => { + const st = JSON.parse(globalThis.__apl_viewStateJson(pid)); + const c = Array.from(document.querySelectorAll('canvas')) + .find(x => x.style && x.style.zIndex === '6'); + if (!c) throw new Error('markers canvas not found'); + const ctx = c.getContext('2d'); + const d = ctx.getImageData(0, 0, c.width, c.height).data; + const iw = st.image_width, ih = st.image_height; + const z = st.zoom || 1; + const s = Math.min(c.width / iw, c.height / ih); + const fw = iw * s, fh = ih * s; + let x = (c.width - fw) / 2, y = (c.height - fh) / 2; + let w = fw, h = fh; + if (z < 1) { + w = fw * z; h = fh * z; + x += (fw - w) / 2; y += (fh - h) / 2; + } + let out = 0; + for (let py = 0; py < c.height; py++) { + for (let px = 0; px < c.width; px++) { + if (px >= x && px <= x + w && py >= y && py <= y + h) continue; + out += d[(py * c.width + px) * 4 + 3]; + } + } + return out; + }""", + p._id, + ) + assert outside_alpha == 0, ( + f"markers leaked outside strict zoom-out visible rect: outside alpha={outside_alpha}") + + def test_display_markers_allow_clip_opt_out(self, interact_page): + img = np.zeros((64, 64), np.float32) + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(img, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + # Display-space marker near canvas top-left (outside zoom-out visible rect). + # clip_display=False keeps it visible as a HUD-style annotation. + hud = p.add_circles([[8.0, 8.0]], name="hud", radius=10, + edgecolors="#ffffff", facecolors="#ffffff", + transform="display") + hud.set(clip_display=False) + page = interact_page(fig) + page.wait_for_timeout(250) + page.evaluate("(pid) => globalThis.__apl_setZoom(pid, 0.75, 0.5, 0.5)", p._id) + page.wait_for_timeout(120) + + outside_alpha = page.evaluate( + """(pid) => { + const st = JSON.parse(globalThis.__apl_viewStateJson(pid)); + const c = Array.from(document.querySelectorAll('canvas')) + .find(x => x.style && x.style.zIndex === '6'); + if (!c) throw new Error('markers canvas not found'); + const ctx = c.getContext('2d'); + const d = ctx.getImageData(0, 0, c.width, c.height).data; + const iw = st.image_width, ih = st.image_height; + const z = st.zoom || 1; + const s = Math.min(c.width / iw, c.height / ih); + const fw = iw * s, fh = ih * s; + let x = (c.width - fw) / 2, y = (c.height - fh) / 2; + let w = fw, h = fh; + if (z < 1) { + w = fw * z; h = fh * z; + x += (fw - w) / 2; y += (fh - h) / 2; + } + let out = 0; + for (let py = 0; py < c.height; py++) { + for (let px = 0; px < c.width; px++) { + if (px >= x && px <= x + w && py >= y && py <= y + h) continue; + out += d[(py * c.width + px) * 4 + 3]; + } + } + return out; + }""", + p._id, + ) + assert outside_alpha > 0, ( + "display markers with clip_display=False should remain visible outside the " + "strict zoom-out visible rect") + + def test_tile_pixels_show_when_zoomed_into_region(self, interact_page): + # BASE = pure black everywhere. TILE = mid-gray left half, white right half. + # If the renderer samples the tile in the zoomed region, the visible pixels + # are gray/white (from the tile), NOT black (base). Distinct values make it + # unambiguous which source is showing. + base = np.zeros((64, 64), np.float32) # base → black + tile = np.full((32, 32), 0.5, np.float32) # tile left half → mid gray + tile[:, 16:] = 1.0 # tile right half → white + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(base, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + p.set_detail(tile, 16, 48, 16, 48) # tile covers image region [16,48]² + page = interact_page(fig) + page.wait_for_timeout(300) + + state = json.loads(page.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p._id)) + assert state.get("detail_region") == [16, 48, 16, 48] + + # Zoom to exactly the tile region: zoom=2, center (0.5,0.5) → visible window + # = 32×32 image px centered → [16,48] → the whole tile fills the view. + page.evaluate("(pid) => globalThis.__apl_setZoom(pid, 2.0, 0.5, 0.5)", p._id) + page.wait_for_timeout(200) + + # Sample the largest canvas (the image) across a mid scanline: the image is + # centered/letterboxed, so probe inside the fit-rect (0.2 and 0.8 of width). + info = page.evaluate("""() => { + const cs = Array.from(document.querySelectorAll('canvas')); + const c = cs.sort((a,b)=>b.width*b.height-a.width*a.height)[0]; + const ctx = c.getContext('2d'); + const w = c.width, h = c.height, y = (h*0.5)|0; + const left = Array.from(ctx.getImageData((w*0.30)|0, y, 1, 1).data); + const right = Array.from(ctx.getImageData((w*0.70)|0, y, 1, 1).data); + return { left, right }; + }""") + lval, rval = info["left"][0], info["right"][0] + # Left = tile mid-gray (~128), right = tile white (~255); NOT black (base=0). + assert lval > 60, f"tile left half not shown (got {lval}, base-black leak?)" + assert rval > 200, f"tile right (white) half not shown (got {rval})" + assert rval > lval + 60, f"tile gray/white split not visible: L={lval} R={rval}" + + def test_partial_cover_stitches_tile_over_base(self, interact_page): + # ANTI-FLASH: when zoomed IN but the visible window is LARGER than the tile + # region (the zoom-OUT transition state, before a wider tile arrives), the + # crisp tile must still show OVER its region while the base fills the margin — + # NOT snap entirely to the blurry base (the old jarring flash). Base = black, + # tile = white: tile region reads white, the margin outside reads base-black. + base = np.zeros((64, 64), np.float32) # base → black + tile = np.ones((32, 32), np.float32) # tile → white + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(base, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + p.set_detail(tile, 16, 48, 16, 48) # tile covers image region [16,48]² + page = interact_page(fig) + page.wait_for_timeout(300) + # Zoom=1.5 centered: visible window = 64/1.5 ≈ 43 px centered → [~10.7, 53.3], + # which is LARGER than the tile region [16,48] → partial cover (the overlay + # path, not _detailUV). Centre must be tile-white; the corners base-black. + page.evaluate("(pid) => globalThis.__apl_setZoom(pid, 1.5, 0.5, 0.5)", p._id) + page.wait_for_timeout(200) + info = page.evaluate("""() => { + const cs = Array.from(document.querySelectorAll('canvas')); + const c = cs.sort((a,b)=>b.width*b.height-a.width*a.height)[0]; + const ctx = c.getContext('2d'); + const w = c.width, h = c.height; + const center = Array.from(ctx.getImageData((w*0.5)|0, (h*0.5)|0, 1, 1).data); + const corner = Array.from(ctx.getImageData((w*0.12)|0, (h*0.5)|0, 1, 1).data); + return { center: center[0], corner: corner[0] }; + }""") + # Centre is inside the tile region → crisp white tile shows (NOT base-black). + assert info["center"] > 200, ( + f"tile not stitched over base at partial cover (flash bug): {info}") + # The margin outside the tile region → base (black), proving it's a COMPOSITE + # (base + tile), not the tile stretched over everything. + assert info["corner"] < 80, ( + f"margin should show the base, not the tile: {info}") + + def test_detail_tile_renders_via_BINARY_transport(self, interact_page): + # THE bug: under binary transport the detail tile's pixels arrive as + # detail_b64_bytes (a Uint8Array) and detail_b64 (the base64 string) is ABSENT + # from state. The render gates (_detailUV / _detailOverlayRect / _gpuUploadDetail) + # must detect the tile via _hasDetail (bytes OR string) — gating on detail_b64 + # alone made binary transport NEVER draw the detail (only the blurry overview), + # so a zoomed-in movie showed a low-res upscale. Here: base black, tile white; + # inject the tile as BYTES ONLY (detail_b64 empty) and zoom in → must be white. + base = np.zeros((64, 64), np.float32) + tile = np.ones((32, 32), np.float32) # white tile + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(base, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + p.set_detail(tile, 16, 48, 16, 48) # build the tile bytes + st = p._state + # The quantised uint8 bytes anyplotlib would ship on the binary channel. + from anyplotlib._utils import _normalize_image + u8, _, _ = _normalize_image(tile, clim=(0.0, 1.0)) + page = interact_page(fig) + page.wait_for_timeout(300) + # Inject EXACTLY like binary transport: detail_b64='' (absent), the pixels in + # a detail_b64_bytes Uint8Array on the geomCache, and the small region/size + # fields on the light state. + page.evaluate("""(args) => { + const [pid, bytes, reg, dw, dh] = args; + const u8 = new Uint8Array(bytes); + const gname = 'panel_'+pid+'_geom'; + let geom = {}; try { geom = JSON.parse(window._aplModel.get(gname)||'{}'); } catch(_){} + delete geom.detail_b64; // binary strips the string + window._aplModel.set(gname, JSON.stringify(geom)); + // Stash bytes the way the binary handler does (geomCache::detail_b64). + (globalThis.__apl_pixbytes ||= {})[gname+'::detail_b64'] = u8; + window._aplModel.set(gname+'::detail_b64', u8.length+':bintest'); + const raw = JSON.parse(window._aplModel.get('panel_'+pid+'_json')); + raw.detail_b64 = ''; // NO base64 string + raw.detail_region = reg; raw.detail_width = dw; raw.detail_height = dh; + raw.detail_seq = 1; + window._aplModel.set('panel_'+pid+'_json', JSON.stringify(raw)); + globalThis.__apl_setZoom(pid, 2.0, 0.5, 0.5); // window ⊆ [16,48] + }""", [p._id, list(int(b) for b in u8.tobytes()), [16, 48, 16, 48], + int(st["detail_width"]), int(st["detail_height"])]) + page.wait_for_timeout(200) + center = page.evaluate("""() => { + const cs = Array.from(document.querySelectorAll('canvas')); + const c = cs.sort((a,b)=>b.width*b.height-a.width*a.height)[0]; + const ctx = c.getContext('2d'); + return ctx.getImageData((c.width*0.5)|0, (c.height*0.5)|0, 1, 1).data[0]; + }""") + assert center > 200, ( + f"detail tile from BINARY bytes did not render (got {center}, base-black " + f"leaked — _hasDetail gate bug)") + + def test_zoomed_out_uses_base_not_tile(self, interact_page): + # At zoom 1 (whole image visible), the window is NOT inside the tile region, + # so the base is shown — the tile only kicks in when zoomed into its region. + base = np.full((64, 64), 0.25, np.float32) + tile = np.full((32, 32), 1.0, np.float32) # very bright tile + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(base, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + p.set_detail(tile, 16, 48, 16, 48) + page = interact_page(fig) + page.wait_for_timeout(300) + # zoom 1 → full image visible → tile NOT active → center pixel ≈ base (dark). + px = page.evaluate("""() => { + const c = document.querySelector('canvas'); + const ctx = c.getContext('2d'); + return Array.from(ctx.getImageData((c.width*0.5)|0, (c.height*0.5)|0, 1, 1).data); + }""") + # base 0.25 on gray → ~64/255; the bright tile (255) must NOT be showing. + assert sum(px[:3]) < 350, f"tile leaked at zoom-out: {px[:3]}" + + def test_tiled_contrast_windows_via_LUT_no_reencode(self, interact_page): + # A contrast change on a TILED plot must re-window via the LUT ONLY (move + # display_min/display_max over the fixed raw_min/raw_max band) — no pixel + # re-encode/re-transfer (the lag bug). We prove the DISPLAY changes: a mid- + # gray overview gets brighter when the display window narrows toward it, all + # by pushing only display_min/display_max (the bytes stay resident). + # Non-uniform data (a horizontal ramp 0→1) so the raw band is real; the + # centre pixel sits at ~0.5. A tiled plot (>threshold). + img = np.tile(np.linspace(0, 1, 2048, dtype=np.float32), (2048, 1)) + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(img, cmap="gray", vmin=0.0, vmax=1.0, gpu=False) + assert p._state["tile_enabled"] is True + page = interact_page(fig) + page.wait_for_timeout(300) + sample = """() => { + const cs = Array.from(document.querySelectorAll('canvas')); + const c = cs.sort((a,b)=>b.width*b.height-a.width*a.height)[0]; + return c.getContext('2d').getImageData((c.width*0.5)|0,(c.height*0.5)|0,1,1).data[0]; + }""" + before = page.evaluate(sample) + # Move the display window IN-BROWSER (like the live model would on a contrast + # drag) — only display_min/display_max change; the resident bytes + fixed + # raw_min/raw_max band are untouched. The centre pixel must re-window via the + # LUT (no pixel re-encode). Window → [0.5,1.0] puts the centre value (~0.5) at + # the dark end. + page.evaluate("""(pid) => { + const key = 'panel_'+pid+'_json'; + const st = JSON.parse(window._aplModel.get(key)); + st.display_min = 0.5; st.display_max = 1.0; + window._aplModel.set(key, JSON.stringify(st)); + }""", p._id) + page.wait_for_timeout(150) + after = page.evaluate(sample) + assert abs(after - before) > 20, ( + f"tiled contrast did not change the display via LUT: {before}→{after}") diff --git a/anyplotlib/tests/test_plot2d/test_first_paint_race.py b/anyplotlib/tests/test_plot2d/test_first_paint_race.py new file mode 100644 index 00000000..82cdf699 --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_first_paint_race.py @@ -0,0 +1,259 @@ +"""First-paint race — a pixel frame that ARRIVES BEFORE ``render()`` must still +paint on the FIRST draw, not leave the panel permanently blank. + +The bug +------- +Every SpyDE figure iframe loads its anywidget ESM via an async +``import(blobUrl)``; ``render()`` (which registers the panel's +``model.on('change:panel__geom', …)`` listeners) therefore runs on a LATER +microtask than the synchronously-registered ``window.addEventListener('message')`` +handler. If the FIRST real pixel push lands in that gap: + +* **binary transport** — the raw uint8 bytes are stashed in the global + ``__apl_pixbytes`` side-table (the model can't carry a ``Uint8Array``) and a + tiny token trait is ``set()``. With no listener registered yet, nothing + consumes the bytes; and the panel's INITIAL paint only parses the slimmed geom + JSON (LUT/flags), which does NOT contain the bytes — so the frame is stranded + in the side-table and the panel stays permanently blank. +* **base64 transport** — the pixels ride the ``panel__geom`` JSON. If that + trait arrived before render, the initial ``_loadGeom`` DOES pick it up; this + test also guards that path so a future refactor can't regress it. + +The fix (figure_esm ``_spliceBinaryBytes`` + its call at panel-init) makes the +initial paint consume any already-arrived binary side-table bytes, mirroring +what the change listener would have done. + +The test drives the ORDER deterministically (it does NOT try to win a real +race): render is gated behind ``window.__aplTestGate``; the test dispatches the +pixel ``message`` event FIRST (exactly as the parent bridge would), THEN releases +render, then asserts the canvas painted the delivered frame (non-blank + the +right brightness). The Canvas2D fallback path is exercised (the splice is +GPU-independent), so it runs in Playwright's WebGPU-less Chromium. +""" +from __future__ import annotations + +import base64 +import json +import pathlib +import tempfile + +import numpy as np + +import anyplotlib as apl + + +FIG_W, FIG_H = 240, 240 +N = 128 # small → plain Canvas2D blit path (no tile, no GPU dependence) + + +def _bright_frame() -> np.ndarray: + """A frame that is BRIGHT on the right half, dark on the left — an easy, + unambiguous non-blank signature (mean brightness well above zero, and a + left/right asymmetry a blank panel can't fake).""" + img = np.zeros((N, N), dtype=np.float32) + img[:, N // 2:] = 1.0 + return img + + +def _gate_html(fig, *, strip_pixels: bool) -> str: + """Return build_standalone_html output, but: + + * gate ``renderFn`` behind a promise the test resolves (``__aplTestGate``), + exposing ``window.__aplReleaseRender()`` to release it; + * expose ``window._aplModel`` for probing; + * when ``strip_pixels`` is set, blank out the pixel value in the baked-in + initial STATE's geom trait so the ONLY pixels the panel can show are the + ones the test delivers via the message handler before render — i.e. the + first real paint truly arrives in the pre-render gap. + """ + from anyplotlib._repr_utils import build_standalone_html + + html = build_standalone_html(fig, resizable=False) + + # Gate render behind a promise the test controls: turn the import handler + # ``async`` and ``await`` the gate before calling renderFn. The message + # listener (registered synchronously below the import) is UNAFFECTED, so the + # test can deliver a frame while render is still parked on the gate. + assert "import(blobUrl).then(mod => {" in html, "template import shape changed" + html = html.replace( + "import(blobUrl).then(mod => {", + "window.__aplTestGate = new Promise(res => " + "{ window.__aplReleaseRender = res; });\n" + "import(blobUrl).then(async mod => {\n" + " await window.__aplTestGate;", + ) + html = html.replace( + "const model = makeModel(STATE);", + "const model = makeModel(STATE);\nwindow._aplModel = model;", + ) + html = html.replace( + "renderFn({ model, el });", + "renderFn({ model, el }); window._aplReady = true;", + ) + + if strip_pixels: + # Blank the pixels in every baked-in geom trait so the panel starts with + # NOTHING to draw; the test then supplies the first frame pre-render. + marker = "const STATE = " + i = html.index(marker) + len(marker) + j = html.index(";\n", i) + state = json.loads(html[i:j]) + for k, v in list(state.items()): + if k.startswith("panel_") and k.endswith("_geom") and isinstance(v, str): + try: + geom = json.loads(v) + except Exception: + continue + for pk in ("image_b64", "overlay_mask_b64", "detail_b64"): + if pk in geom: + geom[pk] = "" + state[k] = json.dumps(geom) + html = html[:i] + json.dumps(state, default=str) + html[j:] + + return html + + +def _open(browser, html): + with tempfile.NamedTemporaryFile( + suffix=".html", mode="w", encoding="utf-8", delete=False + ) as fh: + fh.write(html) + tmp = pathlib.Path(fh.name) + page = browser.new_page() + page.goto(tmp.as_uri()) + return page, tmp + + +def _panel_of(fig): + """The 2-D image panel's id and its geom trait name.""" + p = next(iter(fig._plots_map.values())) + return p._id, f"panel_{p._id}_geom" + + +def _geom_json_and_bytes(fig): + """Push a bright frame and return (slim geom JSON with token, raw uint8 + bytes, real base64 geom JSON) — the wire forms of a single frame the way + ``Figure._push`` / ``_electron._route_change`` produce them.""" + p = next(iter(fig._plots_map.values())) + p.set_data(_bright_frame(), clim=(0.0, 1.0)) + state = p.to_state_dict() + # Real base64 form (what a save_html / non-binary host would send inline). + b64_state = p.resolve_pixel_tokens(dict(state)) + geom_keys = getattr(p, "_GEOM_KEYS", frozenset()) + geom_b64 = {k: b64_state[k] for k in geom_keys if k in b64_state} + raw = geom_b64["image_b64"] + raw_bytes = base64.b64decode(raw) + # Slim geom (LUT kept, pixels → token) — the binary path's JSON companion. + slim = dict(geom_b64) + import zlib + slim["image_b64"] = f"\x00bin:{zlib.adler32(raw_bytes) & 0xFFFFFFFF}" + return json.dumps(slim), raw_bytes, json.dumps(geom_b64) + + +def _mean_brightness(page): + """Mean of the first channel across the plot canvas (0..255). A blank panel + reads ~0 (black background).""" + return page.evaluate( + """() => { + const cvs = Array.from(document.querySelectorAll('canvas')); + for (const c of cvs) { + if (c.width < 8 || c.height < 8) continue; + const ctx = c.getContext('2d'); + if (!ctx) continue; + try { + const d = ctx.getImageData(0, 0, c.width, c.height).data; + let s = 0, n = 0; + for (let i = 0; i < d.length; i += 4) { s += d[i]; n++; } + if (n) return s / n; + } catch (_) { /* tainted */ } + } + return -1; + }""" + ) + + +class TestFirstPaintRace: + def _run(self, browser, *, binary: bool): + # Build the page from a BLANK figure (initial state has no real pixels), + # so the ONLY path to a painted frame is the one delivered pre-render. + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + ax.imshow(np.zeros((N, N), dtype=np.float32), cmap="gray", + vmin=0.0, vmax=1.0, gpu=False) + pid, geom_trait = _panel_of(fig) + + # A separate figure produces the FRAME to deliver (same panel id so the + # geom trait name matches — imshow assigns ids deterministically). + fig2, ax2 = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + ax2.imshow(np.zeros((N, N), dtype=np.float32), cmap="gray", + vmin=0.0, vmax=1.0, gpu=False) + pid2, geom_trait2 = _panel_of(fig2) + assert geom_trait == geom_trait2, "panel ids diverged; test assumption broken" + slim_json, raw_bytes, b64_json = _geom_json_and_bytes(fig2) + + html = _gate_html(fig, strip_pixels=True) + page, tmp = _open(browser, html) + try: + # Wait until the message listener is registered (it is — synchronously + # at module eval) but render is still GATED. + page.wait_for_function( + "() => typeof window.__aplReleaseRender === 'function'", + timeout=15_000) + # Sanity: render has NOT run yet. + assert page.evaluate("() => !window._aplReady"), \ + "render ran before the gate released — ordering not controlled" + + # Deliver the FIRST frame BEFORE render, exactly as the parent bridge + # does: dispatch the same `message` event the iframe host listens for. + if binary: + page.evaluate( + """([trait, geomJson, bytesArr]) => { + const buf = new Uint8Array(bytesArr).buffer; + // 1) the slimmed geom JSON (LUT/flags + token) … + window.dispatchEvent(new MessageEvent('message', { + data: { type: 'awi_state', key: trait, value: geomJson }})); + // 2) … then the raw pixel bytes on the binary channel. + window.dispatchEvent(new MessageEvent('message', { + data: { type: 'awi_state_binary', key: 'image_b64', + header: { geom: trait }, buffer: buf }})); + }""", + [geom_trait, slim_json, list(raw_bytes)], + ) + else: + page.evaluate( + """([trait, geomJson]) => { + window.dispatchEvent(new MessageEvent('message', { + data: { type: 'awi_state', key: trait, value: geomJson }})); + }""", + [geom_trait, b64_json], + ) + + # NOW release render — its INITIAL paint must show the delivered frame. + page.evaluate("() => window.__aplReleaseRender()") + page.wait_for_function("() => window._aplReady === true", timeout=15_000) + page.evaluate( + "() => new Promise(r => requestAnimationFrame(() => " + "requestAnimationFrame(r)))") + + diag = page.evaluate("(pid) => globalThis.__apl_panelDiag(pid)", pid) + bright = _mean_brightness(page) + finally: + page.close() + tmp.unlink(missing_ok=True) + + assert diag is not None, "panel diag missing — render did not build the panel" + assert diag["bytesFp"], ( + f"binary={binary}: panel has NO pixel bytes after first paint — the " + f"frame delivered before render was lost (diag={diag})") + # The bright frame is half-white: mean of the first channel over the whole + # canvas is well above zero. A blank (black) panel reads ~0. + assert bright > 20.0, ( + f"binary={binary}: panel is blank on first paint (mean brightness " + f"{bright:.1f}) — the pre-render frame never painted") + + def test_binary_frame_before_render(self, _pw_browser): + """Binary PLOTBIN path: raw bytes in the side-table before render().""" + self._run(_pw_browser, binary=True) + + def test_base64_frame_before_render(self, _pw_browser): + """Base64 geom-JSON path: pixels in the geom trait before render().""" + self._run(_pw_browser, binary=False) diff --git a/anyplotlib/tests/test_plot2d/test_gpu_image.py b/anyplotlib/tests/test_plot2d/test_gpu_image.py new file mode 100644 index 00000000..8dcf3378 --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_gpu_image.py @@ -0,0 +1,98 @@ +""" +WebGPU 2-D image path — Python-side wiring + Canvas2D-fallback rendering. + +Playwright's bundled Chromium has NO WebGPU, so these tests exercise the FALLBACK +contract: the gpu param maps to gpu_mode, the gpu_active echo starts False, and a +large image still renders correctly via Canvas2D when the GPU is absent. The +GPU-active render path itself is verified in the consuming app on real hardware +(see the SpyDE electron webgpu_image.spec.ts). +""" +from __future__ import annotations + +import numpy as np +import pytest + +import anyplotlib as apl + + +class TestGpuModeParam: + def test_default_is_auto(self): + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((4, 4))) + assert p._state["gpu_mode"] == "auto" + + def test_true_forces_always(self): + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((4, 4)), gpu=True) + assert p._state["gpu_mode"] == "always" + + def test_false_and_off_disable(self): + fig, ax = apl.subplots(1, 1) + assert ax.imshow(np.zeros((4, 4)), gpu=False)._state["gpu_mode"] == "off" + assert ax.imshow(np.zeros((4, 4)), gpu="off")._state["gpu_mode"] == "off" + + +class TestGpuActiveEcho: + def test_starts_false(self): + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((4, 4)), gpu=True) + assert p.gpu_active is False # nothing has reported activation + + def test_set_gpu_active_updates_property_and_state(self): + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((4, 4))) + p._set_gpu_active(True) + assert p.gpu_active is True + assert p._state["gpu_active"] is True + p._set_gpu_active(False) + assert p.gpu_active is False + + def test_gpu_status_event_dispatches_to_plot(self): + # The Figure routes a gpu_status event to plot._set_gpu_active. + fig, ax = apl.subplots(1, 1) + p = ax.imshow(np.zeros((8, 8)), gpu=True) + import json + fig._dispatch_event(json.dumps({ + "panel_id": p._id, "event_type": "gpu_status", "gpu_active": True})) + assert p.gpu_active is True + + +class TestCanvas2dFallbackRender: + def test_gpu_false_always_renders_on_canvas(self, interact_page): + # gpu=False must NEVER take the GPU path, regardless of whether the test + # Chromium has WebGPU — the image renders on the Canvas2D plotCanvas. + w = h = 1200 # > 1 Mpx (would be GPU in auto) + img = np.tile(np.linspace(0, 1, w, dtype=np.float32), (h, 1)) + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(img, cmap="viridis", gpu=False) + page = interact_page(fig) + page.wait_for_timeout(300) + + info = page.evaluate("""() => { + const c = document.querySelector('canvas'); // plotCanvas (image) + const ctx = c.getContext('2d'); + const w = c.width, h = c.height; + const left = Array.from(ctx.getImageData(Math.round(w*0.1), h*0.5|0, 1, 1).data); + const right = Array.from(ctx.getImageData(Math.round(w*0.9), h*0.5|0, 1, 1).data); + return { left, right }; + }""") + # A viridis ramp: non-black and varying L→R → the Canvas2D LUT ran. + assert sum(info["left"][:3]) + sum(info["right"][:3]) > 0 + assert info["left"][:3] != info["right"][:3], "ramp did not render on canvas" + assert p.gpu_active is False # gpu=False never activates GPU + + def test_small_image_stays_on_canvas_in_auto(self, interact_page): + # A sub-threshold image in auto mode stays on Canvas2D (no GPU attempt). + img = np.tile(np.linspace(0, 1, 64, dtype=np.float32), (64, 1)) # 4 Kpx + fig, ax = apl.subplots(1, 1, figsize=(200, 200)) + p = ax.imshow(img, cmap="viridis", gpu="auto") + page = interact_page(fig) + page.wait_for_timeout(300) + rendered = page.evaluate("""() => { + const c = document.querySelector('canvas'); + const ctx = c.getContext('2d'); + const px = Array.from(ctx.getImageData((c.width*0.5)|0, (c.height*0.5)|0, 1, 1).data); + return px.slice(0,3).some(v => v > 0); + }""") + assert rendered, "small image did not render on canvas" + assert p.gpu_active is False diff --git a/anyplotlib/tests/test_plot2d/test_gpu_parity_playwright.py b/anyplotlib/tests/test_plot2d/test_gpu_parity_playwright.py new file mode 100644 index 00000000..adf19992 --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_gpu_parity_playwright.py @@ -0,0 +1,425 @@ +"""GPU-vs-CPU rendering parity for the WebGPU 2-D image path. + +Every test renders the SAME figure twice in the SAME WebGPU-capable browser — +once with ``gpu=True`` (WebGPU texture + shader-LUT path) and once with +``gpu=False`` (the Canvas2D reference path) — applies IDENTICAL interactions +to both, and compares full-composite PNG screenshots. The Canvas2D path is the +fully CI-tested reference, so any GPU divergence (zoom/pan window, letterbox, +colormap, detail-tile stitch) shows up as a pixel diff. + +These run on real WebGPU in headless Chromium (``channel="chromium"`` + +``--enable-unsafe-webgpu`` — see ``_pw_gpu_browser``) and are skipped on +machines with no adapter. ``page.screenshot()`` DOES capture the WebGPU canvas +under the new headless mode, so plain PNG comparison works (the old +"swapchain reads black" caveat applies to Electron offscreen capture, not to +this harness). + +Regression anchor: the shader used to sample the v (row) window MIRRORED about +the texture midline (``1 - mix(v0, v1, q.y)`` instead of interpolating from +``v1`` down to ``v0``), which was invisible at rest and at centred zoom +(v0+v1=1) but flipped the pan direction vertically and detached the image from +the markers/widgets whenever the view was vertically off-centre. The pan tests +below fail hard (>50 % of pixels) on that bug. +""" +from __future__ import annotations + +import json + +import numpy as np +import pytest + +import anyplotlib as apl +from anyplotlib.tests._png_utils import compare_arrays, decode_png + +FIG_W, FIG_H = 400, 300 +PAD_L, PAD_R, PAD_T, PAD_B = 58, 12, 12, 42 +GRID_PAD = 8 +IMG_N = 1200 # > 1 Mpx: over GPU_IMAGE_THRESHOLD even in auto mode + + +# --------------------------------------------------------------------------- +# Scene + helpers +# --------------------------------------------------------------------------- + +def _asym_image(n: int = IMG_N) -> np.ndarray: + """Gradient + distinct corner blocks: any flip/mirror/offset is visible.""" + yy, xx = np.mgrid[0:n, 0:n].astype(np.float32) + img = (xx / n) * 0.3 + (yy / n) * 0.3 + img[(yy < n // 3) & (xx < n // 3)] = 1.0 # bright block TOP-LEFT + img[(yy > 2 * n // 3) & (xx > 2 * n // 3)] = 0.8 # dimmer block BOTTOM-RIGHT + return img + + +def _make_pair(gpu_page, *, tile=False, markers=False, widget=False, + img=None): + """Build gpu=True / gpu=False twins and open both pages. + + Returns (page_gpu, page_cpu, plot_gpu, plot_cpu). + """ + plots, pages = [], [] + for gpu in (True, False): + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + data = _asym_image() if img is None else img + p = ax.imshow(data, cmap="viridis", vmin=0.0, vmax=1.0, + gpu=gpu, tile=tile) + if markers: + p.add_circles( + np.array([[150.0, 200.0], [600.0, 600.0], [900.0, 300.0]], + dtype=np.float32), + name="pts", radius=40, + edgecolors="#ff0000", linewidths=2.5, + ) + if widget: + p.add_widget("rectangle", x=400.0, y=450.0, w=220.0, h=160.0) + plots.append(p) + pages.append(gpu_page(fig, expect_gpu=gpu)) + return pages[0], pages[1], plots[0], plots[1] + + +def _plot_center() -> tuple[int, int]: + cx = GRID_PAD + PAD_L + (FIG_W - PAD_L - PAD_R) // 2 + cy = GRID_PAD + PAD_T + (FIG_H - PAD_T - PAD_B) // 2 + return cx, cy + + +def _snap(page) -> np.ndarray: + return decode_png(page.locator("#widget-root").screenshot()) + + +def _settle(page, ms=150): + page.wait_for_timeout(ms) + page.evaluate( + "() => new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)))" + ) + + +def _assert_parity(name, arr_gpu, arr_cpu, *, tol=10, max_diff_frac=0.005): + """GPU output must match the Canvas2D reference. + + Tolerance covers nearest-sampling rounding differences between the shader + UV math and Canvas2D drawImage at fractional source rects; a coordinate + bug (mirror/offset/flip) moves whole image regions and produces diffs two + orders of magnitude above this. + """ + ok, msg = compare_arrays(arr_gpu, arr_cpu, tol=tol, max_diff_frac=max_diff_frac) + assert ok, f"{name}: GPU render diverged from Canvas2D reference — {msg}" + + +def _assert_gpu_active(page, expect=True): + diag = page.evaluate("() => globalThis.__apl_gpu2d || {}") + assert diag, "no 2-D GPU diagnostic recorded" + for pid, d in diag.items(): + assert d.get("active") is expect, ( + f"panel {pid}: GPU path active={d.get('active')} (expected {expect}) — {d}" + ) + + +def _set_zoom(page, plot, zoom, cx=None, cy=None): + page.evaluate( + "(a) => globalThis.__apl_setZoom(a.pid, a.z, a.cx, a.cy)", + {"pid": plot._id, "z": zoom, "cx": cx, "cy": cy}, + ) + + +def _view_state(page, plot) -> dict: + return json.loads( + page.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", plot._id) + ) + + +# --------------------------------------------------------------------------- +# Image parity: rest / zoom / pan +# --------------------------------------------------------------------------- + +class TestGpuImageParity: + def test_rest_parity(self, gpu_interact_page): + pg, pc, *_ = _make_pair(gpu_interact_page) + _settle(pg); _settle(pc) + _assert_gpu_active(pg) + _assert_gpu_active(pc, expect=False) + _assert_parity("rest", _snap(pg), _snap(pc)) + + def test_zoom_in_centred_parity(self, gpu_interact_page): + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 3.0, 0.5, 0.5) + _settle(page) + _assert_gpu_active(pg) # GPU must NOT fall back on zoom + _assert_parity("zoom-in centred", _snap(pg), _snap(pc)) + + def test_pan_y_parity(self, gpu_interact_page): + # Vertically OFF-CENTRE window — the historical v-mirror bug shows here. + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 3.0, 0.5, 0.22) + _settle(page) + _assert_gpu_active(pg) + _assert_parity("pan-y (off-centre v window)", _snap(pg), _snap(pc)) + + def test_pan_x_parity(self, gpu_interact_page): + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 3.0, 0.2, 0.5) + _settle(page) + _assert_gpu_active(pg) + _assert_parity("pan-x (off-centre u window)", _snap(pg), _snap(pc)) + + def test_pan_corner_parity(self, gpu_interact_page): + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 4.0, 0.85, 0.8) + _settle(page) + _assert_gpu_active(pg) + _assert_parity("pan corner", _snap(pg), _snap(pc)) + + def test_zoom_out_parity(self, gpu_interact_page): + # zoom < 1: the dest-rect (letterbox shrink) branch of the uniform. + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 0.75, 0.5, 0.5) + _settle(page) + _assert_gpu_active(pg) + _assert_parity("zoom-out", _snap(pg), _snap(pc)) + + +class TestGpuMousePanParity: + """End-to-end through the real wheel/drag handlers (not __apl_setZoom).""" + + def test_wheel_zoom_then_drag_pan(self, gpu_interact_page): + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page) + cx, cy = _plot_center() + + for page in (pg, pc): + page.mouse.move(cx, cy) + for _ in range(6): + page.mouse.wheel(0, -120) + _settle(page) + _assert_parity("mouse wheel zoom", _snap(pg), _snap(pc)) + + # Drag DOWN: image content must move down (view up) on BOTH paths. + for page in (pg, pc): + page.mouse.move(cx, cy) + page.mouse.down() + page.mouse.move(cx, cy + 60, steps=8) + page.mouse.up() + _settle(page) + st_g = _view_state(pg, p_gpu) + st_c = _view_state(pc, p_cpu) + assert st_g["zoom"] == st_c["zoom"] + assert st_g["center_x"] == st_c["center_x"] + assert st_g["center_y"] == st_c["center_y"] + _assert_gpu_active(pg) + _assert_parity("mouse pan down", _snap(pg), _snap(pc)) + + # Then drag RIGHT while still vertically off-centre. + for page in (pg, pc): + page.mouse.move(cx, cy) + page.mouse.down() + page.mouse.move(cx + 60, cy, steps=8) + page.mouse.up() + _settle(page) + _assert_gpu_active(pg) + _assert_parity("mouse pan right", _snap(pg), _snap(pc)) + + def test_pan_direction_moves_content_with_cursor(self, gpu_interact_page): + """Row-profile check independent of the CPU reference: dragging DOWN + must move image rows DOWN on screen (the same rows appear lower).""" + pg, _, p_gpu, _ = _make_pair(gpu_interact_page) + cx, cy = _plot_center() + _set_zoom(pg, p_gpu, 3.0, 0.5, 0.5) + _settle(pg) + before = _snap(pg).astype(np.int32) + + pg.mouse.move(cx, cy) + pg.mouse.down() + pg.mouse.move(cx, cy + 40, steps=8) + pg.mouse.up() + _settle(pg) + after = _snap(pg).astype(np.int32) + + # The pan handler moved center_y UP the image (drag down = view up), so + # the "after" frame equals "before" shifted DOWN by ~40 px. Verify by + # comparing a shifted slice — mean row-difference must be far smaller + # for the correct shift than for the opposite one. + h = before.shape[0] + shift = 40 + band = slice(80, h - 80) + correct = np.abs(after[shift:, :][band] - before[:-shift, :][band]).mean() + inverted = np.abs(after[:-shift, :][band] - before[shift:, :][band]).mean() + assert correct < inverted, ( + f"pan-y direction inverted on GPU path: shifted-down diff {correct:.2f} " + f">= shifted-up diff {inverted:.2f}" + ) + + +# --------------------------------------------------------------------------- +# Markers + widgets registration with the GPU image +# --------------------------------------------------------------------------- + +def _feature_marker_image(n: int = IMG_N, fx: int = 780, fy: int = 420): + """Black field with one bright square centred at (fx, fy) image px.""" + img = np.zeros((n, n), dtype=np.float32) + img[fy - 12:fy + 12, fx - 12:fx + 12] = 1.0 + return img, fx, fy + + +def _centroid(mask: np.ndarray): + ys, xs = np.nonzero(mask) + if len(xs) == 0: + return None + return float(xs.mean()), float(ys.mean()) + + +def _feature_and_ring_centroids(arr: np.ndarray): + """Locate the viridis-bright feature block and the red marker ring.""" + r = arr[..., 0].astype(np.int32) + g = arr[..., 1].astype(np.int32) + b = arr[..., 2].astype(np.int32) + # value 1.0 in viridis ≈ (253, 231, 37) + feature = (r > 200) & (g > 200) & (b < 120) + ring = (r > 180) & (g < 90) & (b < 90) + return _centroid(feature), _centroid(ring) + + +class TestGpuMarkerRegistration: + def _scene(self, gpu): + img, fx, fy = _feature_marker_image() + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + p = ax.imshow(img, cmap="viridis", vmin=0.0, vmax=1.0, gpu=gpu) + p.add_circles(np.array([[float(fx), float(fy)]], dtype=np.float32), + name="mark", radius=60, + edgecolors="#ff0000", linewidths=3) + return fig, p, fx, fy + + @pytest.mark.parametrize("view", [ + (1.0, None, None), # rest + (3.0, 0.65, 0.35), # zoom + off-centre both axes + (2.0, 0.5, 0.42), # vertically off-centre (the mirror case); + # feature y=420 stays inside rows [204, 804] + ]) + def test_marker_sits_on_feature(self, gpu_interact_page, view): + """The red circle must stay centred on the bright block it marks, on + BOTH paths, at any zoom/pan — ties the marker transform to the image + transform directly (no reference page needed).""" + zoom, cx, cy = view + for gpu in (True, False): + fig, p, _, _ = self._scene(gpu) + page = gpu_interact_page(fig, expect_gpu=gpu) + if zoom != 1.0: + _set_zoom(page, p, zoom, cx, cy) + _settle(page) + if gpu: + _assert_gpu_active(page) + feat, ring = _feature_and_ring_centroids(_snap(page)) + path = "GPU" if gpu else "CPU" + assert feat is not None, f"{path}: feature block not visible at {view}" + assert ring is not None, f"{path}: marker ring not visible at {view}" + dist = ((feat[0] - ring[0]) ** 2 + (feat[1] - ring[1]) ** 2) ** 0.5 + assert dist < 4.0, ( + f"{path}: marker detached from image feature at zoom={zoom} " + f"center=({cx},{cy}): feature {feat} vs ring {ring} — {dist:.1f} px apart" + ) + + def test_marker_composite_parity_zoom_pan(self, gpu_interact_page): + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page, markers=True) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 2.5, 0.6, 0.3) + _settle(page) + _assert_gpu_active(pg) + _assert_parity("markers over zoom/pan", _snap(pg), _snap(pc)) + + +class TestGpuWidgetParity: + def _rect_center_screen(self, page, plot): + st = _view_state(page, plot) + w = next(x for x in st["overlay_widgets"] if x["type"] == "rectangle") + wcx, wcy = w["x"] + w["w"] / 2, w["y"] + w["h"] / 2 + return page.evaluate( + """(a) => { + const ov = Array.from(document.querySelectorAll('canvas')) + .find(c => c.style && c.style.zIndex === '5'); + const r = ov.getBoundingClientRect(); + const q = globalThis.__apl_imgToCanvas + ? globalThis.__apl_imgToCanvas(a.pid, a.ix, a.iy) : null; + return q ? [r.left + q[0], r.top + q[1]] : null; + }""", + {"pid": plot._id, "ix": wcx, "iy": wcy}, + ) + + def test_widget_drag_same_result_and_pixels(self, gpu_interact_page): + pg, pc, p_gpu, p_cpu = _make_pair(gpu_interact_page, widget=True) + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + _set_zoom(page, plot, 2.0, 0.45, 0.35) + _settle(page) + + # Drag each rectangle by the same screen delta, grabbing its centre. + for page, plot in ((pg, p_gpu), (pc, p_cpu)): + pos = self._rect_center_screen(page, plot) + assert pos is not None, "__apl_imgToCanvas hook missing" + sx, sy = pos + page.mouse.move(sx, sy) + page.mouse.down() + page.mouse.move(sx + 30, sy + 25, steps=10) + page.mouse.up() + _settle(page) + + st_g = _view_state(pg, p_gpu) + st_c = _view_state(pc, p_cpu) + w_g = next(x for x in st_g["overlay_widgets"] if x["type"] == "rectangle") + w_c = next(x for x in st_c["overlay_widgets"] if x["type"] == "rectangle") + for k in ("x", "y", "w", "h"): + assert abs(w_g[k] - w_c[k]) < 1e-6, ( + f"widget {k} diverged: gpu={w_g[k]} cpu={w_c[k]}" + ) + _assert_gpu_active(pg) + _assert_parity("widget drag composite", _snap(pg), _snap(pc)) + + +# --------------------------------------------------------------------------- +# Detail tile (tile mode) on the GPU detail pass +# --------------------------------------------------------------------------- + +class TestGpuDetailTileParity: + def test_detail_tile_zoom_parity(self, gpu_interact_page): + """Zoomed-in view fully covered by a pre-baked detail tile: the GPU + detail pass (uniformBuf2) must place the tile rows exactly like the + Canvas2D detail blit. Tile content differs from the base so sampling + the tile (not the base) is proven, and any v-mirror in the detail + pass moves the pattern.""" + n = IMG_N + base = _asym_image(n) + # Tile covering [480,960]x[180,660] with content DIFFERENT from base. + # Content must be vertically APERIODIC: a monotonic vertical gradient + # (any v-mirror reverses it) + one unique bright band near the tile top + # (periodic stripes can self-align under a mirror and hide the bug). + ty0, ty1, tx0, tx1 = 180, 660, 480, 960 + th = ty1 - ty0 + tile = np.tile( + (0.15 + 0.55 * np.arange(th, dtype=np.float32) / th)[:, None], + (1, tx1 - tx0)) + tile[40:80, :] = 1.0 # unique band, near the TOP of the tile only + + pages, plots = [], [] + for gpu in (True, False): + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + p = ax.imshow(base, cmap="viridis", vmin=0.0, vmax=1.0, + gpu=gpu, tile=True) + p.set_detail(tile, tx0, tx1, ty0, ty1) + # Bake a zoom/centre whose visible window sits INSIDE the tile + # region but vertically OFF its centre (the mirror-sensitive case): + # zoom=4 → 300 px window; centre (0.6, 0.25*?) … + plots.append(p) + page = gpu_interact_page(fig, expect_gpu=gpu) + _set_zoom(page, p, 4.0, 0.6, 0.3) # window x [570,870] y [210,510] + _settle(page, 300) # let the detail blend ramp finish + pages.append(page) + + pg, pc = pages + _assert_gpu_active(pg) + arr_gpu, arr_cpu = _snap(pg), _snap(pc) + # Sanity: the tile's unique bright band (near the tile TOP, inside the + # visible window) must be visible — i.e. the tile was sampled, not the + # base gradient. + bright = (arr_cpu[..., 0] > 200) & (arr_cpu[..., 1] > 200) + assert bright.mean() > 0.03, "detail tile content not visible on reference" + _assert_parity("detail tile zoom", arr_gpu, arr_cpu) diff --git a/anyplotlib/tests/test_plot2d/test_imshow.py b/anyplotlib/tests/test_plot2d/test_imshow.py index 91c575e3..711a2a7f 100644 --- a/anyplotlib/tests/test_plot2d/test_imshow.py +++ b/anyplotlib/tests/test_plot2d/test_imshow.py @@ -45,10 +45,15 @@ def _img(n=32, **kwargs) -> Plot2D: def _decoded(v: Plot2D) -> np.ndarray: - """Return the stored uint8 image as a (H, W) array.""" - raw = base64.b64decode(v._state["image_b64"]) + """Return the stored uint8 image as a (H, W) array. + + Resolves any raw-bytes change-token (used when binary transport is active) + back to real base64 via ``resolve_pixel_tokens`` before decoding, so the + helper works regardless of transport mode.""" + st = v.resolve_pixel_tokens(v.to_state_dict()) + raw = base64.b64decode(st["image_b64"]) return np.frombuffer(raw, dtype=np.uint8).reshape( - v._state["image_height"], v._state["image_width"] + st["image_height"], st["image_width"] ) @@ -167,6 +172,63 @@ def test_set_clim_still_works_after_construction(self): assert v._state["display_max"] == pytest.approx(14.0) +class TestImshowClimQuantization: + """set_data(clim=) / set_clim quantise the uint8 codes over the clim (not the + raw min/max), so a single hot pixel / zero beam can't crush the signal into a + handful of codes. raw_min/raw_max then equal the clim so the JS LUT reconstructs + each code's value correctly.""" + + @staticmethod + def _hot_frame(): + rng = np.random.RandomState(0) + f = rng.gamma(2.0, 400.0, (48, 48)).astype(np.float32) # faint signal ~0-4000 + f[5, 5] = 60000.0 # hot pixel / zero beam + return f + + def test_set_data_clim_quantises_over_clim(self): + fig, ax = apl.subplots() + v = ax.imshow(np.zeros((4, 4))) + frame = self._hot_frame() + clim = (88.0, 2638.0) + v.set_data(frame, clim=clim) + # raw_min/max are the quantisation endpoints (== clim), so the LUT maps back + # correctly; display window is the same range. + assert v._state["raw_min"] == pytest.approx(clim[0]) + assert v._state["raw_max"] == pytest.approx(clim[1]) + assert v._state["display_min"] == pytest.approx(clim[0]) + assert v._state["display_max"] == pytest.approx(clim[1]) + # The signal now spans nearly all 256 codes (vs ~12 quantising over raw + # min/max), and the hot pixel saturates to 255 instead of stealing the range. + u8 = v._raw_u8 + sig_codes = len(np.unique(u8[frame <= clim[1]])) + assert sig_codes > 128, f"signal only got {sig_codes} codes — hot pixel crushed it" + assert u8[5, 5] == 255, "hot pixel above clim must saturate to 255" + + def test_set_data_without_clim_unchanged(self): + """No clim → quantise over raw min/max (backward-compatible).""" + fig, ax = apl.subplots() + v = ax.imshow(np.zeros((4, 4))) + frame = self._hot_frame() + v.set_data(frame) + assert v._state["raw_min"] == pytest.approx(float(frame.min())) + assert v._state["raw_max"] == pytest.approx(float(frame.max())) + + def test_set_clim_requantises_from_raw(self): + """set_clim re-quantises from the cached raw frame so a NEW range is honoured + at full fidelity (not capped by the previous quantisation band).""" + fig, ax = apl.subplots() + v = ax.imshow(np.zeros((4, 4))) + frame = self._hot_frame() + v.set_data(frame, clim=(88.0, 2638.0)) + # Widen the window well past the previous band toward the beam. + v.set_clim(0.0, 30000.0) + assert v._state["raw_min"] == pytest.approx(0.0) + assert v._state["raw_max"] == pytest.approx(30000.0) + # The hot pixel (60000, still above the new max) saturates; a mid value is + # now representable (it wasn't at the old narrow band). + assert v._raw_u8[5, 5] == 255 + + # =========================================================================== # Origin # =========================================================================== diff --git a/anyplotlib/tests/test_plot2d/test_plot2d_api.py b/anyplotlib/tests/test_plot2d/test_plot2d_api.py index 5276cf76..6f1eab80 100644 --- a/anyplotlib/tests/test_plot2d/test_plot2d_api.py +++ b/anyplotlib/tests/test_plot2d/test_plot2d_api.py @@ -199,6 +199,21 @@ def test_set_extent_updates_scale(self): assert p._state["scale_x"] == pytest.approx(1.0) assert p._state["scale_y"] == pytest.approx(2.0) + def test_set_extent_sets_has_axes(self): + # A plot created without axis args starts has_axes=False; set_extent must + # flip it True so the front-end draws physical tick gutters + the scale + # bar (the tiled/GPU path calibrates only through set_extent). + p = _make_plot2d((32, 32)) + assert p._state["has_axes"] is False + p.set_extent(np.linspace(0.0, 10.0, 32), np.linspace(0.0, 20.0, 32)) + assert p._state["has_axes"] is True + + def test_set_extent_updates_units(self): + p = _make_plot2d((32, 32)) + p.set_extent(np.linspace(0.0, 10.0, 32), + np.linspace(0.0, 20.0, 32), units="Å⁻¹") + assert p._state["units"] == "Å⁻¹" + class TestPlot2DColorbar: diff --git a/anyplotlib/tests/test_plot2d/test_rapid_frame_update.py b/anyplotlib/tests/test_plot2d/test_rapid_frame_update.py new file mode 100644 index 00000000..2aa9306b --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_rapid_frame_update.py @@ -0,0 +1,160 @@ +"""Rapid successive frame updates must always END on the LAST-arrived frame. + +Regression guard for the movie-playback / fast-navigator-scrub freeze class of +bug: a live consumer pushes a NEW image frame on every tick (20 fps playback, +or a fast drag), and the display must converge on the frame that arrived LAST. +Frame skipping / newest-wins coalescing is fine — dropping intermediate frames +is expected — but the render caches (GPU texture ``bytesKey`` / Canvas2D +``blitCache.bytesKey``) must NOT latch a stale frame and skip the final +re-upload. A too-weak content key (e.g. a 4-byte sampled fingerprint that +collides across frames) would freeze the display on an early frame even though +byte-distinct frames kept arriving — exactly the reported freeze. + +The scene is a moving BRIGHT VERTICAL BAND whose x-position encodes the frame +index (like ``load_test_data_movie``'s per-frame index band), so the rendered +band column tells us WHICH frame is on screen. We push all N frames rapid-fire +via ``requestAnimationFrame`` (no per-frame settle — the coalescing regime) and +then assert the on-screen band sits at the LAST frame's column. + +Runs on real WebGPU (``gpu=True``) and on the Canvas2D reference (``gpu=False``) +in the same browser, and covers both the PLAIN large-image path and the TILE +path (>1024 edge → overview + detail), since both have their own dedup key. +""" +from __future__ import annotations + +import json + +import numpy as np +import pytest + +import anyplotlib as apl +from anyplotlib.tests._png_utils import decode_png + +FIG_W, FIG_H = 400, 300 +N_FRAMES = 8 + + +def _band_frame(n: int, frame_idx: int, n_frames: int) -> np.ndarray: + """A dim gradient with a BRIGHT vertical band whose column encodes the frame + index — same idea as the SpyDE synthetic movie's per-frame index band.""" + yy, xx = np.mgrid[0:n, 0:n].astype(np.float32) + img = (xx / n) * 0.15 + (yy / n) * 0.15 + bw = max(2, n // 16) + x0 = int((frame_idx + 1) * n / (n_frames + 2)) + img[:, x0:x0 + bw] = 1.0 + return img + + +def _band_col_frac(frame_idx: int, n_frames: int) -> float: + """The band's LEFT-edge x-position as a fraction of image width (matches + ``_band_frame``'s ``x0``).""" + return (frame_idx + 1) / (n_frames + 2) + + +def _frames_state(n: int, *, gpu: bool, tile) -> tuple[str, list[str]]: + """Build a plot and return (panel_id, [state_json per frame]). + + Each frame's state JSON is captured from ``to_state_dict()`` after a + ``set_data`` of that frame (standalone host → real base64 pixels), so the + browser can push them one-by-one with no Python kernel in the loop. + """ + fig, ax = apl.subplots(1, 1, figsize=(FIG_W, FIG_H)) + p = ax.imshow(_band_frame(n, 0, N_FRAMES), cmap="viridis", + vmin=0.0, vmax=1.0, gpu=gpu, tile=tile) + states = [] + for i in range(N_FRAMES): + p.set_data(_band_frame(n, i, N_FRAMES), clim=(0.0, 1.0)) + states.append(json.dumps(p.resolve_pixel_tokens(p.to_state_dict()))) + return fig, p, states + + +def _push_frames_rapid(page, panel_id: str, states: list[str]) -> None: + """Push every state JSON into ``panel__json`` back-to-back, one per + ``requestAnimationFrame`` (the rapid-update / coalescing regime), then let + the final frame settle with two more rAFs.""" + page.evaluate( + """([pid, states]) => new Promise((resolve) => { + const key = 'panel_' + pid + '_json'; + let i = 0; + function step() { + if (i >= states.length) { + // let the LAST frame commit (device queue + compositor) + requestAnimationFrame(() => + requestAnimationFrame(() => + requestAnimationFrame(resolve))); + return; + } + window._aplModel.set(key, states[i]); + i++; + requestAnimationFrame(step); + } + requestAnimationFrame(step); + })""", + [panel_id, states], + ) + + +def _band_center_frac(page) -> float: + """Screenshot ``#widget-root``, find the brightest column across the plot + area, and return its x-position as a fraction of the plot width.""" + arr = decode_png(page.locator("#widget-root").screenshot()) + h, w = arr.shape[:2] + # Green channel peaks for viridis' bright end (the band value 1.0 → yellow). + col_bright = arr[:, :, 1].astype(np.float32).mean(axis=0) + # Ignore the axis gutter on the left/right by trimming a margin. + m = int(w * 0.16) + inner = col_bright[m:w - m] + peak = int(np.argmax(inner)) + m + return peak / float(w) + + +def _run(page_open, *, gpu: bool, tile, n: int): + # Reference: the LAST frame pushed ALONE and allowed to settle — this is the + # ground truth for "the last frame is on screen". Calibration-free (same + # padding/letterbox as the rapid-push page), so it isolates the freeze. + fig_ref, plot_ref, states_ref = _frames_state(n, gpu=gpu, tile=tile) + page_ref = page_open(fig_ref, expect_gpu=gpu) + _push_frames_rapid(page_ref, plot_ref._id, states_ref[-1:]) # last frame only + ref_frac = _band_center_frac(page_ref) + + # An EARLY frame alone — the position a stale-frame freeze would show. + fig_e, plot_e, states_e = _frames_state(n, gpu=gpu, tile=tile) + page_e = page_open(fig_e, expect_gpu=gpu) + _push_frames_rapid(page_e, plot_e._id, states_e[:1]) # first frame only + early_frac = _band_center_frac(page_e) + + # The gap between the first and last band positions must be resolvable — the + # scene is built that way; guard the guard. + assert abs(ref_frac - early_frac) > 0.15, ( + f"gpu={gpu} tile={tile}: first ({early_frac:.3f}) and last " + f"({ref_frac:.3f}) band positions too close — bad scene") + + # The real test: push ALL frames rapid-fire (coalescing regime) and confirm + # the band lands at the LAST frame's column, NOT a stale earlier one. + fig, plot, states = _frames_state(n, gpu=gpu, tile=tile) + page = page_open(fig, expect_gpu=gpu) + _push_frames_rapid(page, plot._id, states) + got = _band_center_frac(page) + + d_last = abs(got - ref_frac) + d_early = abs(got - early_frac) + assert d_last < d_early and d_last < 0.05, ( + f"gpu={gpu} tile={tile}: after rapid push the band is at {got:.3f} " + f"(last={ref_frac:.3f}, first={early_frac:.3f}) — display froze on a " + f"stale frame instead of converging on the LAST frame") + + +class TestRapidFrameUpdate: + """The last pushed frame must always be the one rendered.""" + + def test_plain_gpu(self, gpu_interact_page): + _run(gpu_interact_page, gpu=True, tile=False, n=1200) + + def test_plain_cpu(self, gpu_interact_page): + _run(gpu_interact_page, gpu=False, tile=False, n=1200) + + def test_tile_gpu(self, gpu_interact_page): + _run(gpu_interact_page, gpu=True, tile=True, n=1400) + + def test_tile_cpu(self, gpu_interact_page): + _run(gpu_interact_page, gpu=False, tile=True, n=1400) diff --git a/anyplotlib/tests/test_plot2d/test_tile_backend.py b/anyplotlib/tests/test_plot2d/test_tile_backend.py new file mode 100644 index 00000000..e8e055e6 --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_tile_backend.py @@ -0,0 +1,99 @@ +"""NumpyTileBackend + the TileBackend protocol — the pluggable sampling seam.""" +import numpy as np +import pytest + +from anyplotlib.plot2d._tile_backend import ( + NumpyTileBackend, TileBackend, as_tile_backend, +) + + +class TestNumpyBackendGeometry: + def test_reports_shape_dtype_origin_extent(self): + a = np.zeros((600, 800), np.uint16) + b = NumpyTileBackend(a, extent=(0.0, 8.0, 0.0, 6.0), origin="lower") + assert b.full_shape == (600, 800) + assert b.dtype == np.uint16 + assert b.origin == "lower" + assert b.extent() == (0.0, 8.0, 0.0, 6.0) + + def test_default_extent_is_none(self): + assert NumpyTileBackend(np.zeros((4, 4))).extent() is None + + def test_rejects_non_2d(self): + with pytest.raises(ValueError): + NumpyTileBackend(np.zeros((4, 4, 4))) + + def test_satisfies_protocol(self): + assert isinstance(NumpyTileBackend(np.zeros((4, 4))), TileBackend) + + +class TestSampleMean: + def test_full_region_mean_matches_block_mean(self): + a = np.random.RandomState(0).randint(0, 4000, (64, 64)).astype(np.uint16) + b = NumpyTileBackend(a) + out = b.sample(0, 64, 0, 64, 16, 16, "mean") + assert out.shape == (16, 16) + ref = a.astype(np.float32).reshape(16, 4, 16, 4).mean(axis=(1, 3)) + np.testing.assert_allclose(out, ref, rtol=1e-4) + + def test_mean_preserves_hot_pixel_energy(self): + a = np.zeros((16, 16), np.float32) + a[5, 5] = 1600.0 + out = NumpyTileBackend(a).sample(0, 16, 0, 16, 4, 4, "mean") + assert out.max() == pytest.approx(1600.0 / 16) # spread, not dropped + + def test_subregion(self): + a = np.arange(64 * 64, dtype=np.float32).reshape(64, 64) + out = NumpyTileBackend(a).sample(16, 48, 16, 48, 8, 8, "mean") + assert out.shape == (8, 8) + ref = a[16:48, 16:48].reshape(8, 4, 8, 4).mean(axis=(1, 3)) + np.testing.assert_allclose(out, ref, rtol=1e-4) + + +class TestSampleSubsampleMax: + def test_subsample_drops_between_grid(self): + a = np.zeros((16, 16), np.float32) + a[5, 5] = 100.0 # off the /4 grid + out = NumpyTileBackend(a).sample(0, 16, 0, 16, 4, 4, "subsample") + assert out.max() == 0.0 # dropped by nearest sampling + + def test_max_keeps_the_peak(self): + a = np.zeros((16, 16), np.float32) + a[5, 5] = 100.0 + out = NumpyTileBackend(a).sample(0, 16, 0, 16, 4, 4, "max") + assert out.max() == 100.0 # block max keeps the peak + + +class TestSampleShapesAndClamp: + def test_out_shape_is_exact(self): + a = np.zeros((100, 137), np.float32) + for (ow, oh) in [(50, 50), (33, 41), (200, 10)]: + out = NumpyTileBackend(a).sample(0, 137, 0, 100, ow, oh, "mean") + assert out.shape == (oh, ow) + + def test_upsample_when_out_bigger_than_region(self): + a = np.arange(16, dtype=np.float32).reshape(4, 4) + out = NumpyTileBackend(a).sample(0, 4, 0, 4, 8, 8, "mean") + assert out.shape == (8, 8) # nearest upsample + + def test_region_is_clamped(self): + a = np.ones((32, 32), np.float32) + out = NumpyTileBackend(a).sample(-10, 999, -10, 999, 16, 16, "mean") + assert out.shape == (16, 16) # clamped to [0, 32], no crash + + +class TestSetArray: + def test_set_array_swaps_source(self): + b = NumpyTileBackend(np.zeros((32, 32), np.float32)) + b.set_array(np.full((32, 32), 7.0, np.float32)) + assert b.sample(0, 32, 0, 32, 4, 4, "mean").max() == pytest.approx(7.0) + + +class TestAsTileBackend: + def test_wraps_ndarray(self): + b = as_tile_backend(np.zeros((8, 8))) + assert isinstance(b, NumpyTileBackend) + + def test_passes_backend_through(self): + inner = NumpyTileBackend(np.zeros((8, 8))) + assert as_tile_backend(inner) is inner diff --git a/anyplotlib/tests/test_plot2d/test_tiled_imshow.py b/anyplotlib/tests/test_plot2d/test_tiled_imshow.py new file mode 100644 index 00000000..6f720b8f --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_tiled_imshow.py @@ -0,0 +1,386 @@ +"""Tiled large-image imshow — end-to-end. + +Python side (no browser): tile="auto" builds an overview base + logical dims, and +the INTERNAL view_changed handler samples a detail tile from the backend on zoom. +Browser side (Canvas2D — WebGPU absent headless): the overview base renders at zoom +1, and a detail tile (what the internal loop would send) renders crisp when zoomed +into its region. (The JS→Python→JS round trip needs a real kernel, so the browser +test injects the tile into state directly; the Python test above covers the loop.) +""" +import json +import base64 +import numpy as np +import anyplotlib as apl + +from anyplotlib.tests._png_utils import compare_arrays_exact + + +class TestTiledConstruction: + def test_auto_engages_over_threshold(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((2048, 2048), np.float32)) + st = p._state + assert st["tile_enabled"] is True + assert st["image_width"] == 2048 and st["image_height"] == 2048 # logical + assert 0 < st["base_width"] <= 1024 # overview + assert st["base_width"] < st["image_width"] + + def test_auto_off_under_threshold(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((512, 512), np.float32)) + assert p._state["tile_enabled"] is False + assert p._state["base_width"] == 0 + + def test_tile_false_forces_off(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((4096, 4096), np.float32), tile=False) + assert p._state["tile_enabled"] is False + + def test_internal_loop_sets_tile_on_zoom(self): + from anyplotlib.callbacks import Event + big = np.random.RandomState(0).rand(4096, 4096).astype(np.float32) + p = apl.subplots(1, 1)[1].imshow(big, vmin=0, vmax=1) + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + x0, x1, y0, y1 = p._state["detail_region"] + assert 1900 <= (x1 - x0) <= 2100 # 1024 visible × 2.0 over-fetch + assert p._state["detail_width"] <= 1000 # capped to panel px + # zoom out clears + p.callbacks.fire(Event("view_changed", zoom=1.0, center_x=0.5, center_y=0.5)) + assert p._state["detail_region"] == [] + + def test_update_tile_source_keeps_view_refreshes_pixels(self): + # Live data: the source changes but zoom/subselection persist, and the + # overview + current detail tile refresh from the new frame. + from anyplotlib.callbacks import Event + a = np.zeros((4096, 4096), np.float32) + b = np.full((4096, 4096), 0.9, np.float32) + b[1408:2688, 1408:2688] = 0.5 + p = apl.subplots(1, 1)[1].imshow(a, cmap="gray", vmin=0, vmax=1) + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + reg = list(p._state["detail_region"]) + zoom = p._state["zoom"] + assert len(reg) == 4 + p.update_tile_source(b) # swap data + assert list(p._state["detail_region"]) == reg # region persisted + assert p._state["zoom"] == zoom # zoom persisted + x0, x1, y0, y1 = reg + crop = p._tile_backend.sample(x0, x1, y0, y1, 100, 100, "mean") + # backend swapped a→b: the region (2× over-fetch → [1024:3072]) mixes b's 0.9 + # background with its 0.5 centre patch, so the mean is well above a's all-zero. + assert 0.5 <= float(crop.mean()) <= 0.9 + # no-arg form (backend already mutated its own source) + p._tile_backend.set_array(a) + p.update_tile_source() + assert p._state["base_width"] > 0 + + def test_update_tile_source_noop_when_not_tiled(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((64, 64), np.float32)) + p.update_tile_source(np.ones((64, 64), np.float32)) # no crash, no-op + + def test_custom_backend_is_used(self): + from anyplotlib.callbacks import Event + from anyplotlib.plot2d._tile_backend import NumpyTileBackend + + calls = [] + + class SpyBackend(NumpyTileBackend): + def sample(self, x0, x1, y0, y1, out_w, out_h, method="mean"): + calls.append((x0, x1, y0, y1, out_w, out_h, method)) + return super().sample(x0, x1, y0, y1, out_w, out_h, method) + + b = SpyBackend(np.random.RandomState(0).rand(4096, 4096).astype(np.float32)) + p = apl.subplots(1, 1)[1].imshow(b, tile=True, tile_backend=b, vmin=0, vmax=1) + assert calls, "backend.sample not called for the overview" + overview_call = calls[0] + assert overview_call[6] == "mean", ( + f"overview must use mean integration by default, got {overview_call[6]!r}") + calls.clear() + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + assert calls, "backend.sample not called on zoom (the seam works)" + + +class TestOverviewMethod: + """overview_method selects how the base overview texture is integrated. + + Default is "mean" (consistent with detail tiles so sparse images don't + shift on zoom-in). "subsample" restores the old fast nearest-neighbour + path for consumers that can't afford a full-frame area-mean.""" + + def _spy_backend(self): + from anyplotlib.plot2d._tile_backend import NumpyTileBackend + calls = [] + + class SpyBackend(NumpyTileBackend): + def sample(self, x0, x1, y0, y1, out_w, out_h, method="mean"): + calls.append(method) + return super().sample(x0, x1, y0, y1, out_w, out_h, method) + + b = SpyBackend(np.random.RandomState(0).rand(2048, 2048).astype(np.float32)) + return b, calls + + def test_default_overview_method_is_mean(self): + b, calls = self._spy_backend() + apl.subplots(1, 1)[1].imshow(b, tile=True, tile_backend=b, vmin=0, vmax=1) + assert calls, "backend.sample never called" + assert calls[0] == "mean", ( + f"default overview_method must be 'mean', got {calls[0]!r}") + + def test_overview_method_subsample_opts_out(self): + b, calls = self._spy_backend() + apl.subplots(1, 1)[1].imshow(b, tile=True, tile_backend=b, vmin=0, vmax=1, + overview_method="subsample") + assert calls, "backend.sample never called" + assert calls[0] == "subsample", ( + f"overview_method='subsample' must reach backend, got {calls[0]!r}") + + def test_enable_tile_overview_method_propagates(self): + b, calls = self._spy_backend() + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10)), vmin=0, vmax=1) + calls.clear() + p.enable_tile(b, overview_method="subsample") + assert calls, "backend.sample never called by enable_tile" + assert calls[0] == "subsample", ( + f"enable_tile overview_method must reach backend, got {calls[0]!r}") + + def test_enable_tile_none_preserves_existing_overview_method(self): + """enable_tile(overview_method=None) must not reset the stored method.""" + b, calls = self._spy_backend() + p = apl.subplots(1, 1)[1].imshow(b, tile=True, tile_backend=b, + overview_method="subsample", + vmin=0, vmax=1) + calls.clear() + # Re-enable without passing overview_method — should still use subsample. + p.enable_tile(b) + assert calls, "backend.sample never called on re-enable" + assert calls[0] == "subsample", ( + "enable_tile() without overview_method must preserve the existing " + f"setting; got {calls[0]!r}") + + +class TestSetDataRespectsTiling: + """set_data on a plot ALREADY in tile mode must route through the tile pipeline, + not clobber it. Regression for: a live consumer (movie navigator) that calls + set_data per frame saw the image shrink to the overview size + lose its zoom + detail (flash / snap-back), because set_data wrote a full-res base frame and + reset image_width while base_width still named the old overview.""" + + def _tiled(self, val=0.2): + big = np.full((4096, 4096), val, np.float32) + return apl.subplots(1, 1)[1].imshow(big, cmap="gray", vmin=0, vmax=1) + + def test_set_data_keeps_logical_size_and_overview(self): + p = self._tiled(0.2) + assert p._state["tile_enabled"] is True + base_w0 = p._state["base_width"] + assert 0 < base_w0 < p._state["image_width"] == 4096 + # A new same-size frame via set_data must NOT reset image_width to the frame + # size with a stale base_width — that mismatch is the "shrinks to 1k" bug. + p.set_data(np.full((4096, 4096), 0.7, np.float32), clim=(0, 1)) + assert p._state["image_width"] == 4096 and p._state["image_height"] == 4096 + assert 0 < p._state["base_width"] < 4096 # still an overview, not full-res + assert p._state["tile_enabled"] is True + # overview pixels reflect the NEW frame (0.7 over [0,1] → mid-bright), proving + # the tile source swapped rather than a stale base persisting. + ov = p._tile_backend.sample(0, 4096, 0, 4096, 64, 64, "mean") + assert 0.6 < float(ov.mean()) < 0.8 + + def test_set_data_preserves_zoom_and_detail(self): + from anyplotlib.callbacks import Event + p = self._tiled(0.2) + # The frontend writes zoom/center into state on a real zoom; emulate that so we + # can assert set_data preserves it (headless callbacks.fire only sets the tile). + p._state["zoom"], p._state["center_x"], p._state["center_y"] = 4.0, 0.5, 0.5 + p.callbacks.fire(Event("view_changed", zoom=4.0, center_x=0.5, center_y=0.5, + display_width=1000, display_height=1000)) + reg = list(p._state["detail_region"]) + zoom = p._state["zoom"] + assert len(reg) == 4 and zoom == 4.0 + # set_data must keep the zoom AND refresh the SAME detail region (live update), + # not clear it (which snaps the view back to the blurry overview). + p.set_data(np.full((4096, 4096), 0.7, np.float32), clim=(0, 1)) + assert p._state["zoom"] == zoom + assert list(p._state["detail_region"]) == reg + assert p._state["detail_b64"], "detail tile was cleared (snap-back bug)" + # and the detail pixels reflect the new frame + x0, x1, y0, y1 = reg + crop = p._tile_backend.sample(x0, x1, y0, y1, 64, 64, "mean") + assert 0.6 < float(crop.mean()) < 0.8 + + def test_set_data_applies_new_contrast(self): + p = self._tiled(0.2) + p.set_data(np.full((4096, 4096), 5.0, np.float32), clim=(0, 10)) + assert p._state["display_min"] == 0 and p._state["display_max"] == 10 + + def test_set_data_shape_change_rederives_size(self): + p = self._tiled(0.2) + # A different-size frame (e.g. signal axes changed): image_width must follow + # the NEW frame and tiling stay on. + p.set_data(np.full((2048, 3072), 0.5, np.float32), clim=(0, 1)) + assert p._state["image_width"] == 3072 and p._state["image_height"] == 2048 + assert p._state["tile_enabled"] is True + assert 0 < p._state["base_width"] < 3072 + + def test_set_data_no_stray_full_res_in_base(self): + # The base texture (image_b64) must stay overview-sized, never balloon to the + # full 4096² (which is the whole point of tiling — no full-frame transfer). + p = self._tiled(0.2) + p.set_data(np.full((4096, 4096), 0.7, np.float32), clim=(0, 1)) + base_px = p._state["base_width"] * p._state["base_height"] + assert base_px <= 1024 * 1024, "base grew to full-res — tiling bypassed" + + +class TestSetDataAutoEnablesTiling: + """A live consumer (e.g. the SpyDE movie viewer) starts with a small placeholder + imshow, then set_data's the real large frames. set_data must AUTO-ENABLE tile mode + on the first frame past TILE_THRESHOLD — so the consumer never hand-rolls a + backend / calls enable_tile. Regression for SpyDE's tile path diverging from the + (fixed) set_data path.""" + + def _small(self, n=10): + return apl.subplots(1, 1)[1].imshow(np.zeros((n, n), np.float32)) + + def test_large_frame_auto_enables(self): + p = self._small() + assert p._state["tile_enabled"] is False + p.set_data(np.full((4096, 4096), 0.5, np.float32), clim=(0, 1)) + assert p._state["tile_enabled"] is True + assert p._state["image_width"] == 4096 + assert 0 < p._state["base_width"] <= 1024 + assert p._tile_backend is not None + + def test_small_frame_stays_untiled(self): + p = self._small() + p.set_data(np.full((512, 512), 0.5, np.float32), clim=(0, 1)) + assert p._state["tile_enabled"] is False + assert p._state["image_width"] == 512 + + def test_auto_enable_uses_full_res_contrast(self): + # No clim → the range comes from the full-res frame (native extremes), not the + # overview mean, so a subsequent zoom tile doesn't blow out to white. + p = self._small() + rng = np.random.RandomState(0) + p.set_data(rng.rand(4096, 4096).astype(np.float32)) # NO clim + assert p._state["tile_enabled"] is True + assert p._state["display_min"] < 0.05 + assert p._state["display_max"] > 0.95 + + def test_tile_false_never_auto_enables(self): + p = apl.subplots(1, 1)[1].imshow(np.zeros((10, 10), np.float32), tile=False) + p.set_data(np.full((4096, 4096), 0.5, np.float32), clim=(0, 1)) + assert p._state["tile_enabled"] is False # honoured tile=False + assert p._state["image_width"] == 4096 # plain full-res path + + +class TestTilePayloadParity: + """Parity guards for scenes where tile=True should be byte-identical to plain.""" + + @staticmethod + def _decoded_u8(plot): + st = plot.resolve_pixel_tokens(plot.to_state_dict()) + raw = base64.b64decode(st["image_b64"]) + arr = np.frombuffer(raw, dtype=np.uint8).reshape( + st["image_height"], st["image_width"] + ) + return arr + + def test_forced_tile_matches_plain_bytes_for_1024_frame(self): + # 1024x1024 avoids overview decimation while still exercising tile=True. + img = np.random.RandomState(0).rand(1024, 1024).astype(np.float32) + plain = apl.subplots(1, 1)[1].imshow( + img, cmap="gray", vmin=0.0, vmax=1.0, tile=False, gpu=False + ) + tiled = apl.subplots(1, 1)[1].imshow( + img, cmap="gray", vmin=0.0, vmax=1.0, tile=True, gpu=False + ) + a = self._decoded_u8(plain) + b = self._decoded_u8(tiled) + ok, msg = compare_arrays_exact(a, b) + assert ok, f"plain vs tiled payload mismatch: {msg}" + + def test_set_data_stays_identical_between_plain_and_tile(self): + base = np.zeros((1024, 1024), np.float32) + nxt = np.random.RandomState(1).rand(1024, 1024).astype(np.float32) + plain = apl.subplots(1, 1)[1].imshow( + base, cmap="gray", vmin=0.0, vmax=1.0, tile=False, gpu=False + ) + tiled = apl.subplots(1, 1)[1].imshow( + base, cmap="gray", vmin=0.0, vmax=1.0, tile=True, gpu=False + ) + plain.set_data(nxt, clim=(0.0, 1.0), tile=False) + tiled.set_data(nxt, clim=(0.0, 1.0), tile=True) + a = self._decoded_u8(plain) + b = self._decoded_u8(tiled) + ok, msg = compare_arrays_exact(a, b) + assert ok, f"set_data parity mismatch: {msg}" + + +class TestTiledRenderCanvas: + def test_overview_base_renders(self, interact_page): + # A tiled imshow renders SOMETHING (the overview) on the Canvas2D path. + img = np.tile(np.linspace(0, 1, 2048, dtype=np.float32), (2048, 1)) + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(img, cmap="gray", vmin=0, vmax=1, gpu=False) + page = interact_page(fig) + page.wait_for_timeout(300) + state = json.loads(page.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p._id)) + assert state.get("tile_enabled") is True + assert state.get("base_width", 0) > 0 and state["base_width"] < state["image_width"] + px = page.evaluate("""() => { + const cs = Array.from(document.querySelectorAll('canvas')); + const c = cs.sort((a,b)=>b.width*b.height-a.width*a.height)[0]; + const d = c.getContext('2d').getImageData((c.width*0.7)|0,(c.height*0.5)|0,1,1).data; + return d[0]; + }""") + assert px > 0, "overview base did not render" + + def test_injected_detail_tile_renders_crisp_when_zoomed(self, interact_page): + # Logical 2048² tiled image; base overview is a flat gray. Inject a detail + # tile (gray/white split) for a region + zoom in → the tile's split must show + # (proves the base-overview + detail-tile compose correctly under tile mode). + base = np.full((2048, 2048), 0.3, np.float32) + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(base, cmap="gray", vmin=0, vmax=1, gpu=False) + page = interact_page(fig) + page.wait_for_timeout(300) + # Inject a detail tile covering logical [768:1280]² (a 512² region), with a + # left-gray/right-white split, and zoom to it. + tile = np.full((256, 256), 0.5, np.float32) + tile[:, 128:] = 1.0 + p.set_detail(tile, 768, 1280, 768, 1280) + # Push the injected state into the browser + zoom so the window ⊆ the region. + # detail_b64 is a GEOM key now (rides panel__geom, spliced from geomCache), + # so inject it into the geom trait — setting it on the light view trait would + # be overwritten by _applyGeom from the (empty) geomCache. The small + # region/width/height fields stay on the light view trait. + st = p._state + page.evaluate("""(args) => { + const [pid, detail, geomExtra] = args; + // Merge detail_b64 into the geom trait so geomCache picks it up. + const gname = 'panel_'+pid+'_geom'; + let geom = {}; + try { geom = JSON.parse(window._aplModel.get(gname) || '{}'); } catch (_) {} + Object.assign(geom, geomExtra); + window._aplModel.set(gname, JSON.stringify(geom)); + const raw = JSON.parse(window._aplModel.get('panel_'+pid+'_json')); + Object.assign(raw, detail); + window._aplModel.set('panel_'+pid+'_json', JSON.stringify(raw)); + globalThis.__apl_setZoom(pid, 4.0, 0.5, 0.5); + }""", [p._id, + {"detail_region": st["detail_region"], + "detail_width": st["detail_width"], "detail_height": st["detail_height"], + "detail_seq": st.get("detail_seq", 1)}, + {"detail_b64": st["detail_b64"]}]) + page.wait_for_timeout(200) + info = page.evaluate("""() => { + const cs = Array.from(document.querySelectorAll('canvas')); + const c = cs.sort((a,b)=>b.width*b.height-a.width*a.height)[0]; + const ctx = c.getContext('2d'); const w=c.width,h=c.height,y=(h*0.5)|0; + return { left: ctx.getImageData((w*0.30)|0,y,1,1).data[0], + right: ctx.getImageData((w*0.70)|0,y,1,1).data[0] }; + }""") + # tile right (white) is brighter than tile left, and BOTH are brighter than + # the flat base overview (0.3 → ~77) — so the injected detail tile is showing, + # not the base. (Exact values depend on where the split lands in the fit-rect.) + assert info["right"] > info["left"] + 30, ( + f"detail tile split not visible over the overview: {info}") + assert info["left"] > 100, f"tile not shown — base(0.3~77) leaked: {info}" diff --git a/anyplotlib/tests/test_plot2d/test_zoom_view_write.py b/anyplotlib/tests/test_plot2d/test_zoom_view_write.py new file mode 100644 index 00000000..74709a93 --- /dev/null +++ b/anyplotlib/tests/test_plot2d/test_zoom_view_write.py @@ -0,0 +1,76 @@ +""" +Zoom/pan write payload — the light ``panel__json`` trait must NOT carry the +heavy geometry (pixels / colormap LUT) on interaction writes. + +Regression guard for the zoom-lag bug: the wheel/pan handlers serialise the +panel state back to ``panel__json`` on every mouse tick. ``_applyGeom`` +splices the cached geometry (``image_b64`` / ``colormap_data`` / the binary +``image_b64_bytes``) into ``p.state`` so the renderer can draw, which means a +naive ``JSON.stringify(p.state)`` re-serialises — and re-transmits — the whole +frame every tick. On the binary path that is ``JSON.stringify`` of a Uint8Array +(a ``{"0":..,"1":..}`` object with one key per byte), which stalls zoom on large +images. ``_viewStateJson`` strips the cached geom keys before serialising, and +the interaction handlers use it instead of ``JSON.stringify(p.state)``. + +The geom split is GPU-independent, so Playwright's WebGPU-less Chromium +(Canvas2D fallback) exercises the exact same write path the GPU path uses. +""" +from __future__ import annotations + +import json + +import numpy as np + +import anyplotlib as apl + + +_GEOM_KEYS = ("image_b64", "colormap_data", "overlay_mask_b64", + "image_b64_bytes", "overlay_mask_b64_bytes") + + +def _big_image(n=512): + # A ramp big enough that its base64 pixels dominate any view-state JSON. + row = np.linspace(0, 1, n, dtype=np.float32) + return np.tile(row, (n, 1)) + + +class TestZoomViewWrite: + def test_view_json_excludes_geometry_keys(self, interact_page): + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(_big_image(), cmap="viridis", gpu="auto") + page = interact_page(fig) + page.wait_for_timeout(200) + + raw = page.evaluate("(pid) => globalThis.__apl_viewStateJson(pid)", p._id) + assert raw, "probe __apl_viewStateJson did not return the view JSON" + view = json.loads(raw) + + # The exact string the wheel/pan handler writes per tick must NOT carry + # any heavy geometry key. + for k in _GEOM_KEYS: + assert k not in view, f"geom key {k!r} leaked into the light view write" + + # …but it MUST still carry the view state the handler is persisting. + assert "zoom" in view, "view state (zoom) missing from the light write" + + def test_view_json_is_much_smaller_than_full_state(self, interact_page): + fig, ax = apl.subplots(1, 1, figsize=(300, 300)) + p = ax.imshow(_big_image(), cmap="viridis", gpu="auto") + page = interact_page(fig) + page.wait_for_timeout(200) + + dbg = page.evaluate( + "(pid) => { globalThis.__apl_setZoom(pid, 2.0, 0.5, 0.5);" + " return globalThis.__apl_zoom_dbg; }", + p._id, + ) + assert dbg is not None, "probe did not run — __apl_setZoom missing?" + # The full-state stringify (OLD path) carries the pixels; the view-only + # stringify (NEW path) must be a small fraction of it. + assert dbg["len"] > 10_000, ( + f"test image too small to be a meaningful guard (full len={dbg['len']})" + ) + assert dbg["viewLen"] < dbg["len"] / 4, ( + "view-state JSON still carries geometry: " + f"viewLen={dbg['viewLen']} vs full len={dbg['len']}" + ) diff --git a/upcoming_changes/binary_token_cache_freeze.bugfix.rst b/upcoming_changes/binary_token_cache_freeze.bugfix.rst new file mode 100644 index 00000000..ae49fe19 --- /dev/null +++ b/upcoming_changes/binary_token_cache_freeze.bugfix.rst @@ -0,0 +1,11 @@ +Fixed a display freeze under the Electron binary pixel transport: the routing +layer stripped the pixel key out of the slimmed geom JSON, so the renderer's +"unchanged → skip re-upload" caches (Canvas2D blit cache, WebGPU texture, +overlay-mask cache) fell back to a 4-sampled-byte fingerprint of the buffer — +two frames differing anywhere else collided and the display stayed frozen on +the old frame (seen as a stale overview after a movie scrub). The slimmed geom +now carries a small ``\x00bin:`` content token under the pixel key, +binary buffers are additionally stamped with an arrival sequence as a fallback +key, and the overlay-mask draw path now reads the binary byte side-channel +(it previously only decoded base64, so masks never displayed over the binary +transport). diff --git a/upcoming_changes/gpu_pan_v_mirror.bugfix.rst b/upcoming_changes/gpu_pan_v_mirror.bugfix.rst new file mode 100644 index 00000000..6df29827 --- /dev/null +++ b/upcoming_changes/gpu_pan_v_mirror.bugfix.rst @@ -0,0 +1,11 @@ +Fixed the WebGPU 2-D image path sampling a vertically MIRRORED window when the +view was panned off-centre: the shader applied a global ``1 - v`` flip after +interpolating the ``[v0, v1]`` uv window, which sampled ``[1-v1, 1-v0]`` +instead — correct only for a full or vertically-centred view. Symptoms: pan-y +moved the image the wrong way on GPU-rendered panels, and markers/widgets +(drawn by the shared Canvas2D overlay transform, which was always correct) +appeared detached from the image features they marked. The base and +detail-tile passes share the shader, so both are fixed. GPU-vs-CPU screenshot +parity tests (zoom, pan, markers, widgets, detail tile) now run on real WebGPU +in headless Chromium (``channel="chromium"`` + ``--enable-unsafe-webgpu``) and +skip on machines with no adapter. diff --git a/upcoming_changes/lazy_data_cast.bugfix.rst b/upcoming_changes/lazy_data_cast.bugfix.rst new file mode 100644 index 00000000..545bb523 --- /dev/null +++ b/upcoming_changes/lazy_data_cast.bugfix.rst @@ -0,0 +1,5 @@ +``Plot2D.set_data`` no longer makes a float64 copy of every incoming frame. +The float64 cast now happens lazily in the ``.data`` property (the only reader), +so a frame stream — e.g. scrubbing an in-situ movie — keeps the source dtype and +skips a ~12 ms float64 copy of a 4k frame per tick. ``.data`` still returns a +read-only float64 copy, unchanged for callers. diff --git a/upcoming_changes/raw_pixel_transport.bugfix.rst b/upcoming_changes/raw_pixel_transport.bugfix.rst new file mode 100644 index 00000000..02029326 --- /dev/null +++ b/upcoming_changes/raw_pixel_transport.bugfix.rst @@ -0,0 +1,12 @@ +The Electron binary pixel transport now ships the RAW uint8 image bytes end to +end, instead of base64-encoding them in ``set_data`` only to base64-decode them +straight back in the routing layer. ``Plot2D.set_data`` stashes the raw bytes on +the Figure's ``_raw_pixels`` side-table and leaves a tiny content-checksum +change-token in ``image_b64``; ``_electron._route_change`` ships those bytes to a +PLOTBIN frame directly. This removes the ~20 ms base64 encode, the ~17 ms decode, +and the megabyte ``json.dumps`` of the pixel string from every scrub frame — a +2.2x faster ``set_data`` (≈98 ms → ≈44 ms on a 2048² frame) and ~25% less +bytes-on-wire. Non-Electron hosts (Jupyter / Pyodide / standalone / ``save_html``) +are unchanged: they have no PLOTBIN channel, so the token is resolved back to +inline base64 via ``Plot2D.resolve_pixel_tokens`` when the figure state is +serialised for them. diff --git a/upcoming_changes/stale_overview_zoom_out.bugfix.rst b/upcoming_changes/stale_overview_zoom_out.bugfix.rst new file mode 100644 index 00000000..f1077a3d --- /dev/null +++ b/upcoming_changes/stale_overview_zoom_out.bugfix.rst @@ -0,0 +1,6 @@ +Tile mode: a data update while zoomed in (``update_tile_source`` with a detail +tile shown) refreshes only the detail tile, leaving the overview base on the +old frame — zooming out then flashed the pre-update frame. The skipped +overview is now marked stale and re-sampled once on the next view settle +(riding the same push as the detail/clear), preserving the per-frame skip +optimisation while never exposing stale base pixels. diff --git a/upcoming_changes/webgpu_2d_image.new_feature.rst b/upcoming_changes/webgpu_2d_image.new_feature.rst new file mode 100644 index 00000000..7a04e1fe --- /dev/null +++ b/upcoming_changes/webgpu_2d_image.new_feature.rst @@ -0,0 +1,7 @@ +2-D scalar images can now render on the **GPU via WebGPU** (``imshow(..., +gpu="auto"|True|False)``): the image uploads as an R8 texture and a WGSL fragment +shader applies the colormap LUT + contrast (clim) in one draw, replacing the +per-pixel JavaScript colormap loop. Large images (≳1 megapixel) take the GPU path +automatically; everything below the threshold, RGB images, ``gpu=False``, and any +device without WebGPU keep the identical Canvas2D path. ``plot.gpu_active`` reports +which path ran. Verified on an NVIDIA Pascal GPU. diff --git a/upcoming_changes/zoom_view_write.bugfix.rst b/upcoming_changes/zoom_view_write.bugfix.rst new file mode 100644 index 00000000..a84fa3cc --- /dev/null +++ b/upcoming_changes/zoom_view_write.bugfix.rst @@ -0,0 +1,8 @@ +Interactive zoom/pan on a 2-D image no longer re-serialises (and re-transmits) +the full image on every mouse tick. The wheel/pan/orbit handlers write only the +light *view* state back to the ``panel__json`` trait now, excluding the +cached geometry (pixels, colormap LUT) that ``_applyGeom`` splices into the panel +state for drawing. Previously the whole frame was ``JSON.stringify``-d per tick — +catastrophically so on the binary transport, where the pixel buffer is a +``Uint8Array`` that stringifies to a ``{"0":..,"1":..}`` object with one key per +byte — which stalled zoom on large images.