feat: WebGPU 2-D image rendering, tiled display, binary pixel transport#26
Merged
Conversation
…transport Extends the existing WEBGPU_PLAN.md (3D points/voxels) to 2D large images: a WGSL texture render path (reusing the 3D device singleton + canvas-fallback contract), shader-LUT colormap + clim + mipmaps, and a binary pixel transport off base64. Phased with decision gates; Canvas2D stays the default + fallback. Motivating consumer: the SpyDE in-situ movie viewer. Not yet implemented.
…es (Phase 0) Adds a WebGPU 2-D image path that replaces the Canvas2D atob+64M-iteration LUT loop with one GPU draw: the normalized uint8 frame uploads to an R8 texture, the 256- entry colormap to a 256x1 RGBA texture, and a fullscreen-quad WGSL fragment shader re-stretches by clim (dmin/dmax uniform) and samples the LUT. Reuses the existing 3D _gpuDevice() singleton + device-lost fallback contract. - New 2D gpuCanvas below plotCanvas (decorations keep drawing on plotCanvas over a now-transparent bg); _gpuWanted2d gate (scalar images >= ~1 Mpx, gpu_mode aware). - imshow/Plot2D gain gpu='auto'|True|False -> gpu_mode state; async first-frame- canvas-then-swap; any GPU failure reverts to the Canvas2D blit for that frame. - Verified in the SpyDE Electron app on a Pascal GPU: a 1366x1366 movie frame renders via the shader (gpu active, correct image + scale bar). Diagnostic: globalThis.__apl_gpu2d / __apl_build for the Electron GPU test.
__apl_gpuReadback(panelId, N) re-renders the active image panel into an offscreen RGBA texture and copies it back to CPU — the reliable way to verify the shader output (the live swapchain canvas reads black under automation, per FIGURE_ESM.md). Handles bgra/rgba formats + 256-byte row padding. Verified on the Pascal GPU (SpyDE Electron app, real movie frame): min 0 / max 255 / 96% non-black / gray-cmap values — the shader-LUT + clim produce a correct, varied, properly-contrasted image.
…resolve
The 2D gpuCanvas was inserted as the FIRST child of plotWrap, so tests/callers
doing querySelector('canvas') got the (empty) gpuCanvas instead of the image
plotCanvas — the RGB image read back [0,0,0,0]. Fixed: append gpuCanvas AFTER
plotCanvas in DOM (plotCanvas stays first); z-index (plotCanvas:1 > gpuCanvas:0)
controls the visual stacking so decorations still composite over the GPU image.
Updated the one title test whose plotCanvas detection relied on 'no z-index'
(plotCanvas now has z-index:1) to use DOM-first-canvas instead.
anyplotlib suite: 162 2D/title tests pass; GPU path re-verified on Pascal
(offscreen readback min 0 / max 255 / 96% non-black).
Plot2D gains gpu_active (property) + _set_gpu_active, mirroring Plot3D: the JS _reportGpu fires a gpu_status event → the Figure routes it to _set_gpu_active, so plot.gpu_active reflects the ACTUAL render path (True only when the WebGPU image path activated), not just the requested gpu_mode. test_gpu_image.py (8 tests): gpu param → gpu_mode mapping, gpu_active echo + event dispatch, and the Canvas2D FALLBACK contract — gpu=False and sub-threshold images always render on canvas with gpu_active False (Playwright Chromium has no real GPU device, so this is the fallback path; the GPU-active render is verified on the Pascal GPU in the SpyDE electron test).
…d items Updates LARGE_IMAGE_WEBGPU_PLAN.md status to Phase-0-done/hardware-verified and records the two deferred refinements (mipmaps; binary transport — now incremental since LOD already cut the shipped payload ~35x). Adds a towncrier new_feature fragment for the imshow gpu= WebGPU path.
Review found two CRITICAL bugs masked by only testing the default-clim, unzoomed case: - CRITICAL 1 (clim applied twice): _buildLut32 already bakes the display window (clim) AND scale_mode into a direct pixel->colour LUT; the shader then re-applied the window via a dmin/dmax uniform -> correct only at full-range clim, wrong for any narrowed contrast (and SpyDE's 2-98% nav levels). Fixed: the shader is now a plain IDENTITY LUT lookup (raw/255 -> LUT), uniform removed. This also fixes HIGH (log/symlog — the LUT already encodes the curve). Verified on the Pascal GPU: a narrowed clim [0.3,0.7] matches the numpy windowed-colormap to meanDiff 0.65/255. - CRITICAL 2 (GPU quad ignored zoom/pan): the fullscreen quad drew the full extent while the axes/scale-bar/mask/markers on plotCanvas honor zoom/center -> the base image detached from its overlays when zoomed. Fixed: _gpuWanted2d falls back to Canvas2D whenever zoom != 1 or center != 0.5 (fast on a static view). - MEDIUM: _gpuDisposeImagePanel frees the R8/LUT textures on panel removal, figure dispose, and device loss (was leaking tens of MB per large-image panel close). - MEDIUM: async GPU-init guards on panels.has(p.id) (panel removed mid-await). - LOW: gpu_active echoes False on device loss; sampler is nearest on both textures (pixel-faithful match to Canvas2D imageSmoothingEnabled=false). anyplotlib 2D+title suite green (170); SpyDE movie scrub 5/5 + playback 5 frames.
… HIGH) Re-review found the GPU fullscreen quad ignored _imgFitRect: a non-square image (or any image in a non-square panel) rendered STRETCHED to fill the whole gpuCanvas while the axes/scale-bar/mask/markers drew in the centered aspect-preserved fit-rect — the base image detached + distorted (same failure class as the zoom bug, via aspect mismatch). Square test images masked it. Fix: a vec4 uniform carries the image's fit-rect in clip space (from _imgFitRect, the same rect _blit2d + overlays use); the vertex shader maps a 0..1 quad into it, and the render clears to theme.bgCanvas so the letterbox bars match Canvas2D. The offscreen readback fills the full target (rect [-1,-1,1,1]) to sample the whole image regardless of aspect. uniformBuf freed in dispose. Stale comment fixed. Verified: anyplotlib 2D+title suite green (170); GPU render correct on the Pascal (square movie frame fills panel, scale bar registered, readback min0/max255/96%).
…-in) The GPU 2-D image path used to FALL BACK to Canvas2D on any zoom/pan (the quad drew the full extent, so it would detach from the zoom-honouring overlays). Now the shader honours zoom/center exactly like _blit2d: the uniform carries the on-screen clip rect AND a texture uv sub-region. Zoom-in samples a window of the image over the fit-rect (nearest sampling UPSAMPLES real texels); zoom-out shrinks the dest rect. _imageDrawUniform computes both from st.zoom/center_x/center_y (mirrors the Canvas2D _blit2d formula). Removed the zoom/center early-out from _gpuWanted2d so GPU stays active at any zoom. Uniform grew 16->32B (two vec4); the offscreen readback samples the full texture (uv 0..1) regardless of live zoom. anyplotlib 2D+title suite green (170); GPU render correct at zoom=1 on the Pascal (readback min0/max255/92%).
…(WIP render) Opt-in binary transport for large image pixels over the Electron stdout channel, to cut the ~200ms/frame base64+JSON+atob cost on a 4k movie. OFF by default (APL_BINARY_TRANSPORT=1 to enable); the base64 path is untouched for Jupyter/Pyodide/standalone. Solid + tested (test_electron/test_binary_frame.py, 9 tests, CI matrix): - _binary_frame.py: PLOTBIN:<hlen>:<plen>\n<hdr><payload> wire format, length- prefixed + binary-safe (payload may contain the marker/newlines); encode/decode round-trip + 2-frame concat. - _electron.py: _route_change parses the panel_<id>_geom JSON, pulls image_b64/ overlay_mask_b64 out as raw PLOTBIN frames (emit_binary → sys.stdout.buffer), sends the slimmed geom (LUT/flags) as normal JSON. Routing tested both directions. - figure_esm.js: _imageBytes prefers raw bytes (image_b64_bytes) over atob; _gpuWanted2d + Canvas2D accept binary; _loadGeom preserves *_bytes across a geom refresh; per-panel binary-trait listeners merge bytes into the geomCache. - _repr_utils.py: awi_state_binary handler routes bytes into the panel geom. KNOWN ISSUE (why it's off): the render-side geom-merge doesn't paint the frame yet in the real app (backend emits PLOTBIN correctly — verified marker — but the Signal window stays blank). Framing + demux are proven; the paint wiring needs more work.
…de-table Complete the binary pixel transport render wiring (works end-to-end now): - Pixels are NESTED in the panel_<id>_geom JSON trait, not a top-level image_b64; _route_change parses the geom, pulls image_b64/overlay_mask_b64 out as PLOTBIN frames, sends the slimmed geom (LUT/flags) as JSON. - The anywidget JS model does NOT round-trip a Uint8Array set on an ad-hoc trait (model.get returned undefined) — so the iframe stashes raw bytes in a global side-table (__apl_pixbytes) keyed by geom_trait::pixelKey and bumps a small token trait; the ESM's change: listener reads bytes from the side-table and merges them into the panel geomCache under <key>_bytes. - _imageBytes/_gpuWanted2d/Canvas2D accept image_b64_bytes (no atob); _loadGeom preserves *_bytes across a geom refresh; a content fingerprint keys the cache so same-length frames don't wrongly skip. - _repr_utils awi_state_binary handler + paint telemetry (__apl_paint_kind/_ms). Removed the stderr marker from anyplotlib's emit_binary (the host owns it — see the SpyDE ipc fix; a redirecting host MUST patch emit_binary). Verified: pixel-identical render, ~12ms paint. test_electron 9 tests green (geom-nested routing both ways).
…100ms) Profiling a movie scrub showed _build_colormap_lut was the single biggest paint cost (~100ms/frame) — and it was rebuilt from scratch on EVERY frame (colorcet lookup + 256 hex parses) even though the colormap never changes during a scrub. It's a pure function of the colormap name, so lru_cache it: cold ~330ms -> warm ~0ms. Helps base64 AND binary paths. The returned list is read-only to callers (they JSON-serialise it). anyplotlib 2D+electron suite green (173).
…ame) _normalize_image rescaled through float64 every frame (a full uint8->float64-> uint8 round trip over 4M px = ~60ms on a 2048 frame, every movie frame). Use float32 when the value magnitude/span is within its exact range (uint8/uint16 movies — the common case), keeping float64 for high-magnitude float sources where float32 subtraction would band. Byte-identical result; vmin/vmax still full- precision. anyplotlib 2D+electron suite green (173).
Pushes 8 byte-distinct frames (moving bright band) back-to-back via rAF (the coalescing regime) and asserts the rendered band matches a settled single-frame reference of the LAST frame, not the first — plain + tiled, GPU + Canvas2D. Newest-wins coalescing may drop intermediates; freezing on an early frame is the bug class this catches (all 4 tests fail if the frame dedup key is forced constant).
…nitial paint On binary transport the slimmed geom JSON carries only LUT/flags; the pixel bytes land in the __apl_pixbytes side-table and were merged into _geomCache[<key>_bytes] ONLY by the change listener. The iframe registers its message handler synchronously but render() runs after an async import(blobUrl), so a push landing in that gap updated the model with no listener to consume it: the initial paint parsed the slim JSON, the bytes sat stranded, and the panel stayed permanently blank until an organic re-push (base64 was unaffected — those pixels ride inside the geom JSON the initial paint reads). Fix: factor the listener's merge logic into _spliceBinaryBytes (image / overlay-mask / detail keys, preserves __aplSeq so the dedup key and newest-wins coalescing are unchanged) and call it at initial paint in _createPanelDOM and _createInsetDOM. No-op when the side-table is empty, so the normal path is untouched. test_first_paint_race.py forces the ordering deterministically (gates renderFn behind a promise, delivers the awi_state/awi_state_binary message first): the binary case fails without the fix, base64 guards the already-working path.
On GPU-less CI runners (GitHub ubuntu) requestAdapter() resolves a SwiftShader adapter, so the adapter-only probe passed and every GPU parity / rapid-frame test then timed out (20s each) waiting for the __apl_gpu2d activation that can never happen. Probe what the figure ESM actually needs — adapter, device, canvas 'webgpu' context, configure, one cleared render pass — each step raced against a timeout, and skip the suite when any step fails.
CSSFrancis
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26 +/- ##
==========================================
- Coverage 89.29% 88.97% -0.33%
==========================================
Files 36 38 +2
Lines 2962 3511 +549
==========================================
+ Hits 2645 3124 +479
- Misses 317 387 +70 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WebGPU rendering for large 2-D scalar images, tiled display for arbitrarily large frames, and a binary pixel transport — the anyplotlib half of the SpyDE in-situ movie viewer work (consumed by CSSFrancis/spyde#4, which pins this branch by sha). Targets smooth scrubbing/playback of 4k×4k+ image stacks.
WebGPU 2-D image path
plot.gpu_activeechoes the real state; every failure (no adapter, device lost) falls back to the Canvas2D blit._imgFitRect, review-hardened clim/zoom/leak fixes, offscreen-texture readback hook for correctness tests.Tiled display
view_changed.imshow(huge, tile="auto")public API; pluggableTileBackend(numpy box-mean default).Binary pixel transport
PLOTBINframing + routing: raw-uint8 pixels ride a binary side-channel instead of base64 JSON (~2.2x faster per frame); first-paint race fixed by splicing side-table bytes at initial paint.Perf + misc
clip_displayoption;set_extentupdates axes; pcolormesh rasterization for regular meshes;anyplotlib.__version__.Testing
--enable-unsafe-webgpu). The session probe exercises the full render path (adapter → device → canvas context → render pass) and skips the GPU suites on runners without a usable WebGPU path (e.g. GitHub ubuntu runners), instead of timing out.