Add copy_sorting option to SortingAnalyzer.create_memory#4668
Open
h-mayorquin wants to merge 3 commits into
Open
Add copy_sorting option to SortingAnalyzer.create_memory#4668h-mayorquin wants to merge 3 commits into
h-mayorquin wants to merge 3 commits into
Conversation
When copy_sorting=False, create_memory keeps the given sorting as-is instead of materializing it into a NumpySorting via copy_spike_vector=True. This lets a lazy sorting (e.g. a streamed NWB sorting) stay lazy: its spike times are read on demand rather than pulled up front, which matters for large remote sortings where a view may need only a few units' trains. Default stays True, so existing behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
h-mayorquin
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
create_memoryalways materializes the sorting into an in-memoryNumpySortingviafrom_sorting(..., copy_spike_vector=True), which requests the whole spike vector up front and discards any laziness the sorting provides. This adds acopy_sortingparameter (defaultTrue, so existing behavior is unchanged); withcopy_sorting=Falsethe given sorting is kept as-is, so it stays lazy through analyzer construction and its spike times are read only on demand. This pairs directly with #4662, which keeps theNwbSortingExtractorspike vector lazy (nothing read until first requested): without this optioncreate_memoryimmediately re-materializes exactly what #4662 made lazy, so the two are needed together for a streamed sorting to reach an analyzer without a full spike read. The immediate consumer is the streamingread_nwb_sorting_analyzerreader (#4645).The payoff is building analyzers over large or streamed sortings where the spike vector is often never touched (template-and-metric analysis, batch surveys, memory-constrained builds). Materializing costs roughly three times the on-disk size in RAM and grows with the recording. Querying IBL processed files from dandiset 000409 (Brain Wide Map) by metadata only (via lindi), a single probe insertion runs 16-37M spikes: session
6713a4a7-faed-4df2-acab-ee4e63326f8dhas 898 units and 20.7M spikes (~0.50 GB materialized), and session4364a246-f8d7-4ce7-ba23-a098104b96e4has 1202 units and 36.6M spikes (~0.88 GB), against only 18-25 MB of templates that template-and-metric workflows actually use. A survey across the hundreds of Brain Wide Map sessions would otherwise pay gigabytes of RAM and hundreds of gigabytes of reads for spikes it immediately discards. The default preserves today's behavior.