Skip to content

Added in place operation helpers for Raw Records and Records processi… - #1018

Open
Naderschah wants to merge 12 commits into
AxFoundation:masterfrom
Naderschah:InPlaceOperations
Open

Naderschah wants to merge 12 commits into
AxFoundation:masterfrom
Naderschah:InPlaceOperations

Conversation

@Naderschah

Copy link
Copy Markdown

To optimize raw_records and records production in the straxen plugins DAQReader and PulseProcessor

Some modified variants of general io and pulse processing modules were required. The primary focus was reducing copy
operations. Processing speed increased overall as a result.

Most are added as _inplace variants of existing functions. The only permanent change is the lz4 decompressor where data is not repeatedly allocated anymore, as this was the function producing a vast majority of allocation.

All was already tested on AmBe runs with the detector.

@Naderschah Naderschah mentioned this pull request Sep 25, 2026
1 of 4 tasks
Naderschah and others added 4 commits September 25, 2026 15:48
… in live single chunk testing the two suboperations occupied 5.2 seconds of 66seconds total with peak ram growth of 4.44GB, post change this drops to 0.48 seconds with ~0.0GB ram growth
@Naderschah

Copy link
Copy Markdown
Author

The new peaklet level changes were tested on a number of individual chunks, full processing runs will be tested as more data is available the most recent pushed changes here are required primarily to drop peak ram growth, form per chunk profiling

0 baseline Time: 79.483497 Peak RAM Delta 7.477806
1 candidate Time: 58.654074 Peak Ram Delta 5.280800

Note that this varies per chunk by +- 5 seconds

@WenzDaniel

Copy link
Copy Markdown
Collaborator

Hej two comments before we can continue with the review:

  1. Please adjust the formatting. Currently you have many empty lines which we typically do not do in strax.
  2. Please elaborate in much more detail what your proposed change exactly does. Please provide some examples.

@Naderschah

Copy link
Copy Markdown
Author
  1. Newlines removed to match surrounding style

Ok, code changes, first I fergot to link the corresponding straxen PR XENONnT/straxen#1680 which contains the higher level need for these changes. Context was AmBe preperation to cope with live processing

In general the purpose of the functions is to reduce memory churn and in turn processing time by reducing the amount of data that needs to be allocated, changes here along with the linked PR affect live_data to raw_records, raw_records to records, and records to peaklets, with an overall reduction in computational time of 1/3. Alongside this most changes introduce a in place variant of an existing function that is to be used in the same way as the original. With the remainder being helper functions.

strax/processing/data_reduction.py

cut_outside_hits_inplace:

  • An in place variant of cut_outside_hits which zeroes out left and right portions of a waveform, this requires _cut_outside_hits_inplace_core alongside _build_hit_offsets
    _build_hit_offsets:
  • Pre generate an offsets array to grab hits out of records, this is required as my rewrite iterates records rather than hits
    _cut_outside_hits_inplace_core:
  • The primary difference is that I do not preallocate a zero array but instead zero out on the input array to avoid a copy. This is primarily possible as I assume hits may not be unsorted within records (which is a valid assumption coming from downstream)
    Independent usage of _ functions is not intended and these are primarily there to allow jit compillation and logical seperation. cut_outside_hits_inplace is to be used the same way as cut_outside_hits

strax/processing/general.py

apply_permutation_in_place:

  • Given a permutation array, permute an array such that new[i] = old[permutation[i]], both inputs are mutated with permutations effectively destroyed, does not reallocate only moves data around. Usage example
raw_bytes = np.array([[10], [20], [30]], dtype=np.uint8)
permutation = np.array([2, 0, 1])
apply_permutation_in_place(raw_bytes, permutation)

sort_by_time_in_place:

  • Replacement method for sort_by_time, falls back to same core method as sort_by_time should sorting be unsafe with given indexes,
  • Primary improvement is avoidance of calling _sort_by_time_and_channel, since this returns x[sort_i] producing a copy of the array
  • Used the same way as sort_by_time
    stable_sort_by_time_in_place:
  • An added function for peak_splitting replacing usage of sort_by_time as channel sorting is not required in this case

strax/processing/peak_splitting.py:

PeakSplitter.call:

  • Modified the function, here I did very little and only addressed the easiest modification I could find as this is far more involved than the lower level changes
  • Here strax.sort_by_time(np.concatenate([peaks[~is_split], new_peaks])) is replaced by combine_unsplit_and_new and stable_sort_by_time_in_place
  • This avoids multiple array copies, one from concatenate & peaks[~is_split] via the following function and avoids the multiple intermediaries created in sort_by_time

combine_unsplit_and_new:

  • preallocates split array and populates it with new peaks, produces single allocation rather than the two mentioned above
  • Usage outside the class or anywhere else is not intended

strax/io.py

_lz4_decompress:

  • Usage unchanged, function decompresses a single lz4 compressed record, this was the highest source of memory churn
  • In the original function chunks are loaded, decompressed and then a output array is extended repeatedly
  • The replacement preallocates the compressed array size which is known ahead of time, reading compressed data into it decompressing only on function return, requiring one compressed and one decompressed allocation only
  • Note that this in contrast to everything else done actually increases peak memory, but significantly reduces allocation churn. In testing with DAQReader raw_records allocation churn dropped from 33GB to 15GB
  • Usage is unchanged relative to the previous implementation

@WenzDaniel

Copy link
Copy Markdown
Collaborator

Same style comment also for https://github.com/XENONnT/straxen/pull/1680/changes. Did the line breaking of some of the functions change due to black or something? Because it makes it a bit harder to distinguish changes from just style changes.

@WenzDaniel WenzDaniel left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here already some comments for data_reduction.py



@export
def cut_outside_hits_inplace(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some reading and cleanliness improvement. Better to sort the code like any other text from general to specific. When I review your code, or would like to better understand what your functions are doing I start here. And only if I am interested to learn more about how _build_hit_offsets works while reading cut_outside_hits_inplace, I scroll down to read it,

Comment on lines +237 to +238
left_extension=2,
right_extension=15,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Maybe better put the straxen defaults for consistency (3, 20).

left_extension=2,
right_extension=15,
):
"""In-place equivalent of strax.cut_outside_hits for the PulseProcessing use case.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe one day this function will be the default and we will remove the other function. So in this case I think it would be good to generalize the doc string such that it explains what the function is doing.

):
"""In-place equivalent of strax.cut_outside_hits for the PulseProcessing use case.

Hits must be ordered by ``record_i``, as produced by ``strax.find_hits``.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really mean record_i as in the fragment index within a pulse or do you mean the index of the records? Because the fragment index wraps at the end of a pulse, while hits are a continuous single dimensional array. So I am not entirely sure how a sorting by record_i would look like. Just want to make sure.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, after reading your code above, I understand better know. Okay this is indeed confusing and misleading. record_i in records means something different than in hits. This is of course not nice, maybe it makes sense to change the dtype of hits in this case since the only plugin which is exposing this dtype as stored data are aqmon_hits that are fast to recompute. However, this would then break a compatibility of a new strax version with older data.

h_i = 0
for r_i in range(n_records):
offsets[r_i] = h_i
while h_i < len(hits) and hits[h_i]["record_i"] == r_i:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make conditions like this one easier to read I generally suggest to write things like this:

_hits_are_not_finished_and_current_hit_is_in_record_i = (
    (h_i < len(hits))
    and (hits[h_i]["record_i"] == r_i)
)

while _hits_are_not_finished_and_current_hit_is_in_record_i:
    ...

In this way I know instantly what the while loop is doing, while when writing only the condition it takes me 20 to 30 s to understand what is going on. In the meantime one forgets about the main code of the function making reviewing and understanding the code much harder.

I know that this looks a bit tedious when one is not used to it, but if one keeps this in mind and it becomes second nature it cost nothing and improves readability by a lot.

Comment on lines +195 to +198
if a < 0:
a = 0
if a < suffix_start:
suffix_start = a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, but with min instead,

Comment on lines +210 to +213
if a < 0:
a = 0
if b > record_length:
b = record_length

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same min/max comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my line 202 to line 228 comment first since this may null other comments for these lines.

Comment on lines +208 to +209
a = hits[h_i]["left"] - left_extension
b = hits[h_i]["right"] + right_extension

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use meaningful variable names instead of a and b? How about left_hit_bound and right_hit_bound or something similar?

Comment on lines +214 to +215
if b <= a:
continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not this raise rather an error? We should never run in this condition should we?

Comment on lines +202 to +228
cursor = prefix_end
record_length = int(r["length"])
for h_i in range(
hit_offsets[r_i],
hit_offsets[r_i + 1],
):
a = hits[h_i]["left"] - left_extension
b = hits[h_i]["right"] + right_extension
if a < 0:
a = 0
if b > record_length:
b = record_length
if b <= a:
continue
if a >= suffix_start:
break
if b > suffix_start:
b = suffix_start
if b <= cursor:
continue
if a > cursor:
r["data"][cursor:a] = 0
if b > cursor:
cursor = b

if cursor < suffix_start:
r["data"][cursor:suffix_start] = 0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole for-loop with the many if conditions is very complicated. Would not the following do the same thing:

zle_buffer = np.zeros(samples_per_record, np.int16)
zle_buffer[:suffix_start] = 1
zle_buffer[prefix_end:] = 1

for h_i in range(
    hit_offsets[r_i],
    hit_offsets[r_i + 1],
    ):
    left_hit_bound = hits[h_i]["left"] - left_extension
    left_hit_bound  = max(0, left_hit_bound )
    right_hit_bound  = hits[h_i]["right"] + right_extension
    right_hit_bound = max(right_hit_bound, samples_per_record) 
    
    zle_buffer[left_hit_bound:right_hit_bound  ] = 1
    r["data"] *= zle_buffer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants