Repository navigation
Added in place operation helpers for Raw Records and Records processi… - #1018
Naderschah wants to merge 12 commits into
Conversation
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… in live single chunk testing the two suboperations occupied 5.2 seconds of 66seconds total with peak ram growth of 4.44GB, post change this drops to 0.48 seconds with ~0.0GB ram growth
for more information, see https://pre-commit.ci
|
The new peaklet level changes were tested on a number of individual chunks, full processing runs will be tested as more data is available the most recent pushed changes here are required primarily to drop peak ram growth, form per chunk profiling 0 baseline Time: 79.483497 Peak RAM Delta 7.477806 Note that this varies per chunk by +- 5 seconds |
|
Hej two comments before we can continue with the review:
|
for more information, see https://pre-commit.ci
Ok, code changes, first I fergot to link the corresponding straxen PR XENONnT/straxen#1680 which contains the higher level need for these changes. Context was AmBe preperation to cope with live processing In general the purpose of the functions is to reduce memory churn and in turn processing time by reducing the amount of data that needs to be allocated, changes here along with the linked PR affect live_data to raw_records, raw_records to records, and records to peaklets, with an overall reduction in computational time of 1/3. Alongside this most changes introduce a in place variant of an existing function that is to be used in the same way as the original. With the remainder being helper functions. strax/processing/data_reduction.py cut_outside_hits_inplace:
strax/processing/general.py apply_permutation_in_place:
raw_bytes = np.array([[10], [20], [30]], dtype=np.uint8)
permutation = np.array([2, 0, 1])
apply_permutation_in_place(raw_bytes, permutation)sort_by_time_in_place:
strax/processing/peak_splitting.py: PeakSplitter.call:
combine_unsplit_and_new:
strax/io.py _lz4_decompress:
|
for more information, see https://pre-commit.ci
|
Same style comment also for https://github.com/XENONnT/straxen/pull/1680/changes. Did the line breaking of some of the functions change due to black or something? Because it makes it a bit harder to distinguish changes from just style changes. |
|
|
||
|
|
||
| @export | ||
| def cut_outside_hits_inplace( |
There was a problem hiding this comment.
Some reading and cleanliness improvement. Better to sort the code like any other text from general to specific. When I review your code, or would like to better understand what your functions are doing I start here. And only if I am interested to learn more about how _build_hit_offsets works while reading cut_outside_hits_inplace, I scroll down to read it,
| left_extension=2, | ||
| right_extension=15, |
There was a problem hiding this comment.
Minor: Maybe better put the straxen defaults for consistency (3, 20).
| left_extension=2, | ||
| right_extension=15, | ||
| ): | ||
| """In-place equivalent of strax.cut_outside_hits for the PulseProcessing use case. |
There was a problem hiding this comment.
Maybe one day this function will be the default and we will remove the other function. So in this case I think it would be good to generalize the doc string such that it explains what the function is doing.
| ): | ||
| """In-place equivalent of strax.cut_outside_hits for the PulseProcessing use case. | ||
|
|
||
| Hits must be ordered by ``record_i``, as produced by ``strax.find_hits``. |
There was a problem hiding this comment.
Do you really mean record_i as in the fragment index within a pulse or do you mean the index of the records? Because the fragment index wraps at the end of a pulse, while hits are a continuous single dimensional array. So I am not entirely sure how a sorting by record_i would look like. Just want to make sure.
There was a problem hiding this comment.
Ah okay, after reading your code above, I understand better know. Okay this is indeed confusing and misleading. record_i in records means something different than in hits. This is of course not nice, maybe it makes sense to change the dtype of hits in this case since the only plugin which is exposing this dtype as stored data are aqmon_hits that are fast to recompute. However, this would then break a compatibility of a new strax version with older data.
| h_i = 0 | ||
| for r_i in range(n_records): | ||
| offsets[r_i] = h_i | ||
| while h_i < len(hits) and hits[h_i]["record_i"] == r_i: |
There was a problem hiding this comment.
To make conditions like this one easier to read I generally suggest to write things like this:
_hits_are_not_finished_and_current_hit_is_in_record_i = (
(h_i < len(hits))
and (hits[h_i]["record_i"] == r_i)
)
while _hits_are_not_finished_and_current_hit_is_in_record_i:
...In this way I know instantly what the while loop is doing, while when writing only the condition it takes me 20 to 30 s to understand what is going on. In the meantime one forgets about the main code of the function making reviewing and understanding the code much harder.
I know that this looks a bit tedious when one is not used to it, but if one keeps this in mind and it becomes second nature it cost nothing and improves readability by a lot.
| if a < 0: | ||
| a = 0 | ||
| if a < suffix_start: | ||
| suffix_start = a |
There was a problem hiding this comment.
Same as above, but with min instead,
| if a < 0: | ||
| a = 0 | ||
| if b > record_length: | ||
| b = record_length |
There was a problem hiding this comment.
Please see my line 202 to line 228 comment first since this may null other comments for these lines.
| a = hits[h_i]["left"] - left_extension | ||
| b = hits[h_i]["right"] + right_extension |
There was a problem hiding this comment.
Can we use meaningful variable names instead of a and b? How about left_hit_bound and right_hit_bound or something similar?
| if b <= a: | ||
| continue |
There was a problem hiding this comment.
Should not this raise rather an error? We should never run in this condition should we?
| cursor = prefix_end | ||
| record_length = int(r["length"]) | ||
| for h_i in range( | ||
| hit_offsets[r_i], | ||
| hit_offsets[r_i + 1], | ||
| ): | ||
| a = hits[h_i]["left"] - left_extension | ||
| b = hits[h_i]["right"] + right_extension | ||
| if a < 0: | ||
| a = 0 | ||
| if b > record_length: | ||
| b = record_length | ||
| if b <= a: | ||
| continue | ||
| if a >= suffix_start: | ||
| break | ||
| if b > suffix_start: | ||
| b = suffix_start | ||
| if b <= cursor: | ||
| continue | ||
| if a > cursor: | ||
| r["data"][cursor:a] = 0 | ||
| if b > cursor: | ||
| cursor = b | ||
|
|
||
| if cursor < suffix_start: | ||
| r["data"][cursor:suffix_start] = 0 |
There was a problem hiding this comment.
This whole for-loop with the many if conditions is very complicated. Would not the following do the same thing:
zle_buffer = np.zeros(samples_per_record, np.int16)
zle_buffer[:suffix_start] = 1
zle_buffer[prefix_end:] = 1
for h_i in range(
hit_offsets[r_i],
hit_offsets[r_i + 1],
):
left_hit_bound = hits[h_i]["left"] - left_extension
left_hit_bound = max(0, left_hit_bound )
right_hit_bound = hits[h_i]["right"] + right_extension
right_hit_bound = max(right_hit_bound, samples_per_record)
zle_buffer[left_hit_bound:right_hit_bound ] = 1
r["data"] *= zle_buffer
To optimize raw_records and records production in the straxen plugins DAQReader and PulseProcessor
Some modified variants of general io and pulse processing modules were required. The primary focus was reducing copy
operations. Processing speed increased overall as a result.
Most are added as _inplace variants of existing functions. The only permanent change is the lz4 decompressor where data is not repeatedly allocated anymore, as this was the function producing a vast majority of allocation.
All was already tested on AmBe runs with the detector.