Repository navigation
Conversation
`eegnb.devices.vr.VR` is now a PsychoPy window presented through any OpenXR runtime (pyopenxr) instead of a subclass of PsychXR's LibOVR `Rift`. The class name, constructor and the methods experiments call are kept, so `use_vr=True` works as before: - `setBuffer` / `flip` / `close` / `setDefaultView`, monoscopic or per-eye stereoscopic, head-locked. - Controller input on OpenXR actions behind the `Rift` input API: `getIndexTriggerValues`, `getButtons` (continuous / pressed / released) and `updateInputState`, for `LeftTouch` / `RightTouch` through the `oculus/touch_controller` profile. `Experiment`'s start (A / X / trigger) and cancel (B / Y) prompts work unchanged. An Xbox gamepad binding is suggested but untested; the keyboard works regardless. What changes for callers: - `getTrackingState()` / `calcEyePoses()` are gone. The backend is head-locked, so head pose has no meaning; the calls are removed from `Experiment._draw` and the SSVEP drawing code. - `sync_vr_clock` stores its clock offset as `runtime_to_wallclock_offset` (was `libovr_to_wallclock_offset`), and the `_timing.csv` sidecar gains a `predicted_display_time_s` column, the runtime's predicted display time (see Timing). Rows come from `log_telemetry`, which nothing in this PR calls yet, so the column stays empty until an experiment or a marker subscriber calls it. The pixels-per-degree calculation inside `log_display_info` is also available as `pixels_per_degree`. - `VR(runtime=...)` and every experiment's `vr_runtime=...` choose the OpenXR runtime: a manifest path, or a name (`'oculus'`, `'steamvr'`, `'virtualdesktop'`, `'monado'`) matched against the installed runtimes. `None` uses the system's active runtime. A machine with several runtimes can have its active one switched by installing or opening another, and a recording then runs on a different compositor without notice; the runtime used is logged and written to `_timing.csv`. `registered_runtimes()` lists what is installed, and the VR docs page shows how to call it. - A session the runtime stops or loses mid-run raises `RuntimeError`, so the run aborts instead of carrying on with nothing on the headset. - Passthrough: `VR.set_passthrough(True/False)`, `passthrough_available` and `passthrough_on`, through `XR_FB_passthrough` (optional extension; the frame stays opaque with a warning where the runtime lacks it). `BaseExperiment` and `BlockExperiment` take `passthrough_instructions=True` to show the room behind the instruction text, so the eye adapts to the room between blocks; the experiments in the repo don't pass it on yet. Text then sits at `vr_text_distance_m` instead of at infinity; combined with an experiment's `stimulus_distance_m` it gives text and stimulus one vergence. Over Link the Meta app may need passthrough over Link turned on. - `force_high_res_timer` (the trial loop's 1 ms Windows timer) first opts the process out of Windows 11 ignoring timer requests from a process whose windows are all hidden. Without it a stimulus window that opened behind another window fell back to the 15.6 ms tick, and the runtime's frame wait paced a 120 Hz Quest 2 at 78-86 Hz. Timing: the backend records the runtime's predicted display time for each frame (`last_predicted_display_time_s`, on the `time.perf_counter` clock, and `predicted_display_time_s` in the `_timing.csv` sidecar once `log_telemetry` is called), so a slipped frame moves the prediction instead of hiding behind a constant. Measured with a photodiode on a Quest 2 over Link at 120 Hz across several sessions, with the sensor re-mounted between some of them: median marker-to-photon 20 to 25 ms per session, interquartile range 2.0 to 2.2 ms within each, flat across blocks. The offset moves between sessions, so a latency measurement still needs its own photodiode calibration. The PsychXR path stamped flip time plus a constant and drifted by a refresh period between sessions. Tested on hardware: this backend has run on a Quest 2 over Link with the Oculus runtime at 120 Hz, with controller input on both hands. Passthrough switching on and off was also checked (Oculus runtime 1.208.0; median frame 8.36 ms, 99th percentile 9.6 ms). The calibration numbers below were measured on my working branch, which has this backend plus marker subscription, the display-timing mixin and an in-session photodiode strobe bracket; this PR is the backend part of it, and reproducing them needs those follow-up PRs. A photodiode on the left lens with the headset off the head (Link, Oculus runtime, 120 Hz, an OpenBCI Cyton recording the photodiode) scored the strobe bracket from `predicted_display_time_s`: marker-to-photon median 27.7 ms (interquartile range 2.3 ms) over 20 markers at the start and 26.8 ms (1.8 ms) over 20 at the end, 0.9 ms apart; an earlier run gave 27.6 and 28.2 ms. One further run raised the stopped-session `RuntimeError` partway through with no cause found. Not yet checked on hardware: any runtime other than Oculus (SteamVR is selectable with `vr_runtime='steamvr'`), and controllers other than Touch (only the Touch and Xbox profiles have suggested bindings; the keyboard works regardless). Packaging: the PsychXR wheels and the forked psychopy pin are replaced by `psychopy~=2026.2.0`, `pyopenxr>=1.1` (not on macOS) and `PyOpenGL`. Tests: fake-xr unit tests for the window, input, callers and runtime choice, plus `tests/test_vr_monado.py`, which runs the real backend against Monado's null compositor with scripted controller input. The new VR Monado workflow runs those on `ubuntu-26.04` (Monado 25.0 from apt) under xvfb. Limits: built and timed on Windows with the Oculus runtime and a Quest 2 over Link; other runtimes and headsets are expected to work through OpenXR but are untested.
pellet
changed the title
…s finally after the recording-fix (NeuroTechX#345); telemetry-on-abort test added
…ssing Windows time conversion Three failure paths in the OpenXR backend, found in review. save_telemetry called with_name and stem on its argument, so an experiment given a string save_fn raised AttributeError in its cleanup block, after the recording had stopped but before window.close(), hiding whatever error ended the run. It now converts the argument to a Path, which covers both BaseExperiment.run and BlockExperiment.run. If anything after xr.create_instance failed, for example xr.get_system when the runtime loads but Link is not connected, the instance was never destroyed: the constructor marks the window closed until PsychoPy has built it, so the destructor never reached close(). Runtimes that allow one instance per process then refuse a retry. The constructor now destroys the OpenXR handles it created before re-raising, using the teardown it shares with close(). When the failure comes after PsychoPy has opened the desktop window, for example xr.create_session failing or the session not reaching READY because the headset is asleep, the window is now closed too. Otherwise its unprocessed messages hang the next window's event pump in the same process: pyglet drains the messages of every window on the thread, the orphan's repaint request is never cleared and keeps coming back, and the retry never reaches its own timeout. The performance counter frequency was read on every Windows runtime even when XR_KHR_win32_convert_performance_counter_time was not enabled, and every frame then called the missing conversion and raised. It is now read only when that extension is enabled, as the timespec path already required, and the predicted display time is reported as unavailable otherwise.
… each eye Monoscopic mode draws one image and copies it into both eye swapchains, but each projection layer view was submitted with that eye's own asymmetric field of view. On a Quest 2 (left eye -52/+45 degrees, right eye -45/+52) the image centre therefore landed about 8 degrees temporal in each eye, 16 degrees of divergent disparity, so the default VR N170, P300 and SSVEP runs and every monoscopic instruction screen could not be fused. PsychoPy's LibOVR backend avoided this by forcing a symmetric eye field of view whenever monoscopic was set. Submitting both layer views with one symmetric field of view is not enough: on a Quest 2 over Link the Oculus runtime (1.208.0) composites every projection view with the eye's own field of view whatever the layer says. A layer field of view of +-52 degrees, +-30 degrees or the per-eye default all produced the same picture in an adb screencap of the headset. Monoscopic frames are now drawn into a private texture at one symmetric field of view: the widest half-angle on each side across the two eyes, with the shorter axis widened so that the eye buffer, whose size is fixed before the views are known, keeps square pixels in angle. On submit each eye's swapchain image is filled from the part of that texture its own field of view covers, and the layer views keep each eye's own field of view, as in stereoscopic mode, so the result does not depend on how a runtime treats the layer field of view. The image centre now sits straight ahead of each eye. eye_fov reports the symmetric field of view the frame is drawn at, so pixels_per_degree and the optical axis offsets describe what is shown, and the offsets are zero in monoscopic mode. The same buffer now spans the wider field, so monoscopic screens get about 11% fewer pixels per degree at the centre on a Quest 2. Stereoscopic mode is unchanged. The private texture and the two copy framebuffers are now released in the shared OpenXR teardown, with the window's context made current first, so closing a window, or a construction that fails part way, no longer leaks them.
pellet
marked this pull request as ready for review
November 8, 2026 08:40
pellet
marked this pull request as draft
November 9, 2026 01:13
Every VR experiment now places its stimulus and its instruction text at one virtual distance, BaseExperiment.stimulus_distance_m, 1.0 m by default, instead of on each eye's optical axis, which is optically at infinity. Text and stimulus share one vergence with passthrough on or off, so the eyes do not re-converge when the instructions give way to the trials. The distance must be a positive, finite number of metres; setup() raises ValueError otherwise. vr_text_distance_m and the optical-axis option are removed. Monitor runs do not use any of this. N170, P300 and SSVEP are monoscopic: they draw once and the headset window copies that frame into each eye. Each copy is now shifted towards the nose so the two meet at the stimulus distance, rounded to a whole pixel. The copy keeps its size; the strip uncovered at the outer edge is filled with the window colour, or left transparent under passthrough. Stereoscopic pattern reversal places each eye's checkerboard, fixation and block text at the same per-eye positions. setup() hands the distance to the window as VR.stimulus_distance_m, which VR.__init__ sets to 1.0 so code using VR directly gets the same placement. Monoscopic and stereoscopic placement both read the window's value, and the window writes it to the "# backend" row of the _timing.csv file, as a final stimulus_distance_m pair after the existing fields, and to get_session_summary. Placement therefore cannot differ from the distance a recording states. Placement needs the eye positions the headset reports at the start of every frame. If they are missing, offsets_at_distance, nasal_shift_deg and the monoscopic shift raise RuntimeError rather than placing at infinity without saying so. Checked on a Quest 2 over Link: every stimulus and text screen moved by the expected 24 eye-buffer pixels (1.95 degrees at 68.2 mm eye separation) with identical size.
Every run now writes <recording>_session.json beside the recording, on a
monitor or a headset. It says whether the trial loop ran to its planned
end ("completed") and, if it did not, why ("reason"): "Aborted by user"
when the participant pressed the cancel key or controller button, or
Ctrl+C in the terminal, and otherwise the type and message of the
exception that ended the run. Headset runs add a "vr" section holding the
window's session summary. This follows BIDS (Brain Imaging Data
Structure): tables hold rows of data, JSON holds facts about the session.
The _timing.csv file loses its "#" rows and is now the column header plus
one row per trial. Everything those rows carried is in the summary: the
backend, runtime, headset, stimulus distance, the runtime-to-wall-clock
offset and its bracket, and the missed-frame counts. log_display_info no
longer adds an "# ipd_mm" row; the summary carries the eye separation and
pixels per degree instead.
BaseExperiment and BlockExperiment now share one cleanup, _end_run: stop
the recording, write the session file, save the timing file, close the
window. The session file is written even when the run raised. A failure
to write it is logged and swallowed, so it never hides the original
exception or stops the recording being saved and the window closed; the
original exception propagates as before. No file is written when save_fn
is None, or when the run fails or is cancelled before the recording
starts, for example when setup() rejects the distance, because there is
then no recording to describe.
On a headset, the trial loop records the eye separation when the run's
first trial starts and checks it before every later trial. If it has
moved by more than 1 mm, the run ends with a RuntimeError, so the session
file says completed false and gives the reason. The summary's
eye_separation_mm is the value the trials started with, so it is present
even when the run ended because the headset stopped.
Checked on a Quest 2 over Link and in a desktop window: N170 and pattern
reversal runs to completion, and runs ended with the cancel key. On the
headset, a pattern reversal run still completed after the headset was put
to sleep and woken on the second block's instruction screen, and one in
which the recorded eye separation was moved by 5 mm between trials ended
with completed false and the eye separation message.
load_data and fetch_dataset treat every *.csv in a session folder as a recording, so the <recording>_timing.csv file that headset runs write beside the recording was loaded as EEG data, and load_data stopped with an IndexError on it. A new helper, eegnb.is_sidecar, names the files the experiments write beside a recording, <recording>_timing.csv and <recording>_session.json, and both readers now skip them. It matches the whole ending, extension included, so a recording whose own name ends in "_session" is still loaded. Nothing else in eegnb, the examples or the command-line tool reads a data folder's files as recordings: the local data report and the zip prompt list folders only, and zipping a site keeps every file, which is wanted. Checked: fetch_dataset lists the same 94 files of the downloaded N170, P300 and SSVEP example datasets as before, and load_data on session folders from today's Quest 2 and monitor runs loads the recording alone.
The headset window's save_telemetry and log_telemetry methods collect and write the per-trial frame timing that is saved in the _timing.csv file next to each recording. They are now called save_timing and log_timing, so their names say what the file holds. The line printed after a run now reads "Saved headset frame timing to ..." instead of "Saved VR timing telemetry to ...". The file name and its columns are unchanged, and the callers and tests are updated to match.
While a stimulus is held on screen, the trial loop draws a new frame only if the experiment provides a present_soa method, and no experiment provided one. During each hold the headset therefore received no frames at all, so its runtime counted every display refresh in the hold as missed, and a pattern reversal could reach the eyes up to one frame late. The N170, P300 and pattern-reversal experiments now provide present_soa and redraw their current stimulus on every refresh while it is held. The N170 and P300 experiments remember the image they chose so they can draw it again. The pattern-reversal experiment now draws through a separate draw_frame method, so that a redraw does not send a second event marker. On a short N170 headset run with a simulated EEG device, the missed refreshes fell from 636 to 19, and the longest gap between frames fell from 1011 ms to 117 ms. Monitor runs use the same loop, so they now redraw during the hold as well, and their timing changes slightly. In a simulation of the loop at 120 Hz, each N170 image now stays on screen for 36 refreshes, which is the requested 0.3 seconds; the old loop showed it for 35. Pattern reversals that should come every 0.5 seconds now come on average every 505 ms, because the loop starts the next reversal from the moment it notices the hold has ended, and it now notices this only once per refresh. The next commit corrects that.
The trial loop used to time each trial from the moment it noticed that the previous trial had ended. Now that a held stimulus is redrawn on every display refresh, the loop checks the time only once per refresh, so it notices the end up to one refresh late, and that lateness was added to every trial that followed. In a simulation at 120 Hz, pattern reversals requested every 0.5 seconds came on average every 505 milliseconds. Each trial now starts from the previous trial's planned end, so the lateness no longer adds up and the reversals keep the requested period. The first trial of a run, and of each block, still starts from the moment the loop begins, because there is no earlier trial to count from. Two behaviours change as a result. If the loop is held up for longer than a trial, the trials that fell due in the meantime are now shown straight away, one refresh each and each with its marker, until the loop is back on schedule; before, the whole schedule moved later instead. A stimulus that runs past its planned end, such as a flicker that misses frames, now shortens the following rest period instead of pushing later trials back. A headset that stops during the trials still ends the run, and pauses remain limited to instruction screens, so taking the headset off cannot cause a burst of overdue trials. The new tests drive the real loop with a simulated clock and a display that waits for the next refresh. They check that reversals keep a 0.5 second period with no build-up of delay, that overdue trials run at once after a stall and the schedule then resumes, and that a long screen between blocks does not make the next block's trials overdue.
Collaborator
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
eegnb.devices.vr.VRis now a PsychoPy window presented through any OpenXR runtime, instead of a subclass of PsychXR'sRift.use_vr=Trueworks as before: same class name, constructor, drawing calls and Touch controller input, so the start and cancel prompts are unchanged.Breaking changes
VR(headLocked=False)raises. World-locked rendering, where the stimulus stays put as the head moves, is tracked in World-locked rendering for the OpenXR backend #347 and will follow in an upcoming pull request. Existing example experiments in the repo were already head-locked by default.getTrackingState(),calcEyePoses()and the other PsychXRRiftmethods are no longer on the window. Reading the head position comes back with World-locked rendering for the OpenXR backend #347.psychopy~=2026.2.0,pyopenxr(skipped on macOS: no OpenXR loader or runtime exists there, so the headset backend is Windows/Linux only) andPyOpenGLreplace the PsychXR wheels and the psychopy fork.New
vr_runtime=on every headset experiment picks the OpenXR runtime by name or manifest path, and the one used is logged.registered_runtimes()lists what is installed.set_passthrough(), andpassthrough_instructions=Trueto show the room behind instruction screens.RuntimeErrorand ends the run._timing.csvfile next to the recording. It holds only its header row for now: the per-marker rows need Route every event marker throughpush_marker()/subscribe_marker()#328.VR, read from the runtime's view configuration: field of view per eye, interpupillary distance, on-screen offsets and nasal shift for a stimulus at a given distance, pixels per degree, and a session summary dict (eye_fov,eye_separation_mm,offsets_at_distance,nasal_shift_deg,pixels_per_degree,get_session_summary).stimulus_distance_m) instead of at optical infinity, so the eyes converge at the same distance for instructions and trials, with passthrough on or off. Monoscopic experiments (N170, P300, SSVEP) keep their size: each eye's copy is shifted towards the nose. The distance is written to the session file (_session.json).Tested
_timing.csvmay be the only in-session check required. A run that misses frames is off by whole frames (one session at 78 Hz, before the Windows 11 timer fix in this pull request: median 18 ms, spread 9 ms), so check those rows before trusting a session.xrmodule, no headset needed.tests/test_vr_monado.pydrives the real backend against Monado's null compositor in a newvr-monado.ymlworkflow (Linux, functional only, not timing).