Skip to content

perf(code-index): reuse staged read and durability handles - #4792

Open
ScriptedAlchemy wants to merge 118 commits into
masterfrom
fix/4533-staged-segment-read-opens
Open

ScriptedAlchemy wants to merge 118 commits into
masterfrom
fix/4533-staged-segment-read-opens

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Nov 9, 2026 — #4533 cold-pointer authority review (PARTIAL / HOLD MERGE)

Published head: e86a4e5, extending the existing issue PR by three exact-parent, non-force expected-head updates: ebb9f1d, fcbe281, and e86a4e5. Refs #4533 only.

New confirmed source defect: A cold pointer read that detected an inode/byte mismatch during its final authenticated-descriptor readback only skipped memo installation and still returned the stale parsed pointer. The Unix cold path now refuses instead. A changed/missing pathname (including a legitimate concurrent atomic pointer rotation) yields a retryable CompareAndSwap, avoiding destructive reset of a healthy new pointer; symlink/nonfile or same-inode byte/metadata mutation yields reset-required corruption. It reuses the already opened descriptor and existing stat witnesses; no additional production open().

New owning-crate regression: A real nonempty original pointer, a same-byte different-inode replacement, a missing pathname, a symlink alias resolving to the original open inode, and exact inode restoration exercise the same production final-witness function. Positive original/restored cases succeed, rotation/missing cases must return CAS, and symlink must return reset-required corruption. The restoration control refreshes legitimate Unix ctime after rename, rather than comparing the pre-rename timestamp. Source is published, but not yet proven passing on hosted Bazel.

Validation HOLD: current-head Bazel CI #42752854848 queued; no current-head nonzero test, fmt or Clippy pass, independent approval, or native macOS evidence. Architectural O(distinct cold segments) still requires packing/offset addressing or an equivalent proven bound; daemon-wide open counts, final-V2 retention/read proof, and managed-endpoint before/after measurements remain outstanding. Do not merge, auto-merge or close #4533. Kev/decision-model PRs (including #4776) stay DRAFT and unmerged; release #4763 remains read-only.


November 9, 2026 — latest #4533 owner checkpoint (PARTIAL; HOLD MERGE)

Published exact head: 20c5a356, non-force fast-forward from 53b765c with an expected-head guard; existing PR to master, no duplicate. Refs #4533 only.

New correctness repair: The active-pointer commit CAS preflight used metadata(), following symlinks. A symlink to a matching foreign pointer or a dangling target could therefore be treated as a valid/absent pointer slot, and tempfile.persist would replace the symlink. require_regular_pointer_slot now uses symlink_metadata() to reject both shapes as reset-required corruption. Cold pointer authentication's final pathname witness also uses symlink_metadata() rather than following a symlink to the same opened inode. No additional pathname opens, no bypass of the existing store lock or byte comparisons.

Falsifiable nonzero regression: The runtime owning-crate test exercises both existing-target and dangling symlink slots against the real commit CAS, requires typed corruption and unchanged pathname/foreign bytes, then restores an ordinary pointer file and requires exact-byte publication. The old code admits the symlink and fails the refusal assertion. The test is committed but not yet executed on the new head.

Validation HOLD: hosted Bazel CI #42747934032 queued at inspection. Earlier runs were cancelled/superseded; no new-head passing Rust, format, Clippy, or native macOS claim. Independent approval absent. The central O(distinct cold segments) file-open bound, audit of all daemon open paths, final-V2 pack/offset/read/retention proof and managed-endpoint before/after measurements remain incomplete. Do not merge, auto-merge or close #4533. Kev/decision-model (including draft #4776) and release #4763 are unchanged.


Nov 9, 2026 17:28 PT — issue owner update (Refs #4533, PARTIAL / HOLD MERGE)

Published head: 30747aaa on the existing master-targeting issue PR. Two parent-linked nonforce expected-head updates: 46c01415 and a840b5c9, then the kernel-exclusion regression on 30747aa. No rewritten or discarded peer work and no duplicate PR.

Confirmed compiler fixes: Hosted Bazel run #42726694390 failed Clippy/test compilation: expect_err on SealedGenerationManifestMetadataV1 required an unimplemented Debug bound, and a redundant closure was denied by Clippy. 46c0141 preserves both error-kind assertions while using .err().expect(...), and uses Result::ok in the real retired-segment fd test. Repository gates passed on the predecessor; no current-head passing claim.

New measured read-path improvement: Historical range/whole requests had still reopened .code-generation-retention.lock on every page. The Unix retained-read path now owns one clone-shared cached lock-file descriptor, but obtains and explicitly releases a new kernel shared lock per read, serializing same-process lease ownership, verifying pathname/inode identity before and after acquisition, and refusing replaced lock authorities. A Linux inotify production-read regression requires one cold lock-file open over 17 historical reads, zero additional opens from a cloned store, and a positive real-open observer control, while existing pointer/segment exact-byte checks remain. A separate real-kernel regression verifies concurrent exclusive-writer refusal, release/reacquire, and stale-lock-inode refusal. Not a global lock bypass or a claim that distinct-segment cold opens are solved.

Validation HOLD: current-head hosted Bazel #42737264260 queued. Do not merge or close until nonzero Bazel tests/Clippy and independent review pass; native macOS is skipped by the workflow. Core #4533 still requires packed/offset storage or equivalent distinct-cold-segment bound, daemon-wide open path counts, on-device managed-endpoint before/after latency, and final-V2 snapshot/retirement proof. Protected Kev/decision-model work (#4776 included) remains draft; #4763 read-only; no Cargo, Kody/Copilot, release, secrets or infrastructure actions.


Current owner checkpoint — Nov 9, 2026 (Refs #4533, partial; HOLD MERGE)

Published head: ec15c307, following 63a01e7, on the same master-targeting issue branch. Both are exact-parent non-force expected-head fast-forwards, not rewrites. No duplicate issue PR.

Confirmed CI source blockers corrected: hosted Bazel #42716161880 failed native Linux/Clippy compilation for two stale revision-reader test calls, two private sibling-test cache field accesses, and a missing eighth optional retirement argument in an isolated maintenance test. Hosted #42723182058 found one exact Rustfmt hunk in the new reset regression (all eight repository policy tests passed). 63a01e7 updates tests to the production metadata-open reader without deleting assertions; aligns the cache field visibility, supplies None for the test-only absent callback, and applies the exact rustfmt diff. ec15c30 aligns the cache element type's scheduler-private visibility to prevent a private_interfaces warning. No production behavior change, weakened test, Cargo, CI workflow or infrastructure mutation.

Validation is NOT GREEN: Current-head hosted Bazel CI #42726694390 queued at inspection. Rust tests, Clippy, format and independent approval must pass on this head, not old runs. Native macOS remains skipped. Central acceptance is still open: O(distinct cold segments) first-opens, generation lock opens per historical page, full daemon path audit, correct packed/offset-index representation or equivalent, and same managed-endpoint before/after open counts and latency. Do not merge/auto-merge or close #4533; protected Kev/decision-model #4776 stays DRAFT and release #4763 read-only.


Current ownership checkpoint — November 9, 2026 — Refs #4533 (partial; HOLD MERGE)

Published head: 61394f7, on existing issue PR targeting master. Exact remote parent 9ea8a39, non-force update guarded by expected_sha. A concurrent peer's separate retention commits were preserved; no duplicate PR, rewrite, release or protected model work.

New source repair: Normal code-generation retention unlinked historical segments while the daemon publication-store's newly added eight-entry cross-request reader cache could still hold the deleted inodes. Unix therefore retained actual allocated segment bytes indefinitely on an idle daemon. The production mounted/retiring registry now provides its shared reader-cache release to the canonical maintenance executor. Under the exclusive generation/segments locks and after verifying the active pointer, the executor releases authenticated read handles before any segment unlink; a poisoned cache refuses deletion. Read-path locking, pointer/index authority and dry-run behavior remain unchanged. Applies to both mounted and registered maintenance paths. Every direct executor test caller was migrated.

New falsifiable regression: A real two-generation source publication with an old-only nonempty segment opens that historical segment through the production cached read, proves its actual Linux process descriptor pins the inode, executes normal authenticated retention with the callback, then requires one callback, closed descriptors, absent old-only pathname, no remaining process FD to its inode, refusal to serve its retired identity, and the active generation still readable. This test is committed, not yet Bazel-verified.

Hosted formatting repair: preceding CI #42706670785 passed all eight script checks but failed the two Rustfmt targets in exactly five test-source hunks. Commit 9ea8a39 applies only those five emitted hunks; no production behavior or assertions were changed. The old CI's Linux/Clippy jobs were superseded, not passed.

Validation HOLD: current-head hosted Bazel CI #42712609686 queued; no current-head native Rust, Linux fleet, formatter, Clippy or macOS passing claim. Independent approval absent; ordinary workflow skips macOS. The central O(distinct cold segments) issue is still not solved: packed/offset-indexed generation storage (or formally proven equivalent bound), full daemon open-path audit, snapshot/retirement/durability proof on the final representation, and same managed-endpoint before/after syscall and latency measurements remain required. Do not merge #4792 or close #4533. Keep Kev/decision-model work (including mixed draft #4776) draft and release #4763 read-only.


Current owner update — 2026-11-09 (Refs #4533; partial, DO NOT MERGE)

Published head: b91f221 on the existing master-targeting PR. Exact expected-head dacc5fc62f626af2334df7d40cfa206e0ab883dd, non-force fast-forward, no duplicate PR.

Confirmed native Bazel evidence on predecessor dacc5fc: hosted run #42677767864 executed and passed //crates/tracedecay-code-index-runtime:unit_test (68.9s), including the new Linux native inotify open-count/cold-snapshot/inode-substitution regression target, and //crates/tracedecay-private-fs:unit_test (0.8s); Clippy, Dashboard, change-scope and benchmark checks passed. Linux fleet was NOT green: tracedecay:dashboard_api_test and tracedecay-global-db:unit_test each timed out at 300s. All eight repository policy tests passed, but the Rustfmt genrule failed on exactly two formatted assertions in staged_segment_reads.rs; Feature gates inherited this red. b91f221 applies only those two exact hosted rustfmt hunks without changing behavior or assertions. New-head CI #42695862696 is pending; no current-head green claimed. Native macOS has been skipped.

Acceptance HOLD: A bounded eight-descriptor warm cache avoids repeated page opens within one reader but still cold-opens O(distinct sealed segments). Implement packed/offset-indexed generation storage or demonstrably equivalent O(1) first opens, fully audit daemon opens (verify, restore/reseat, retention, transcripts, SQLite temporary storage, response handles), prove integrity, snapshots and retirement, and compare syscall open counts/latency on the same managed endpoint-scanned machine. Independent approval remains outstanding. No Fixes #4533, merge/auto-merge, issue closure, Cargo, external delegation, forced update, infrastructure or protected Kev/#4776 or release/#4763 mutation.


Previous owner update — 2026-11-09 (Refs #4533, partial)

Published head: dacc5fc, appended to the existing master-targeting #4792 branch using an exact expected-head non-force fast-forward; no duplicate PR.

New source improvement: The retained-generation segment_reader previously opened a pathname for every whole/range read, even for multiple pages in the same segment. On Unix, each reader now keeps an 8-descriptor bounded LRU for its lifetime; a cache hit verifies both the path and opened inode, and post-read validation clears any corrupted/changed handles and output. Cold misses admit a validated new handle before evicting another. Non-Unix behavior is unchanged. This is a real open reduction for repeated pages, not a packed-store or O(1) distinct-segment cold-open claim.

New regression: A Linux production publication + cold snapshot test uses an inotify-backed native observer to demand exactly 1 kernel pathname open for 1 whole + 16 range reads, validates all returned bytes and cold snapshot identity, controls the observer with an additional observed open, and rejects a same-byte/different-inode pathname replacement without exposing stale data or reopening it. Previous code opens that file 17 times and does not retain the inode witness. The regression is published but is not yet confirmed passing on hosted Bazel. Current-head hosted CI was queued on inspection; native macOS is skipped by the workflow.

Merge/closure hold: Independent approval, packed storage (or equivalent proof of O(1) cold first opens across distinct segments), complete daemon-path open-count audit, snapshot/retirement proof, and same-managed-endpoint before/after measurements remain outstanding. Do not merge/auto-merge #4792 or close #4533 yet. Protected Kev/decision-model including draft #4776 and release #4763 are untouched. No Cargo, Kody/Copilot, forced history, gate bypass, or infrastructure action.

Current head — owning-crate durability regression + hosted format repair (2026-11-09)

Published, Refs #4533 (partial): 33d1b0c applies exactly the two Rustfmt hunks from hosted Bazel repository gates #42649192033; it changes no assertions or production behavior. 0384da6 responds to the new independent review finding that the final-barrier regression had left its owning crate. It moves the regression into tracedecay-private-fs's Bazel unit_test, deleting the duplicate runtime-only source. The existing test-helper fsync-latency fixture now permits a channel-controlled pause at the real sync boundary, avoiding a fragile sleep, cyclic runtime-core dependency, mocked fsync, or tracing subscriber policy bypass. The new nonzero test requires 33 real nonempty writers, a real observed fsync, same-byte/different-inode replacement after preflight, typed refusal, retained ownership, exact bytes and refusal without further fsync on retry.

Exact branch head: 0384da63eacf27f7973fc891ce3dccc6ea0fcba0, base master, expected-head non-force fast-forward. New hosted Bazel run #42655847587 was queued on inspection; no current-head Rust, formatting or Clippy green is claimed. Native macOS skipped. The review thread has been answered but remains pending independent re-review and positive owning-crate test execution. Prior run #42649192033 failed only the two reported Rustfmt lines in the code-index durability test at its Repository gates (8 policy targets passed); do not describe it as green.

No merge/auto-merge/closure: O(distinct-segment) first-opens persist, requiring real packed/offset-index storage or equivalent, all daemon-path open counts, byte-exact snapshot and retirement proofs, and measured managed endpoint-scanned before/after latency. No protected Kev, #4776, release #4763, or infrastructure changes.


Latest hosted Bazel repairs (2026-11-09) — Refs #4533

Current head: 3d12658, existing issue branch targeting master, exact-head non-force fast-forward from 8e17075. New hosted Bazel run #42649192033 is queued; no current-head CI green is claimed.

At the previous exact head, run #42631275844 confirmed a rustfmt failure in the latest immutable-segment source/tests and Clippy failures for four disallowed lone tracing::subscriber::with_default installs and two assert_is_empty usages. It also executed 182 of 183 Linux fleet Bazel targets successfully, including noncached //crates/tracedecay-code-index-runtime:unit_test (69.5 seconds) and //crates/tracedecay-private-fs:unit_test (1.0 second); the unrelated //crates/tracedecay-global-db:unit_test timed out after 300 seconds. Those passes belong to the prior head, not the new revision. Dashboard, benchmark self-tests, and change-scope passed; feature gate was blocked by rustfmt; native macOS was skipped. The timeout's root cause remains unproven.

Commit 52b335d applies the exact hosted rustfmt changes, preserves the existing source protections, and uses the canonical tracedecay_runtime_core::logging::set_tracing_capture in the staged-open and immutable-collision regressions. The Linux final-barrier test was moved from the filesystem kernel crate to its actual code-index runtime caller so it can use that sanctioned capture helper without introducing a cyclic kernel dependency, unsafe scoped trace interest, suppressed Clippy policy, or a test-only production hook. It still executes real fsync on a nonempty 33-file durability batch, substitutes a distinct inode during the final barrier, and requires an InvalidData refusal, exact bytes, and refused no-fsync retry. The existing private-fs batch tests continue to assert 32+1 real fsync-barrier counts and ownership. Commit 3d12658 formats the moved test's retry assertions. No Rust dependencies, generated rules, CI workflows, or unrelated infrastructure changed. The new moved test and whole head still require nonzero Bazel verification.

Merge/closure hold: The core acceptance remains incomplete: distinct sealed segments still cold-open O(segments) pathnames. Implement packed/offset-based storage or prove an equivalent bound, audit verification/restore/reseat/retention/transcripts/SQLite/response paths, prove byte-exact snapshot/read/corruption/retirement/durability, and measure before/after OS-level opens and wall time on the same managed endpoint-scanned large repository. Independent approval and native macOS are also absent. Do not merge/auto-merge #4792 or close #4533 based on partial improvements; no Kev/decision-model or release changes.


Published: staged inode authority across canonical deduplication (2026-11-09)

Commit 8e17075 closes a source correctness race: a private segment inode is authenticated during staging, but the canonical-collision comparator independently opens and measures its current pathname. If the staging pathname is replaced with another same-byte/different-inode file after the earlier witness check but before comparator metadata acquisition, byte equality alone previously authorized deletion of the original staged copy. Both the existing-canonical path and atomic no-replace collision path now recheck the originally authenticated temporary inode after equality, before discarding the staging copy. The checks are metadata-only, adding zero pathname opens. The new outer immutable-path comparison tracing span measures the endpoint-authorized opens; the original inner byte-read span is preserved.

A production-writer/durability-path regression stages nonempty SHA-256-addressed bytes, installs a competing identical canonical name, then uses the new production comparison span to deterministically swap the temporary inode after the initial witness check and before the comparator samples metadata. It requires a nonzero real comparison, typed refusal, intact canonical bytes and both staging file contents, retained pending ownership, and retry refusal even after the old inode is renamed back (ctime changed). The predecessor accepts and deletes the replaced pending name. This test is committed, not yet verified passing. Exact-head hosted Bazel CI is queued; macOS native is not enabled. No Cargo, no force update, no CI workflow changes.

Refs #4533, partial only. Cold first-opens remain O(distinct segments); packed/offset storage, full daemon open-path audit, managed endpoint-scanned before/after measurements, independent approval, and on-master acceptance remain outstanding. No merge/auto-merge or issue closure.

Latest source fix — canonical deduplication identity (2026-11-09)

Commit e18d295 makes canonical/pending immutable-segment equality fail closed when either pathname or opened inode changes while a potentially slow comparison runs. Previously files_equal could report equal opened bytes after a concurrent same-byte different-inode pathname swap, then authorize removal of the private staged segment. Unix checks the original pathname/fd identity before and after comparison, without extra pathname opens. A real nonempty same-byte replacement test swaps the canonical inode inside the production comparison span, demands refusal, checks exact bytes/inode and recovery. The predecessor would accept the comparison. Refs #4533; do not merge yet: the new regression and formatting are not validated on the current head, and O(1) cold first-opens and full endpoint acceptance remain outstanding.

Refs #4533

Latest source repair — failed cold open leaves healthy LRU intact (2026-11-09)

Commit 2e1b241 fixes a verified source-level cache-admission gap: after a valid ninth staged segment passed pathname metadata checks, the eight-entry pinned LRU evicted its oldest healthy handle before File::open. If that pathname disappeared or open was refused in the intervening syscall, the healthy descriptor was lost, and a subsequent read paid another endpoint-authorized open. The candidate now opens and authenticates before the old descriptor is evicted. The retained cache stays at eight entries with one transient candidate at admission. Existing inode, digest, corruption, output-clearing, and poison checks are unchanged.

The new Linux real-filesystem regression staged_failed_cold_open_preserves_eight_healthy_handles stages nine different nonempty content-addressed segments, proves eight successful opens with exact bytes, and deterministically moves the ninth file at the existing production open trace event after metadata validation but before its real syscall. It requires one observed failed open attempt, cleared output, all eight healthy handles retained, and zero further opens for the original oldest segment. Restoring the same inode then permits exact ninth-segment bytes; the predecessor evicts one handle and fails this test. The assertion is nonvacuous and uses production reader/writer behavior, not a mocked opener.

Validation hold: the patch is published but has not been proven passing by Bazel at this head. Current hosted Bazel CI #42620737107 was pending at inspection; native macOS was not requested by that workflow. No Cargo was used. Independent approval, full Linux/Windows/native verification, packed O(1) cold storage, daemon-wide open-count audit and same-managed-endpoint before/after timing remain missing. Refs #4533 only; do not merge this partial PR or close the issue based solely on green summary gates.

Latest published repair — staged checksum failure poisons preparation

Commit 76b7556 closes a fail-closed gap: a corrupt auxiliary staged segment could be opened after mutation, fail its first whole-segment SHA-256 read, yet remain cached and leave the preparation writable when the inode metadata stayed stable during the read. The reader now invalidates the entire open preparation on a failed authenticated read, clears the returned bytes and pinned handles, and refuses repeated requests without another pathname open. Neither restoring the segment bytes nor committing a valid-looking manifest can reauthorize the transaction.

The new real-filesystem regression staged_segment_cold_digest_mismatch_poison_survives_restoration deliberately corrupts a nonempty staged segment before the first pinned open, so the SHA-256 check rather than a changing-metadata check detects the error. It requires a positive file-open count, cleared output, zero retained handles after the first failed read, no retry opens, refusal after restoration, and failed write/commit with no installed active generation. This is a new published regression, not yet proven passing; the previous implementation retains a handle and therefore fails its zero-handles assertion.

Hosted Bazel CI run #42581012282 is queued on this head. No local Bazel executable/checkout is available, and no Cargo was executed.

Scope and current state

This remains a partial code-index storage improvement. It reduces redundant pathname opens and fixes integrity/durability failures, but does not yet implement packed storage or O(1) cold first-opens across distinct segments. Keep the PR unmerged and #4533 open on this PR's summary checks alone.

The PR now targets master, not draft #4837. Integration commits 4ea334e and 835bd99 retain the existing branch history and adopt the exact CI configuration already merged on master, ending at 02e3c8b28b8dfc73d6fcf9b4e36120256dbe01c0. The five preceding storage feature files were preserved byte-for-byte during integration. The resulting proposed diff has six storage source/test files and no CI, workflow, dependency, or infrastructure changes. No action was taken on #4837 itself.

Current head: 76b7556730ce493ed871f172a2c9639f04640163.

Latest source repair: final durability barriers

Commit e9b6e8efd8d7355b10a00739e7dfdd9aa7f45184 fixes a gap in Linux DurableFileBatch::sync: earlier flushed windows were checked only before the final pending window's potentially blocking fsyncs. An earlier pathname could be replaced during that last barrier and the batch could still report success.

The final sync now checks earlier identities again after the pending barriers complete. The initial fail-fast checks, error propagation, descriptor bound and failed ownership remain intact. This adds metadata checks, not pathname opens or additional fsyncs.

The new real-filesystem regression framed_log::snapshot_tests::durable_batch_rechecks_earlier_windows_after_the_final_barrier:

  • Writes 33 nonempty real members, proving the first 32 barriers occurred and an intact final sync succeeds.
  • Uses the existing production fsync tracing span to deterministically rename an equal-byte, different-inode replacement over an earlier member during the final barrier. It uses a thread-local subscriber, one pending member, no sleeps for scheduling, no mocked fsync, and no new production test hook.
  • Requires one actual observed span entry, positive durability-barrier counts, typed InvalidData, exact replacement bytes, retained batch ownership, and refusal on retry before another barrier.

The preceding implementation returns success in this interleaving; the new assertion would fail it. This is a published regression, not yet an executed passing result. Existing batch and staged-reader tests are unchanged.

Existing production improvements retained

  • Eight-entry Unix/macOS staged-reader LRU with exact request validation before cache mutation; original inode identity survives eviction, pre/post-read checks reject changes, and failures clear output.
  • Linux durability reuses at most 32 duplicated writer descriptors and concurrent bounded fsync windows instead of reopening each segment. Admission is bound to authenticated writer metadata; failed writes/admissions do not leave registered deleted temporaries.
  • Existing, newly staged and generation-evidence segments retain checked SHA-256/inode witnesses, avoiding repeated digest pathname opens without accepting substituted inodes.
  • Atomic no-replace publication for immutable generation and segment names, exact-content deduplication and collision refusals; manifest hashing reuses the writer handle.
  • Unix directory identity checks avoid redundant directory opens. Pointer validation and CAS obtain authenticated bytes from one descriptor; pointer memoization reuses the persisted tempfile descriptor and rejects symlink substitution.
  • Retained reads preserve pre/post-I/O cancellation/deadline precedence, empty failure output, whole-segment SHA-256 checks, and fresh-store snapshot reload coverage.

These are source-level improvements. They are not managed-endpoint wall-time measurements and do not eliminate the first open of each distinct segment.

Validation

Prior-head hosted Bazel run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42579869168 (superseded by the published checksum repair). New-head run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42581012282 . No current-head passing Rust, formatting or Clippy result is claimed here.

The prior head b42f7ca396 had successful hosted repository gates, benchmark-harness checks and Dashboard at the last inspection. Its Linux fleet was queued and Clippy was still running. An earlier owner report records 18 staged-reader tests and 33 private-filesystem unit tests passing under local Bazel on macOS at that prior head; that report is not evidence for this new Linux-only regression or the current head.

Relevant generated Bazel targets remain unchanged:

bazel test //crates/tracedecay-private-fs:unit_test --test_output=all --test_arg=framed_log::snapshot_tests::durable_batch_rechecks_earlier_windows_after_the_final_barrier
bazel test //crates/tracedecay-private-fs:unit_test //crates/tracedecay-private-fs:framed_log --test_output=all
bazel test //crates/tracedecay-code-index-runtime:unit_test --test_output=all --test_arg=code_index_scheduler::publication_store::staged_segment_reads
bazel build //crates/tracedecay-private-fs:rustfmt_check_2024 //crates/tracedecay-code-index-runtime:rustfmt_check_2024

Require a nonzero executed count and inspect the tested SHA. This environment has no local Bazel executable/checkout or outbound checkout DNS; no Cargo or direct Rust compiler/formatter execution was used. The available GitHub connector has workflow inspection/reruns but no native workflow-dispatch action. Native hosted macOS and current-head independent approval remain outstanding. Existing resolved inline review threads are not approving reviews.

Remaining issue acceptance

The implementation still needs the architectural packed/offset-index cutover (or a demonstrated equivalent), including every production writer, decoder, graph/lexical reader, sparse successor and retention caller. A fresh generation still has O(distinct segments) cold first-opens. Also outstanding: the complete daemon-path open-count audit (verification, restore/reseat, retention, session transcripts, SQLite temporary files and response handles), full snapshot/read/corruption/retirement/durability proofs, and before/after OS-level opens plus wall time on the same managed endpoint-scanned large repository.

Graph-specific work remains separately owned by #4796. Keep #4533 open until all acceptance is proven on master. No release, tag, production, credentials, protected Kev/decision-model changes, or external delegation.

The owner’s master synchronization subsequently moved this PR back to master at 835bd99377. Its six-file diff is confined to the original five storage paths and the added final-barrier regression. The current master cache snapshot is the second merge parent and the PR is mergeable. On this head all 18 staged-segment read tests and all 33 macOS private filesystem tests passed under Bazel. The final-barrier interleaving uses the Linux-only deferred fsync windows and awaits the hosted Linux target.

@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 7e83286

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review November 8, 2026 00:55

Copy link
Copy Markdown
Owner Author

#4533 owner / sweep coordination (partial PR, Refs #4533): Please do not merge or enable auto-merge based solely on green workflow gates. This head needs a nonzero Bazel Rust regression run, relevant native hosted checks and an independent correctness review first. The full packed-generation storage / large-repository endpoint-security acceptance remains separate, so #4533 must stay open even if this reviewed partial lands. PR #4796 owns the related graph-pass path; avoid overlapping edits. No release or Kev/decision-model changes are authorized in this lane.

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure.

The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader.

Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used.

The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements.

Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes.

Keep the concurrent formatting commit in the branch history. Rehome and
extend its production-reader regression to cover in-place mutation,
transaction poisoning, failed reads, and escaped-reader cleanup.

Refs #4533
Keep the first-open file identity for the preparation even after its
bounded read cache evicts the descriptor. Refuse a replaced pathname
before another open, including across the cache's eviction boundary.
Prove the refusal with a real nine-segment regression.

Refs #4533
Exercise a real staged generation's whole and repeated range reads,
verify the content digest and exact bytes, then publish and cold-decode
its durable snapshot without changing the production storage format.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits 262e340 (original pin identity, pre-open failure output hygiene, mutation/commit tests), b850dcb (persist first-open identity across LRU eviction + poisoned no-reopen regression), 18269c6 (real staged whole/range bytes through successful commit and cold snapshot reload regression) are now on fix/4533-staged-segment-read-opens. Tests are added but not yet executed; hosted Bazel CI for the current tip is pending, macOS native CI is not running by default, and independent review is still missing. Please do not merge/auto-merge this partial PR solely on default green CI. Fresh audit finding: stage_segment still creates one file per new segment; verify_prepared_segments invokes state_digest_file for each distinct segment (another per-file open); on Linux, DurableFileBatch::sync calls sync_file_at for every written member, which uses OpenOptions::open again per file. The full #4533 packed-storage cutover must remove those O(segments) first opens and prove durability/retention as well as repeated read behavior. PR #4796 independently owns graph-read instrumentation; do not merge overlapping change here. Refs #4533; keep issue open. No release, production, Kev/decision-model or restricted actions.

Quote the pathname API in documentation and retain exact empty-buffer
regressions using equality assertions required by the pinned Clippy policy.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head 103101b6 and fixed its actionable Clippy failures in commit c76238b, appended non-force with expected remote head to the existing issue-specific PR. The blocking Rust Clippy errors were doc_markdown (unquoted Handle::from_path in publication_store.rs:2162) and two assert_is_empty findings in the staged-reader regressions (staged_segment_reads.rs:240,334). Both equality assertions continue proving the output buffer is exactly empty; no test was removed, skipped or weakened. The earlier Linux fleet failure (UNAVAILABLE: io exception building tracedecay-daemon-service) is a separate remote execution failure, not evidence of a source fix. New hosted run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42441637324 — not yet green at this writing. Native macOS is skipped in default CI; independent review and nonzero focused Bazel execution remain required. The full O(1)-first-open packed-storage cutover, before/after macOS endpoint-security measurements and index/restore/snapshot acceptance remain open. No duplicates, merging, release, Kev work or infrastructure actions. Keep #4533 open and do not auto-merge #4792 on CI green alone; coordinate with #4796 graph reader work.

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

#4533 owner — production fix published at 3286852 on this PR. Appended with force=false and expected remote head c76238bd05e9eb356ee4963010f1036f62f88ba4; concurrent Clippy repairs are preserved. No duplicate PR or merge.

Changed crates/tracedecay-private-fs/src/framed_log.rs: Linux durability batches retain at most 32 duplicated writer descriptors, flush full windows concurrently, and use those descriptors rather than reopening every pathname. Earlier windows retain original metadata so missing/replaced/modified members still refuse final sync. Pending path/descriptor identity is checked before and after fsync. A failed full window retains its handles and cannot be marked successful or admit another descriptor. Non-Linux durability, existing integration tests, checksums, format and publication ordering are unchanged.

Added seven real-filesystem regressions in src/framed_log_batch_tests.rs, wired into the existing Bazel unit target: positive-control kernel IN_OPEN counts across 67 files, bounded descriptors, exact byte readback, nonzero early/final barrier counts, mismatched writer/path refusal, pending/early-flushed replacement, missing/modified early files, and failed-window ownership/retry. Expected open count is 67 writer opens with no durability reopens, NOT an executed result. Existing member-only and concurrent slow-fsync integration tests remain intact.

Verified generated targets: //crates/tracedecay-private-fs:unit_test, :framed_log, and :rustfmt_check_2024; no BUILD/dependency or workflow changes. Root diff reviewed against the exact parent: changes are limited to the batch implementation/imports/test wiring and the new tests. #4796's graph-reader lane and existing staged-read code were not overwritten.

Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used.

This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work.

Reuse duplicated writer descriptors for Linux durability windows instead
of reopening every segment. Preserve concurrent member fsyncs and refuse
missing, replaced, or mutated members, including earlier flushed windows.

Add kernel-observed open-count, barrier, ownership, and failure regressions.
Refs #4533

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

Published the direct GitHub source repair at 4d0ad18, appended with force=false and expected remote head 32868523be33a85fb091feebc1d2b11c1b346143. The concurrent framed_log.rs durability commit and prior Clippy fixes are preserved. No duplicate PR or merge.

The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not.

Added staged_segment_rejected_reads_preserve_full_cache_and_recency using the real preparation/writer/reader: nine staged files, eight positively exercised cached handles, overflow/out-of-bounds/wrong-size/missing-path refusals, empty output and unchanged cache on each refusal, then exact bytes and open counts 9/9/10 to prove eviction order. All five pre-existing staged-reader tests are unchanged; no source-shape acceptance or weakened assertions.

Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval.

The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533.

@ScriptedAlchemy ScriptedAlchemy changed the title perf(code-index): pin staged segment reads within preparation perf(code-index): reuse staged read and durability handles Nov 8, 2026
Validate ranges and named extents before changing staged-reader LRU state.
Reuse admitted metadata across the open, retaining replacement checks.
Add a real full-cache regression for cold refusals and eviction order.

Refs #4533
Apply the exact rustfmt change reported by hosted Repository gates to the
same-inode durability regression without changing its assertions.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner — hosted gate repaired on the existing PR (no merge). At 4d0ad18, Repository gates failed its Bazel rustfmt_check_2024 on one line in framed_log_batch_tests.rs:214. I committed the exact required formatting correction as 688f3a6 and advanced fix/4533-staged-segment-read-opens by expected-head non-force update. No test assertion or production semantics changed in this repair. New-head hosted CI is pending; macOS is skipped in the standard workflow and no independent review/nonzero Rust test result is established. Maintenance sweeps: please preserve the no-merge/auto-merge hold. Refs #4533 remains partial: packed storage, broad open audit and endpoint-scanned before/after verification are unfinished. No changes to #4796, releases, infrastructure or Kev.

Publish missing canonical names with atomic no-replace semantics. If another
publisher wins, verify byte identity before deduplicating and preserve any
conflicting immutable content. Exercise the production path for conflicting,
identical and absent destinations.

Refs #4533
On Unix use the directory device/inode identity directly when preparing and
validating a publication. Preserve existing Windows handle identity checks
and reject substituted directories before commit. Add a real prepared
publication regression for changed and restored directory identities.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:

  • 70188c58: atomic no-replace for staged immutable segment names; verified collision/dedupe/absence paths in a new real-filesystem regression. Prevents a concurrent publisher from being overwritten by a stale missing-path observation.
  • 7184d12e: on Unix/macOS replace six per-generation directory Handle::from_path opens (three on prepare and three on commit validation) with dev/inode metadata identity comparisons; preserve Windows checks. Added a real staged-publication substitution/restore regression exercising compare-and-swap refusal.

Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work.

Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it.

Update legitimate pointer-transition fixtures to use the same staged,
atomic replacement as production, preserving the strict pinned-descriptor
corruption refusal. Exercise same-size/same-mtime atomic replacement and
add a nonzero regression requiring typed refusal of in-place mutations
with exact-byte preservation and recovery via a new inode.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — authenticated GitHub owner handoff; PARTIAL, HOLD MERGE.

Two exact-parent, expected-head non-force fast-forwards are published on this existing master-targeting issue PR:

  • 91764164 fixes the confirmed hosted Bazel #42777818917 Rustfmt (three hunks) and Clippy (cached lock Drop) failures and restores the retention census's existing typed Storage refusal for directories, while preserving UnsafeState on symlink substitution.
  • 333fa750 updates the owning runtime test fixtures to model atomic inode replacement rather than an illicit in-place rewrite of a pinned publication pointer. The old Linux runtime suite had six failures, several with the expected active code-generation pointer bytes changed while pinned refusal. Added a separate nonzero Unix regression requiring typed refusal and exact-byte preservation for same-inode, same-size, same-mtime mutation, then correct read after atomic replacement. Assertions and production fail-closed behavior are preserved.

Latest verified PR head: 333fa7505979bc3e44bbfb5975ccb622e390883f. Current-head Bazel CI #42895965019 is running, not green. The sixth Linux runtime failure concerning root readiness/active decode waiter accounting has not been proven corrected. No local Bazel checkout was available; do not claim unexecuted regressions pass, and keep native macOS marked unverified.

Do not merge or close #4533. Core O(distinct cold segments) still needs a packed/offset representation or equivalent bounded first-open proof, a complete daemon open-path census, same-machine managed-endpoint counts+latency, final-V2 snapshot/retirement correctness, and independent approval. Separate sweep/PR maintenance should preserve the exact head and not duplicate/merge this partial PR. Protected Kev/decision-model #4776 remains draft; release #4763 stays read-only; no Cargo, force pushes, test weakening, external agents, or infrastructure operations.

Use one descriptor/path identity witness for warm, cold, and memo-hit
validation so a concurrent atomic publication is a retryable CAS, not
an erroneous corrupt-store reset. Retain symlink and same-inode mutation
refusals without extra pathname opens. Exercise real inode rotations,
missing path, alias refusal, restoration, and in-place mutation.

Refs #4533
Restore the original method declaration after factoring the Unix
pointer witness. No behavioral or test changes.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — published new warm-pointer atomic-publication correctness repair on the existing master-targeting #4792 (PARTIAL; HOLD MERGE).

Exact-parent non-force expected-head updates: 035f4fe from 333fa750, followed by d235689 correcting a method declaration; published ref checked. No duplicate PR or rewritten peer commits.

Source defect: The cached Unix pointer read (and matching-digest cold memo reuse) classified a legitimate concurrent atomic pathname rotation as reset-required corruption when the new inode arrived after its initial metadata witness. The cold path already properly returned retryable CompareAndSwap. Both paths now use the same production inode/path witness as cold reads. Path disappearance/replacement is CAS, while symlinks and same-inode modifications remain reset-required corruption; no additional open() or bypassed locking. A real-filesystem owning-crate regression exercises an unchanged nonempty pointer, same-byte different-inode atomic rotation, absent pathname, same-inode symlink alias, restoration with refreshed ctime and an in-place mutation, including typed denial and byte preservation.

Validation: Source and regression are committed, not yet proven passing. Exact-head hosted Bazel run #42897484535 is in progress; no nonzero test, formatting, Clippy, native macOS, or independent approval claim yet. No local checkout/Bazel (GitHub checkout DNS unavailable); no Cargo was used. Keep #4792 unmerged: the O(distinct cold segments) open bound, final-V2 pack/offset or equivalent, daemon-wide open count audit, managed-endpoint before/after performance, snapshot/retention proof and independent approval are still outstanding. Do not auto-merge or close #4533. Kev/decision-model incl. #4776 remain draft, #4763 release untouched.

Apply eight exact Rustfmt hunks from hosted Bazel run 42897484535
across retention and runtime code and tests. Preserve all original
assertions, protections, production behavior, and peer changes.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — published exact hosted Bazel formatting repair (partial; HOLD MERGE).

Hosted run #42897484535 established Repository gates FAILED on Rustfmt, not a passing test result; all eight repository script tests passed (one executed, seven cached). The emitted output had exactly 8 formatting hunks in 5 owned retention/runtime source and regression files. Published f97c7acc as a non-force, exact-parent expected_sha fast-forward from d235689. It applies those eight hunks verbatim, with no assertion, production behavior, workflow, dependency or safeguard change. Current branch head verified at this SHA.

New exact-head hosted Bazel CI #42898538162 is queued, not yet green; nonzero current-head Rust tests, Clippy, native macOS and independent review remain unproven. Core issue acceptance still requires O(1) distinct-segment cold opens through packed/offset storage or a demonstrable equivalent, daemon-wide opens audit, final-V2 snapshot/read/retention proof, and same managed-endpoint before/after syscall counts and latency. Do not merge/auto-merge #4792 or close #4533. Kev/#4776 and release/#4763 untouched; no Cargo, force updates or external delegation.

Refs #4533. Preserve the existing issue changes and adopt the 67
nonoverlapping master commits without rewriting branch history.
Validation must rerun on the resulting merge head; this is not
packed-storage acceptance or approval to merge the issue PR.

Copy link
Copy Markdown
Owner Author

Refs #4533 — published master integration, not acceptance (2026-11-10).

Advanced the existing issue PR to c929b03c, a two-parent merge of exact prior remote head f97c7acc and exact master 6d2797ee, using expected_sha=f97c7acc and force=false. The 67 intervening master commits touched 77 paths with zero overlap against this PR's 20 storage/test paths; the merge tree retains all 20 issue-path blobs verbatim from the prior head and adopts current master everywhere else. Post-publication compare confirmed ahead 103 / behind 0, exactly 20 issue-path changes. No duplicate PR, discarded peer work, history rewrite, protected decision-model PR merge, release, or infrastructure action.

The pre-integration hosted Bazel CI run #42898538162 passed repository gates, Clippy, Linux fleet (including the code-index runtime, retention and private-fs unit targets), Linux aggregate, dashboard, feature gates, change-scope and benchmark self-tests. Its Windows job was still in progress at the last check; native macOS was skipped. These are results for f97c7acc, not a green claim for c929b03c; no new-head workflow had registered when inspected. Preserve new-head native verification and independent review before merging.

HOLD MERGE / KEEP ISSUE OPEN: O(distinct sealed segments) cold first-opens remain; packed/offset storage or an equivalent proven bound, daemon-wide OS-level open counts, byte-exact snapshot/retention proof, and same managed-endpoint before/after timing are still missing. Existing protections for Kev/decision-model (including draft #4776) and read-only release #4763 remain unchanged.

Bind non-Linux Unix fsync admission to the already-authenticated inode
and pathname before and after the real barrier. A concurrent same-byte
path substitution must not be published from an unrelated writer FD.

Reuse the Linux identity witness without additional pathname opens;
add an owning-crate Unix regression for foreign-inode substitution.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — partial owning-branch durability correction, HOLD MERGE.

Published 3029dc06 as an exact-parent, non-force expected-head fast-forward from c929b03c, after the prior merge had integrated current master without overlapping #4533 source paths. Remote PR head verified at the new commit; no peers discarded or duplicate PR.

Confirmed macOS/Unix correctness gap: DurableFileBatch::written_with_identity previously ignored the supplied authenticated metadata outside Linux and fsynced the passed writer descriptor without checking whether its pathname still named that inode. A same-byte foreign-inode replacement could therefore be accepted as an authorized staged writer. The non-Linux Unix arm now reuses the existing metadata/inode/ctime witness before and after the real fsync, validating both descriptor and directory entry, with zero additional pathname opens. Windows behavior is unchanged (requires separate platform-appropriate identity treatment). Added an owning-crate Unix real-filesystem regression that replaces the authenticated writer's pathname with same-byte distinct-inode content and requires InvalidData, retaining the replacement bytes.

Validation HOLD: source/test are published but no local Bazel checkout/DNS; no tests claimed passing on this head. Exact-head hosted CI run #42911326739 is pending. Native macOS is skipped in this workflow, so that branch remains unverified on macOS. Earlier c929b03c repository gates/Clippy/dashboard passed but Linux fleet was still executing at inspection, not full green. The only unresolved review thread is the older cancellation-order finding; its source fix is present but independent resolution/approval outstanding.

Do not merge/auto-merge or close #4533. Core O(distinct segments) cold opens, packed/offset storage or equivalent, daemon-wide audited syscall counts, final-V2 snapshot/retention proof, same managed-endpoint before/after latency and independent review are still missing. Sweep/PR maintenance: preserve current head, avoid branch rewrites/duplicate PRs. Protected Kev/decision-model PRs (including draft #4776) and release #4763 remain untouched. No Cargo, external delegation, test weakening or infrastructure/release action.

The non-Linux Unix durability batch fsyncs admitted writers inline,
but its final sync was a no-op. A staged pathname replaced after an
accepted writer could therefore be published without reauthentication.

Retain each already-authenticated Unix pathname identity, recheck the
batch before commit without extra opens or fsyncs, and cover same-byte
inode substitution with positive exact-byte recovery in the owning crate.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — published additional Unix durability correctness repair; PARTIAL / HOLD MERGE.

Published 9cd370f1 on the existing issue PR by an exact-parent, non-force expected-head update from 3029dc06. Verified GitHub's branch ref and PR head now both resolve to the new commit; no rewrite, duplicate PR or peer commits discarded.

New confirmed source gap: The immediately preceding macOS/non-Linux Unix repair authenticated each staged writer before/after its inline fsync, but DurableFileBatch::sync() remained a no-op off Linux. A different inode could replace an already admitted pathname before the final generation commit, and the old successful fsync would be treated as authority for the substituted file. This is distinct from a replacement before admission.

Code: The non-Linux Unix batch now retains the already-authenticated pathname/metadata witness after its successful inline barrier, and final sync() refuses replaced, missing or modified admitted members via symlink_metadata. This does not add any production pathname open() or extra fsync(), does not change the Linux bounded-window algorithm, and leaves Windows behavior unchanged.

Falsifiable owning-crate regression: staged_durability_refuses_replacement_after_successful_admission first authenticates a real nonempty writer and requires unchanged sync success. It then atomically substitutes a different real inode holding identical bytes, requires InvalidData on final sync and exact preservation of replacement bytes, and verifies successful fresh-batch reauthentication and readback. The predecessor's macOS arm returns Ok(()) and fails the denial assertion. The test also runs on Linux, but native macOS CI is skipped.

Validation hold: new-head Bazel CI #42914141152 queued; no new-head nonzero tests, rustfmt, Clippy or independent approval claimed. The prior f97c7acc Linux/Bazel gates passed, but they are not validation of this head. Core acceptance still requires O(1) distinct cold-segment opens through pack/offset indexing (or proven equivalent), complete daemon open-path census, final-V2 snapshot/read/retention proof, and same managed Defender/CorpLink machine before/after syscall counts and latency. Do not merge/auto-merge #4792 or close #4533; related #4796 owns its separate #4532 scope. Kev/decision-model work including #4776 stays draft/unmerged; #4763 release untouched. No Cargo, workflow bypass, delegated work or infrastructure mutation.

Apply the pinned formatter's method-chain wrapping in the staged
pathname identity regression tests.
Keep the opened pointer descriptor through validation on Windows. Check the
current pathname's inode against that descriptor and reread its bytes
before a matching memo or cold parse may return. A replaced pathname is
retryable CAS; in-place byte changes remain corruption. Refuse symlink
slots without following them, and bound the initial read as on Unix.

Exercise same-byte inode rotation and in-place mutation in owning-crate
Windows regression tests. No additional pathname opens on cold reads.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — published correctness repair on existing PR #4792; PARTIAL / HOLD MERGE.

Published 29f3a3e as an exact-parent, expected_sha=662a2586, non-force fast-forward. GitHub's PR head and branch ref both now confirm this SHA; no peer commits were replaced, no duplicate PR was opened.

Confirmed non-Unix defect: read_publication_pointer_with_bytes on Windows could return a memoized pointer after a same-byte pathname rotation without checking the opened inode; a cold parse could also skip memo installation on failed final byte readback yet return the already-parsed, stale pointer. Its pre-read metadata() also followed symlinks. This differed from the Unix fail-closed pointer authority.

Fix: All platforms now retain the original bounded pointer read descriptor through validation. The Windows path re-reads bytes through that descriptor, uses the existing same_file::Handle identity primitive to compare the descriptor with the current pathname, refuses symlink/nonfile slots, returns retryable CAS for an absent or replaced name, and requires corruption reset on in-place byte mutation. The cold path retains two pathname opens, not an extra third open; Windows warm memo hits now require one additional identity open for correctness (performance cost still needs measurement). Added owning-crate, Windows-gated real-file regressions for same-byte different-inode rotation and same-inode rewrite. No legacy API, workflow or protections changed.

Validation: new exact-head Bazel CI #42925008268 is queued; there is no passing new-head Rust/Windows/native macOS claim. The predecessor's repository gates passed, but Linux/Clippy/feature jobs were queued at inspection. Independent approval is absent; do not merge or close. Original #4533 acceptance still requires O(1) cold opens across distinct sealed segments (packed/offset storage or proven equivalent), full daemon open-count audit, final-V2 read/retirement proof, and measured before/after endpoint-security latency on the same managed machine. Separate #4532 PR #4796 is untouched. Keep protected Kev/decision-model PRs (including #4776) draft, and release #4763 read-only.

devin-ai-integration[bot]

This comment was marked as resolved.

Retain a verified Windows publication-pointer descriptor across warm
reads, then authenticate the current pathname once using the existing
same-file witness. A rotated pathname clears the memo and returns a
retryable CAS; changed pinned bytes remain reset-required corruption.

Authenticate persisted Windows handles before memo installation and
cover repeated warm reads, same-byte inode rotation and in-place edits.

Refs #4533
devin-ai-integration[bot]

This comment was marked as resolved.

A Windows warm pointer memo pins the old inode across atomic publication.
When its pathname witness detects a completed rotation, retire the stale
memo and authenticate the new publication in the same reader call rather
than leaking a transient CompareAndSwap to a reader of stable bytes.

A pathname withdrawn between the initial probe and cold open remains a
retryable CAS. Extend the real store regression with same-byte new-inode
publication, first-read recovery, repeated warm read, and exact byte checks.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — published Windows pointer-publication correctness repair on existing PR #4792 (PARTIAL; HOLD MERGE).

Commit 1a08e5b is a single-parent, expected-head (6ac133d), non-force fast-forward; GitHub confirms the PR head and only two changed files in this commit. No parallel branch, history rewrite or protected work touched.

Fix: A warm Windows pointer memo retains the inode preceding an atomic publication. Previously, its correct CompareAndSwap identity mismatch escaped to an ordinary reader even when the new pathname already named a fully committed, stable successor. The read now discards that stale memo and authenticates the new pathname using the existing cold validation within the same call. If the pathname disappears between the initial metadata probe and cold open, the operation still returns retryable CAS. In-place mutation, nonregular files, digest/extent validation and final same-file checks remain fail-closed. This changes no filesystem format, read limit, CI policy or production pathname opens on a normal warm read.

Falsifiable regression: Extended the real publication-store test to atomically install a fresh inode holding identical pointer bytes after a prior pointer was memoized, then require the first read to succeed, the following cached read to succeed, and the stored bytes to remain exact. The existing different-byte/same-size/same-mtime atomic replacement check remains. On the preceding Windows path, the added first-read assertion would fail with CompareAndSwap.

Evidence boundary: Exact-head hosted CI #42938114834 has registered but remains queued. No nonzero Rust/Bazel or Windows-native pass is claimed for this commit; no local Bazel checkout is available, and the default workflow skips native macOS. The latest independent review marked the previous Windows CAS finding resolved, not the full PR approved.

Keep #4533 open and #4792 unmerged until packed/offset storage or equivalent O(1) distinct-segment cold opens, daemon-wide open-count audit, final-V2 snapshot/retirement checks, same managed-endpoint before/after syscall/wall-time evidence, exact-head Bazel/native CI and independent approval are all demonstrated on master. #4796 retains separate #4532 ownership; Kev/decision-model drafts including #4776 and release #4763 remain untouched.

Apply the pinned formatter layout reported by repository gates.

Validation: owning Bazel formatter target passed; independent review completed.
Join the fixture runtime retirement before reopening SQLite for schema
assertions. Capture the admission baseline on the same published runtime
and retain cancellation rollback and successful reopen coverage.
Use the existing retirement authority before hiding parent files so cached descriptors cannot mask unchanged segment reads. Keep strict identity refusal and all byte-exact carry assertions. Preserve full typed problems in transport reset failures for diagnosis. Focused verification is running.
The decode waiter counter includes background owner work. Requesting completion only queues that work, so sampling its count races the later park and can attribute it to readiness. Drain the existing owner through its parked phase and retain the single background admission permit before holding the decode. Require an initial zero waiter count and preserve both readiness identity and no-new-waiter assertions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

code-index: store and read the index in few files so endpoint security open() cost does not dominate

1 participant