Repository navigation
perf(code-index): reuse staged read and durability handles - #4792
ScriptedAlchemy wants to merge 118 commits into
Conversation
|
|
#4533 owner / sweep coordination (partial PR, |
|
#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure. The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader. Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used. The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements. Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes. |
Keep the concurrent formatting commit in the branch history. Rehome and extend its production-reader regression to cover in-place mutation, transaction poisoning, failed reads, and escaped-reader cleanup. Refs #4533
Keep the first-open file identity for the preparation even after its bounded read cache evicts the descriptor. Refuse a replaced pathname before another open, including across the cache's eviction boundary. Prove the refusal with a real nine-segment regression. Refs #4533
Exercise a real staged generation's whole and repeated range reads, verify the content digest and exact bytes, then publish and cold-decode its durable snapshot without changing the production storage format. Refs #4533
|
#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits |
Quote the pathname API in documentation and retain exact empty-buffer regressions using equality assertions required by the pinned Clippy policy. Refs #4533
|
#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head |
|
#4533 owner — production fix published at 3286852 on this PR. Appended with Changed Added seven real-filesystem regressions in Verified generated targets: Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used. This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work. |
Reuse duplicated writer descriptors for Linux durability windows instead of reopening every segment. Preserve concurrent member fsyncs and refuse missing, replaced, or mutated members, including earlier flushed windows. Add kernel-observed open-count, barrier, ownership, and failure regressions. Refs #4533
|
Published the direct GitHub source repair at 4d0ad18, appended with The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not. Added Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval. The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533. |
Validate ranges and named extents before changing staged-reader LRU state. Reuse admitted metadata across the open, retaining replacement checks. Add a real full-cache regression for cold refusals and eviction order. Refs #4533
Apply the exact rustfmt change reported by hosted Repository gates to the same-inode durability regression without changing its assertions. Refs #4533
|
#4533 owner — hosted gate repaired on the existing PR (no merge). At |
Publish missing canonical names with atomic no-replace semantics. If another publisher wins, verify byte identity before deduplicating and preserve any conflicting immutable content. Exercise the production path for conflicting, identical and absent destinations. Refs #4533
On Unix use the directory device/inode identity directly when preparing and validating a publication. Preserve existing Windows handle identity checks and reject substituted directories before commit. Add a real prepared publication regression for changed and restored directory identities. Refs #4533
|
Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:
Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work. Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it. |
Update legitimate pointer-transition fixtures to use the same staged, atomic replacement as production, preserving the strict pinned-descriptor corruption refusal. Exercise same-size/same-mtime atomic replacement and add a nonzero regression requiring typed refusal of in-place mutations with exact-byte preservation and recovery via a new inode. Refs #4533
|
Refs #4533 — authenticated GitHub owner handoff; PARTIAL, HOLD MERGE. Two exact-parent, expected-head non-force fast-forwards are published on this existing master-targeting issue PR:
Latest verified PR head: Do not merge or close #4533. Core O(distinct cold segments) still needs a packed/offset representation or equivalent bounded first-open proof, a complete daemon open-path census, same-machine managed-endpoint counts+latency, final-V2 snapshot/retirement correctness, and independent approval. Separate sweep/PR maintenance should preserve the exact head and not duplicate/merge this partial PR. Protected Kev/decision-model #4776 remains draft; release #4763 stays read-only; no Cargo, force pushes, test weakening, external agents, or infrastructure operations. |
Use one descriptor/path identity witness for warm, cold, and memo-hit validation so a concurrent atomic publication is a retryable CAS, not an erroneous corrupt-store reset. Retain symlink and same-inode mutation refusals without extra pathname opens. Exercise real inode rotations, missing path, alias refusal, restoration, and in-place mutation. Refs #4533
Restore the original method declaration after factoring the Unix pointer witness. No behavioral or test changes. Refs #4533
|
Refs #4533 — published new warm-pointer atomic-publication correctness repair on the existing master-targeting #4792 (PARTIAL; HOLD MERGE). Exact-parent non-force expected-head updates: 035f4fe from Source defect: The cached Unix pointer read (and matching-digest cold memo reuse) classified a legitimate concurrent atomic pathname rotation as reset-required corruption when the new inode arrived after its initial metadata witness. The cold path already properly returned retryable Validation: Source and regression are committed, not yet proven passing. Exact-head hosted Bazel run #42897484535 is in progress; no nonzero test, formatting, Clippy, native macOS, or independent approval claim yet. No local checkout/Bazel (GitHub checkout DNS unavailable); no Cargo was used. Keep #4792 unmerged: the O(distinct cold segments) open bound, final-V2 pack/offset or equivalent, daemon-wide open count audit, managed-endpoint before/after performance, snapshot/retention proof and independent approval are still outstanding. Do not auto-merge or close #4533. Kev/decision-model incl. #4776 remain draft, #4763 release untouched. |
Apply eight exact Rustfmt hunks from hosted Bazel run 42897484535 across retention and runtime code and tests. Preserve all original assertions, protections, production behavior, and peer changes. Refs #4533
|
Refs #4533 — published exact hosted Bazel formatting repair (partial; HOLD MERGE). Hosted run #42897484535 established Repository gates FAILED on Rustfmt, not a passing test result; all eight repository script tests passed (one executed, seven cached). The emitted output had exactly 8 formatting hunks in 5 owned retention/runtime source and regression files. Published f97c7acc as a non-force, exact-parent New exact-head hosted Bazel CI #42898538162 is queued, not yet green; nonzero current-head Rust tests, Clippy, native macOS and independent review remain unproven. Core issue acceptance still requires O(1) distinct-segment cold opens through packed/offset storage or a demonstrable equivalent, daemon-wide opens audit, final-V2 snapshot/read/retention proof, and same managed-endpoint before/after syscall counts and latency. Do not merge/auto-merge #4792 or close #4533. Kev/#4776 and release/#4763 untouched; no Cargo, force updates or external delegation. |
Refs #4533. Preserve the existing issue changes and adopt the 67 nonoverlapping master commits without rewriting branch history. Validation must rerun on the resulting merge head; this is not packed-storage acceptance or approval to merge the issue PR.
|
Refs #4533 — published master integration, not acceptance (2026-11-10). Advanced the existing issue PR to The pre-integration hosted Bazel CI run #42898538162 passed repository gates, Clippy, Linux fleet (including the code-index runtime, retention and private-fs unit targets), Linux aggregate, dashboard, feature gates, change-scope and benchmark self-tests. Its Windows job was still in progress at the last check; native macOS was skipped. These are results for HOLD MERGE / KEEP ISSUE OPEN: O(distinct sealed segments) cold first-opens remain; packed/offset storage or an equivalent proven bound, daemon-wide OS-level open counts, byte-exact snapshot/retention proof, and same managed-endpoint before/after timing are still missing. Existing protections for Kev/decision-model (including draft #4776) and read-only release #4763 remain unchanged. |
Bind non-Linux Unix fsync admission to the already-authenticated inode and pathname before and after the real barrier. A concurrent same-byte path substitution must not be published from an unrelated writer FD. Reuse the Linux identity witness without additional pathname opens; add an owning-crate Unix regression for foreign-inode substitution. Refs #4533
|
Refs #4533 — partial owning-branch durability correction, HOLD MERGE. Published Confirmed macOS/Unix correctness gap: Validation HOLD: source/test are published but no local Bazel checkout/DNS; no tests claimed passing on this head. Exact-head hosted CI run #42911326739 is pending. Native macOS is skipped in this workflow, so that branch remains unverified on macOS. Earlier Do not merge/auto-merge or close #4533. Core O(distinct segments) cold opens, packed/offset storage or equivalent, daemon-wide audited syscall counts, final-V2 snapshot/retention proof, same managed-endpoint before/after latency and independent review are still missing. Sweep/PR maintenance: preserve current head, avoid branch rewrites/duplicate PRs. Protected Kev/decision-model PRs (including draft #4776) and release #4763 remain untouched. No Cargo, external delegation, test weakening or infrastructure/release action. |
The non-Linux Unix durability batch fsyncs admitted writers inline, but its final sync was a no-op. A staged pathname replaced after an accepted writer could therefore be published without reauthentication. Retain each already-authenticated Unix pathname identity, recheck the batch before commit without extra opens or fsyncs, and cover same-byte inode substitution with positive exact-byte recovery in the owning crate. Refs #4533
|
Refs #4533 — published additional Unix durability correctness repair; PARTIAL / HOLD MERGE. Published New confirmed source gap: The immediately preceding macOS/non-Linux Unix repair authenticated each staged writer before/after its inline Code: The non-Linux Unix batch now retains the already-authenticated pathname/metadata witness after its successful inline barrier, and final Falsifiable owning-crate regression: Validation hold: new-head Bazel CI #42914141152 queued; no new-head nonzero tests, rustfmt, Clippy or independent approval claimed. The prior |
Apply the pinned formatter's method-chain wrapping in the staged pathname identity regression tests.
Keep the opened pointer descriptor through validation on Windows. Check the current pathname's inode against that descriptor and reread its bytes before a matching memo or cold parse may return. A replaced pathname is retryable CAS; in-place byte changes remain corruption. Refuse symlink slots without following them, and bound the initial read as on Unix. Exercise same-byte inode rotation and in-place mutation in owning-crate Windows regression tests. No additional pathname opens on cold reads. Refs #4533
|
Refs #4533 — published correctness repair on existing PR #4792; PARTIAL / HOLD MERGE. Published Confirmed non-Unix defect: Fix: All platforms now retain the original bounded pointer read descriptor through validation. The Windows path re-reads bytes through that descriptor, uses the existing Validation: new exact-head Bazel CI #42925008268 is queued; there is no passing new-head Rust/Windows/native macOS claim. The predecessor's repository gates passed, but Linux/Clippy/feature jobs were queued at inspection. Independent approval is absent; do not merge or close. Original #4533 acceptance still requires O(1) cold opens across distinct sealed segments (packed/offset storage or proven equivalent), full daemon open-count audit, final-V2 read/retirement proof, and measured before/after endpoint-security latency on the same managed machine. Separate #4532 PR #4796 is untouched. Keep protected Kev/decision-model PRs (including #4776) draft, and release #4763 read-only. |
Retain a verified Windows publication-pointer descriptor across warm reads, then authenticate the current pathname once using the existing same-file witness. A rotated pathname clears the memo and returns a retryable CAS; changed pinned bytes remain reset-required corruption. Authenticate persisted Windows handles before memo installation and cover repeated warm reads, same-byte inode rotation and in-place edits. Refs #4533
A Windows warm pointer memo pins the old inode across atomic publication. When its pathname witness detects a completed rotation, retire the stale memo and authenticate the new publication in the same reader call rather than leaking a transient CompareAndSwap to a reader of stable bytes. A pathname withdrawn between the initial probe and cold open remains a retryable CAS. Extend the real store regression with same-byte new-inode publication, first-read recovery, repeated warm read, and exact byte checks. Refs #4533
|
Refs #4533 — published Windows pointer-publication correctness repair on existing PR #4792 (PARTIAL; HOLD MERGE). Commit Fix: A warm Windows pointer memo retains the inode preceding an atomic publication. Previously, its correct Falsifiable regression: Extended the real publication-store test to atomically install a fresh inode holding identical pointer bytes after a prior pointer was memoized, then require the first read to succeed, the following cached read to succeed, and the stored bytes to remain exact. The existing different-byte/same-size/same-mtime atomic replacement check remains. On the preceding Windows path, the added first-read assertion would fail with Evidence boundary: Exact-head hosted CI #42938114834 has registered but remains queued. No nonzero Rust/Bazel or Windows-native pass is claimed for this commit; no local Bazel checkout is available, and the default workflow skips native macOS. The latest independent review marked the previous Windows CAS finding resolved, not the full PR approved. Keep #4533 open and #4792 unmerged until packed/offset storage or equivalent O(1) distinct-segment cold opens, daemon-wide open-count audit, final-V2 snapshot/retirement checks, same managed-endpoint before/after syscall/wall-time evidence, exact-head Bazel/native CI and independent approval are all demonstrated on master. #4796 retains separate #4532 ownership; Kev/decision-model drafts including #4776 and release #4763 remain untouched. |
Apply the pinned formatter layout reported by repository gates. Validation: owning Bazel formatter target passed; independent review completed.
Join the fixture runtime retirement before reopening SQLite for schema assertions. Capture the admission baseline on the same published runtime and retain cancellation rollback and successful reopen coverage.
Use the existing retirement authority before hiding parent files so cached descriptors cannot mask unchanged segment reads. Keep strict identity refusal and all byte-exact carry assertions. Preserve full typed problems in transport reset failures for diagnosis. Focused verification is running.
The decode waiter counter includes background owner work. Requesting completion only queues that work, so sampling its count races the later park and can attribute it to readiness. Drain the existing owner through its parked phase and retain the single background admission permit before holding the decode. Require an initial zero waiter count and preserve both readiness identity and no-new-waiter assertions.
Nov 9, 2026 — #4533 cold-pointer authority review (PARTIAL / HOLD MERGE)
Published head: e86a4e5, extending the existing issue PR by three exact-parent, non-force expected-head updates: ebb9f1d, fcbe281, and e86a4e5. Refs #4533 only.
New confirmed source defect: A cold pointer read that detected an inode/byte mismatch during its final authenticated-descriptor readback only skipped memo installation and still returned the stale parsed pointer. The Unix cold path now refuses instead. A changed/missing pathname (including a legitimate concurrent atomic pointer rotation) yields a retryable
CompareAndSwap, avoiding destructive reset of a healthy new pointer; symlink/nonfile or same-inode byte/metadata mutation yields reset-required corruption. It reuses the already opened descriptor and existing stat witnesses; no additional production open().New owning-crate regression: A real nonempty original pointer, a same-byte different-inode replacement, a missing pathname, a symlink alias resolving to the original open inode, and exact inode restoration exercise the same production final-witness function. Positive original/restored cases succeed, rotation/missing cases must return CAS, and symlink must return reset-required corruption. The restoration control refreshes legitimate Unix ctime after rename, rather than comparing the pre-rename timestamp. Source is published, but not yet proven passing on hosted Bazel.
Validation HOLD: current-head Bazel CI #42752854848 queued; no current-head nonzero test, fmt or Clippy pass, independent approval, or native macOS evidence. Architectural O(distinct cold segments) still requires packing/offset addressing or an equivalent proven bound; daemon-wide open counts, final-V2 retention/read proof, and managed-endpoint before/after measurements remain outstanding. Do not merge, auto-merge or close #4533. Kev/decision-model PRs (including #4776) stay DRAFT and unmerged; release #4763 remains read-only.
November 9, 2026 — latest #4533 owner checkpoint (PARTIAL; HOLD MERGE)
Published exact head: 20c5a356, non-force fast-forward from 53b765c with an expected-head guard; existing PR to
master, no duplicate. Refs #4533 only.New correctness repair: The active-pointer commit CAS preflight used
metadata(), following symlinks. A symlink to a matching foreign pointer or a dangling target could therefore be treated as a valid/absent pointer slot, andtempfile.persistwould replace the symlink.require_regular_pointer_slotnow usessymlink_metadata()to reject both shapes as reset-required corruption. Cold pointer authentication's final pathname witness also usessymlink_metadata()rather than following a symlink to the same opened inode. No additional pathname opens, no bypass of the existing store lock or byte comparisons.Falsifiable nonzero regression: The runtime owning-crate test exercises both existing-target and dangling symlink slots against the real commit CAS, requires typed corruption and unchanged pathname/foreign bytes, then restores an ordinary pointer file and requires exact-byte publication. The old code admits the symlink and fails the refusal assertion. The test is committed but not yet executed on the new head.
Validation HOLD: hosted Bazel CI #42747934032 queued at inspection. Earlier runs were cancelled/superseded; no new-head passing Rust, format, Clippy, or native macOS claim. Independent approval absent. The central O(distinct cold segments) file-open bound, audit of all daemon open paths, final-V2 pack/offset/read/retention proof and managed-endpoint before/after measurements remain incomplete. Do not merge, auto-merge or close #4533. Kev/decision-model (including draft #4776) and release #4763 are unchanged.
Nov 9, 2026 17:28 PT — issue owner update (Refs #4533, PARTIAL / HOLD MERGE)
Published head: 30747aaa on the existing
master-targeting issue PR. Two parent-linked nonforce expected-head updates: 46c01415 and a840b5c9, then the kernel-exclusion regression on 30747aa. No rewritten or discarded peer work and no duplicate PR.Confirmed compiler fixes: Hosted Bazel run #42726694390 failed Clippy/test compilation:
expect_erronSealedGenerationManifestMetadataV1required an unimplementedDebugbound, and a redundant closure was denied by Clippy. 46c0141 preserves both error-kind assertions while using.err().expect(...), and usesResult::okin the real retired-segment fd test. Repository gates passed on the predecessor; no current-head passing claim.New measured read-path improvement: Historical range/whole requests had still reopened
.code-generation-retention.lockon every page. The Unix retained-read path now owns one clone-shared cached lock-file descriptor, but obtains and explicitly releases a new kernel shared lock per read, serializing same-process lease ownership, verifying pathname/inode identity before and after acquisition, and refusing replaced lock authorities. A Linux inotify production-read regression requires one cold lock-file open over 17 historical reads, zero additional opens from a cloned store, and a positive real-open observer control, while existing pointer/segment exact-byte checks remain. A separate real-kernel regression verifies concurrent exclusive-writer refusal, release/reacquire, and stale-lock-inode refusal. Not a global lock bypass or a claim that distinct-segment cold opens are solved.Validation HOLD: current-head hosted Bazel #42737264260 queued. Do not merge or close until nonzero Bazel tests/Clippy and independent review pass; native macOS is skipped by the workflow. Core #4533 still requires packed/offset storage or equivalent distinct-cold-segment bound, daemon-wide open path counts, on-device managed-endpoint before/after latency, and final-V2 snapshot/retirement proof. Protected Kev/decision-model work (#4776 included) remains draft; #4763 read-only; no Cargo, Kody/Copilot, release, secrets or infrastructure actions.
Current owner checkpoint — Nov 9, 2026 (Refs #4533, partial; HOLD MERGE)
Published head: ec15c307, following 63a01e7, on the same master-targeting issue branch. Both are exact-parent non-force expected-head fast-forwards, not rewrites. No duplicate issue PR.
Confirmed CI source blockers corrected: hosted Bazel #42716161880 failed native Linux/Clippy compilation for two stale revision-reader test calls, two private sibling-test cache field accesses, and a missing eighth optional retirement argument in an isolated maintenance test. Hosted #42723182058 found one exact Rustfmt hunk in the new reset regression (all eight repository policy tests passed). 63a01e7 updates tests to the production metadata-open reader without deleting assertions; aligns the cache field visibility, supplies None for the test-only absent callback, and applies the exact rustfmt diff. ec15c30 aligns the cache element type's scheduler-private visibility to prevent a private_interfaces warning. No production behavior change, weakened test, Cargo, CI workflow or infrastructure mutation.
Validation is NOT GREEN: Current-head hosted Bazel CI #42726694390 queued at inspection. Rust tests, Clippy, format and independent approval must pass on this head, not old runs. Native macOS remains skipped. Central acceptance is still open: O(distinct cold segments) first-opens, generation lock opens per historical page, full daemon path audit, correct packed/offset-index representation or equivalent, and same managed-endpoint before/after open counts and latency. Do not merge/auto-merge or close #4533; protected Kev/decision-model #4776 stays DRAFT and release #4763 read-only.
Current ownership checkpoint — November 9, 2026 — Refs #4533 (partial; HOLD MERGE)
Published head:
61394f7, on existing issue PR targeting master. Exact remote parent9ea8a39, non-force update guarded byexpected_sha. A concurrent peer's separate retention commits were preserved; no duplicate PR, rewrite, release or protected model work.New source repair: Normal code-generation retention unlinked historical segments while the daemon publication-store's newly added eight-entry cross-request reader cache could still hold the deleted inodes. Unix therefore retained actual allocated segment bytes indefinitely on an idle daemon. The production mounted/retiring registry now provides its shared reader-cache release to the canonical maintenance executor. Under the exclusive generation/segments locks and after verifying the active pointer, the executor releases authenticated read handles before any segment unlink; a poisoned cache refuses deletion. Read-path locking, pointer/index authority and dry-run behavior remain unchanged. Applies to both mounted and registered maintenance paths. Every direct executor test caller was migrated.
New falsifiable regression: A real two-generation source publication with an old-only nonempty segment opens that historical segment through the production cached read, proves its actual Linux process descriptor pins the inode, executes normal authenticated retention with the callback, then requires one callback, closed descriptors, absent old-only pathname, no remaining process FD to its inode, refusal to serve its retired identity, and the active generation still readable. This test is committed, not yet Bazel-verified.
Hosted formatting repair: preceding CI #42706670785 passed all eight script checks but failed the two Rustfmt targets in exactly five test-source hunks. Commit
9ea8a39applies only those five emitted hunks; no production behavior or assertions were changed. The old CI's Linux/Clippy jobs were superseded, not passed.Validation HOLD: current-head hosted Bazel CI #42712609686 queued; no current-head native Rust, Linux fleet, formatter, Clippy or macOS passing claim. Independent approval absent; ordinary workflow skips macOS. The central O(distinct cold segments) issue is still not solved: packed/offset-indexed generation storage (or formally proven equivalent bound), full daemon open-path audit, snapshot/retirement/durability proof on the final representation, and same managed-endpoint before/after syscall and latency measurements remain required. Do not merge #4792 or close #4533. Keep Kev/decision-model work (including mixed draft #4776) draft and release #4763 read-only.
Current owner update — 2026-11-09 (Refs #4533; partial, DO NOT MERGE)
Published head:
b91f221on the existingmaster-targeting PR. Exact expected-headdacc5fc62f626af2334df7d40cfa206e0ab883dd, non-force fast-forward, no duplicate PR.Confirmed native Bazel evidence on predecessor
dacc5fc: hosted run #42677767864 executed and passed//crates/tracedecay-code-index-runtime:unit_test(68.9s), including the new Linux native inotify open-count/cold-snapshot/inode-substitution regression target, and//crates/tracedecay-private-fs:unit_test(0.8s); Clippy, Dashboard, change-scope and benchmark checks passed. Linux fleet was NOT green:tracedecay:dashboard_api_testandtracedecay-global-db:unit_testeach timed out at 300s. All eight repository policy tests passed, but the Rustfmt genrule failed on exactly two formatted assertions instaged_segment_reads.rs; Feature gates inherited this red.b91f221applies only those two exact hosted rustfmt hunks without changing behavior or assertions. New-head CI #42695862696 is pending; no current-head green claimed. Native macOS has been skipped.Acceptance HOLD: A bounded eight-descriptor warm cache avoids repeated page opens within one reader but still cold-opens O(distinct sealed segments). Implement packed/offset-indexed generation storage or demonstrably equivalent O(1) first opens, fully audit daemon opens (verify, restore/reseat, retention, transcripts, SQLite temporary storage, response handles), prove integrity, snapshots and retirement, and compare syscall open counts/latency on the same managed endpoint-scanned machine. Independent approval remains outstanding. No
Fixes #4533, merge/auto-merge, issue closure, Cargo, external delegation, forced update, infrastructure or protected Kev/#4776 or release/#4763 mutation.Previous owner update — 2026-11-09 (Refs #4533, partial)
Published head:
dacc5fc, appended to the existingmaster-targeting #4792 branch using an exact expected-head non-force fast-forward; no duplicate PR.New source improvement: The retained-generation
segment_readerpreviously opened a pathname for every whole/range read, even for multiple pages in the same segment. On Unix, each reader now keeps an 8-descriptor bounded LRU for its lifetime; a cache hit verifies both the path and opened inode, and post-read validation clears any corrupted/changed handles and output. Cold misses admit a validated new handle before evicting another. Non-Unix behavior is unchanged. This is a real open reduction for repeated pages, not a packed-store or O(1) distinct-segment cold-open claim.New regression: A Linux production publication + cold snapshot test uses an inotify-backed native observer to demand exactly 1 kernel pathname open for 1 whole + 16 range reads, validates all returned bytes and cold snapshot identity, controls the observer with an additional observed open, and rejects a same-byte/different-inode pathname replacement without exposing stale data or reopening it. Previous code opens that file 17 times and does not retain the inode witness. The regression is published but is not yet confirmed passing on hosted Bazel. Current-head hosted CI was queued on inspection; native macOS is skipped by the workflow.
Merge/closure hold: Independent approval, packed storage (or equivalent proof of O(1) cold first opens across distinct segments), complete daemon-path open-count audit, snapshot/retirement proof, and same-managed-endpoint before/after measurements remain outstanding. Do not merge/auto-merge #4792 or close #4533 yet. Protected Kev/decision-model including draft #4776 and release #4763 are untouched. No Cargo, Kody/Copilot, forced history, gate bypass, or infrastructure action.
Current head — owning-crate durability regression + hosted format repair (2026-11-09)
Published, Refs #4533 (partial): 33d1b0c applies exactly the two Rustfmt hunks from hosted Bazel repository gates #42649192033; it changes no assertions or production behavior. 0384da6 responds to the new independent review finding that the final-barrier regression had left its owning crate. It moves the regression into
tracedecay-private-fs's Bazelunit_test, deleting the duplicate runtime-only source. The existing test-helper fsync-latency fixture now permits a channel-controlled pause at the real sync boundary, avoiding a fragile sleep, cyclic runtime-core dependency, mocked fsync, or tracing subscriber policy bypass. The new nonzero test requires 33 real nonempty writers, a real observed fsync, same-byte/different-inode replacement after preflight, typed refusal, retained ownership, exact bytes and refusal without further fsync on retry.Exact branch head:
0384da63eacf27f7973fc891ce3dccc6ea0fcba0, basemaster, expected-head non-force fast-forward. New hosted Bazel run #42655847587 was queued on inspection; no current-head Rust, formatting or Clippy green is claimed. Native macOS skipped. The review thread has been answered but remains pending independent re-review and positive owning-crate test execution. Prior run #42649192033 failed only the two reported Rustfmt lines in the code-index durability test at its Repository gates (8 policy targets passed); do not describe it as green.No merge/auto-merge/closure: O(distinct-segment) first-opens persist, requiring real packed/offset-index storage or equivalent, all daemon-path open counts, byte-exact snapshot and retirement proofs, and measured managed endpoint-scanned before/after latency. No protected Kev, #4776, release #4763, or infrastructure changes.
Latest hosted Bazel repairs (2026-11-09) — Refs #4533
Current head:
3d12658, existing issue branch targeting master, exact-head non-force fast-forward from8e17075. New hosted Bazel run #42649192033 is queued; no current-head CI green is claimed.At the previous exact head, run #42631275844 confirmed a rustfmt failure in the latest immutable-segment source/tests and Clippy failures for four disallowed lone
tracing::subscriber::with_defaultinstalls and twoassert_is_emptyusages. It also executed 182 of 183 Linux fleet Bazel targets successfully, including noncached//crates/tracedecay-code-index-runtime:unit_test(69.5 seconds) and//crates/tracedecay-private-fs:unit_test(1.0 second); the unrelated//crates/tracedecay-global-db:unit_testtimed out after 300 seconds. Those passes belong to the prior head, not the new revision. Dashboard, benchmark self-tests, and change-scope passed; feature gate was blocked by rustfmt; native macOS was skipped. The timeout's root cause remains unproven.Commit
52b335dapplies the exact hosted rustfmt changes, preserves the existing source protections, and uses the canonicaltracedecay_runtime_core::logging::set_tracing_capturein the staged-open and immutable-collision regressions. The Linux final-barrier test was moved from the filesystem kernel crate to its actual code-index runtime caller so it can use that sanctioned capture helper without introducing a cyclic kernel dependency, unsafe scoped trace interest, suppressed Clippy policy, or a test-only production hook. It still executes real fsync on a nonempty 33-file durability batch, substitutes a distinct inode during the final barrier, and requires anInvalidDatarefusal, exact bytes, and refused no-fsync retry. The existing private-fs batch tests continue to assert 32+1 real fsync-barrier counts and ownership. Commit3d12658formats the moved test's retry assertions. No Rust dependencies, generated rules, CI workflows, or unrelated infrastructure changed. The new moved test and whole head still require nonzero Bazel verification.Merge/closure hold: The core acceptance remains incomplete: distinct sealed segments still cold-open O(segments) pathnames. Implement packed/offset-based storage or prove an equivalent bound, audit verification/restore/reseat/retention/transcripts/SQLite/response paths, prove byte-exact snapshot/read/corruption/retirement/durability, and measure before/after OS-level opens and wall time on the same managed endpoint-scanned large repository. Independent approval and native macOS are also absent. Do not merge/auto-merge #4792 or close #4533 based on partial improvements; no Kev/decision-model or release changes.
Published: staged inode authority across canonical deduplication (2026-11-09)
Commit
8e17075closes a source correctness race: a private segment inode is authenticated during staging, but the canonical-collision comparator independently opens and measures its current pathname. If the staging pathname is replaced with another same-byte/different-inode file after the earlier witness check but before comparator metadata acquisition, byte equality alone previously authorized deletion of the original staged copy. Both the existing-canonical path and atomic no-replace collision path now recheck the originally authenticated temporary inode after equality, before discarding the staging copy. The checks are metadata-only, adding zero pathname opens. The new outer immutable-path comparison tracing span measures the endpoint-authorized opens; the original inner byte-read span is preserved.A production-writer/durability-path regression stages nonempty SHA-256-addressed bytes, installs a competing identical canonical name, then uses the new production comparison span to deterministically swap the temporary inode after the initial witness check and before the comparator samples metadata. It requires a nonzero real comparison, typed refusal, intact canonical bytes and both staging file contents, retained pending ownership, and retry refusal even after the old inode is renamed back (ctime changed). The predecessor accepts and deletes the replaced pending name. This test is committed, not yet verified passing. Exact-head hosted Bazel CI is queued; macOS native is not enabled. No Cargo, no force update, no CI workflow changes.
Refs #4533, partial only. Cold first-opens remain O(distinct segments); packed/offset storage, full daemon open-path audit, managed endpoint-scanned before/after measurements, independent approval, and on-master acceptance remain outstanding. No merge/auto-merge or issue closure.
Latest source fix — canonical deduplication identity (2026-11-09)
Commit
e18d295makes canonical/pending immutable-segment equality fail closed when either pathname or opened inode changes while a potentially slow comparison runs. Previouslyfiles_equalcould report equal opened bytes after a concurrent same-byte different-inode pathname swap, then authorize removal of the private staged segment. Unix checks the original pathname/fd identity before and after comparison, without extra pathname opens. A real nonempty same-byte replacement test swaps the canonical inode inside the production comparison span, demands refusal, checks exact bytes/inode and recovery. The predecessor would accept the comparison. Refs #4533; do not merge yet: the new regression and formatting are not validated on the current head, and O(1) cold first-opens and full endpoint acceptance remain outstanding.Refs #4533
Latest source repair — failed cold open leaves healthy LRU intact (2026-11-09)
Commit
2e1b241fixes a verified source-level cache-admission gap: after a valid ninth staged segment passed pathname metadata checks, the eight-entry pinned LRU evicted its oldest healthy handle beforeFile::open. If that pathname disappeared or open was refused in the intervening syscall, the healthy descriptor was lost, and a subsequent read paid another endpoint-authorized open. The candidate now opens and authenticates before the old descriptor is evicted. The retained cache stays at eight entries with one transient candidate at admission. Existing inode, digest, corruption, output-clearing, and poison checks are unchanged.The new Linux real-filesystem regression
staged_failed_cold_open_preserves_eight_healthy_handlesstages nine different nonempty content-addressed segments, proves eight successful opens with exact bytes, and deterministically moves the ninth file at the existing production open trace event after metadata validation but before its real syscall. It requires one observed failed open attempt, cleared output, all eight healthy handles retained, and zero further opens for the original oldest segment. Restoring the same inode then permits exact ninth-segment bytes; the predecessor evicts one handle and fails this test. The assertion is nonvacuous and uses production reader/writer behavior, not a mocked opener.Validation hold: the patch is published but has not been proven passing by Bazel at this head. Current hosted Bazel CI #42620737107 was pending at inspection; native macOS was not requested by that workflow. No Cargo was used. Independent approval, full Linux/Windows/native verification, packed O(1) cold storage, daemon-wide open-count audit and same-managed-endpoint before/after timing remain missing. Refs #4533 only; do not merge this partial PR or close the issue based solely on green summary gates.
Latest published repair — staged checksum failure poisons preparation
Commit
76b7556closes a fail-closed gap: a corrupt auxiliary staged segment could be opened after mutation, fail its first whole-segment SHA-256 read, yet remain cached and leave the preparation writable when the inode metadata stayed stable during the read. The reader now invalidates the entire open preparation on a failed authenticated read, clears the returned bytes and pinned handles, and refuses repeated requests without another pathname open. Neither restoring the segment bytes nor committing a valid-looking manifest can reauthorize the transaction.The new real-filesystem regression
staged_segment_cold_digest_mismatch_poison_survives_restorationdeliberately corrupts a nonempty staged segment before the first pinned open, so the SHA-256 check rather than a changing-metadata check detects the error. It requires a positive file-open count, cleared output, zero retained handles after the first failed read, no retry opens, refusal after restoration, and failed write/commit with no installed active generation. This is a new published regression, not yet proven passing; the previous implementation retains a handle and therefore fails its zero-handles assertion.Hosted Bazel CI run #42581012282 is queued on this head. No local Bazel executable/checkout is available, and no Cargo was executed.
Scope and current state
This remains a partial code-index storage improvement. It reduces redundant pathname opens and fixes integrity/durability failures, but does not yet implement packed storage or O(1) cold first-opens across distinct segments. Keep the PR unmerged and #4533 open on this PR's summary checks alone.
The PR now targets master, not draft #4837. Integration commits
4ea334eand835bd99retain the existing branch history and adopt the exact CI configuration already merged on master, ending at02e3c8b28b8dfc73d6fcf9b4e36120256dbe01c0. The five preceding storage feature files were preserved byte-for-byte during integration. The resulting proposed diff has six storage source/test files and no CI, workflow, dependency, or infrastructure changes. No action was taken on #4837 itself.Current head:
76b7556730ce493ed871f172a2c9639f04640163.Latest source repair: final durability barriers
Commit
e9b6e8efd8d7355b10a00739e7dfdd9aa7f45184fixes a gap in LinuxDurableFileBatch::sync: earlier flushed windows were checked only before the final pending window's potentially blocking fsyncs. An earlier pathname could be replaced during that last barrier and the batch could still report success.The final sync now checks earlier identities again after the pending barriers complete. The initial fail-fast checks, error propagation, descriptor bound and failed ownership remain intact. This adds metadata checks, not pathname opens or additional fsyncs.
The new real-filesystem regression
framed_log::snapshot_tests::durable_batch_rechecks_earlier_windows_after_the_final_barrier:InvalidData, exact replacement bytes, retained batch ownership, and refusal on retry before another barrier.The preceding implementation returns success in this interleaving; the new assertion would fail it. This is a published regression, not yet an executed passing result. Existing batch and staged-reader tests are unchanged.
Existing production improvements retained
These are source-level improvements. They are not managed-endpoint wall-time measurements and do not eliminate the first open of each distinct segment.
Validation
Prior-head hosted Bazel run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42579869168 (superseded by the published checksum repair). New-head run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42581012282 . No current-head passing Rust, formatting or Clippy result is claimed here.
The prior head
b42f7ca396had successful hosted repository gates, benchmark-harness checks and Dashboard at the last inspection. Its Linux fleet was queued and Clippy was still running. An earlier owner report records 18 staged-reader tests and 33 private-filesystem unit tests passing under local Bazel on macOS at that prior head; that report is not evidence for this new Linux-only regression or the current head.Relevant generated Bazel targets remain unchanged:
Require a nonzero executed count and inspect the tested SHA. This environment has no local Bazel executable/checkout or outbound checkout DNS; no Cargo or direct Rust compiler/formatter execution was used. The available GitHub connector has workflow inspection/reruns but no native workflow-dispatch action. Native hosted macOS and current-head independent approval remain outstanding. Existing resolved inline review threads are not approving reviews.
Remaining issue acceptance
The implementation still needs the architectural packed/offset-index cutover (or a demonstrated equivalent), including every production writer, decoder, graph/lexical reader, sparse successor and retention caller. A fresh generation still has O(distinct segments) cold first-opens. Also outstanding: the complete daemon-path open-count audit (verification, restore/reseat, retention, session transcripts, SQLite temporary files and response handles), full snapshot/read/corruption/retirement/durability proofs, and before/after OS-level opens plus wall time on the same managed endpoint-scanned large repository.
Graph-specific work remains separately owned by #4796. Keep #4533 open until all acceptance is proven on master. No release, tag, production, credentials, protected Kev/decision-model changes, or external delegation.
The owner’s master synchronization subsequently moved this PR back to master at
835bd99377. Its six-file diff is confined to the original five storage paths and the added final-barrier regression. The current master cache snapshot is the second merge parent and the PR is mergeable. On this head all 18 staged-segment read tests and all 33 macOS private filesystem tests passed under Bazel. The final-barrier interleaving uses the Linux-only deferred fsync windows and awaits the hosted Linux target.