Repository navigation
feat(code-index): index authenticated Git submodules - #4801
ScriptedAlchemy wants to merge 60 commits into
Conversation
Keep parent repository source admission unchanged; use one shared walker for exclusion evidence instead of a second filesystem scan. Refs #4593
|
|
Refs #4593 — validation blocker at exact head The repository-gates job failed before Rust execution: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42433144833/job/128586224987 . The actual failures are:
No source fix, Rust test execution, independent review, branch update, merge or issue closure was completed in this follow-up. This environment has no executable repository checkout or Bazel; direct repository retrieval failed, and no usable remote execution/workflow-dispatch capability was exposed. This is a validation limitation, not evidence that the implementation is correct. #4801 remains diagnostic-only and draft. It does not yet satisfy authenticated submodule admission, pinned child-tree capture, child-aware freshness, or consistent retrieval/graph behavior. Retain Coordination: the dedicated #4593 owner remains responsible for implementation; PR maintenance owns CI assistance without racing its branch. Do not repair unrelated decision-model generated paths by importing forbidden work. #4776 and all Kev/decision-model work remain draft and unmergeable by instruction; #4763 remains read-only. No protections or tests were weakened. |
|
Refs #4593 — exact-head CI repair The PR is still partial: it adds skipped-nested-repository diagnostics and regression source, but NOT authenticated submodule indexing, freshness, or indexed retrieval. HOLD MERGE even if its current-head diagnostics tests pass: keep this same PR open to complete #4593. Being reviewable or passing preliminary tests does not mean the issue's acceptance is met. No complete-implementation, native macOS or independent-review proof exists. Keep |
Limit skipped-repository samples during traversal, honor directory policy before collecting diagnostics, and quote bounded path labels. Propagate source-walk failures instead of returning an empty successful audit. Add behavioral coverage for bounded sampling, policy scope, preserved ecosystem caveats, Unicode/control-character paths, and missing roots. Refs #4593
|
Refs #4593 — published source repair Fixed unbounded skipped-repository collection, diagnostics escaping configured directory exclusions, unquoted/unbounded path labels, and swallowed filesystem-walk failures. Added six behavioral regression cases (one Unix-only filename case); existing initialized-submodule and linked/nested-repository exclusions are preserved. Only Current-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442123433 — queued when inspected. These Rust regressions have NOT executed; author diff review is NOT independent review. The prior successful draft run skipped Rust/Clippy/native jobs, so it is not acceptance evidence for these changes. The existing merge hold remains. This repairs the current audit implementation, not the still-unimplemented optional authenticated submodule indexing feature. Keep the PR draft/open and #4593 open. Continue the integrated admission/capture/freshness/retrieval implementation here rather than opening a duplicate or treating more diagnostic fixes as feature completion. |
|
Refs #4593 — published nonforce, expected-head fast-forward |
Use one source boundary for interactive walks, Git candidate classification, worktree capture (including ignored-file admissions), and freshness. Parent tracked files hidden by a new nested .git marker become tombstones, invalidate clean HEAD-tree reuse, and return when the marker disappears. Reject source paths passing through symlinked directories or lexical escapes. Add real Git classification and filesystem regressions for nested directory, gitdir-file, symlink, path traversal and removal behavior. Refs #4593
|
Refs #4593 — published source repair |
|
Refs #4593 — published exact-head CI repair |
Sample .git marker evidence for tracked and explicitly admitted source ancestors even when Git dirwalk never recurses into tracked-only folders. Test marker creation and removal against a real Git parent index. Refs #4593
Directory metadata changes when the marker is removed, so the original cached roster must not be treated as restored merely by marker absence. Refs #4593
A registered gitlink remains an untrusted nested source boundary even if its child .git marker is absent. Keep live grep and audit in default-deny parity with the parent index; include/glob cannot lift repository trust. Exercise a real registered Git submodule with its marker removed. Refs #4593
|
Refs #4593 — published narrow default-deny source fix Current-head run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42466514013 (queued at check). NO Bazel behavioral Rust pass or independent review claim. Do not merge or mark ready. Feature acceptance remains integrated authenticated opt-in, pinned historical child capture, child status/freshness and cross-boundary graph/index/live retrieval, with nonzero Bazel/native validation. Preserve #4800 and protected Kev/release lanes. |
Apply the two exact rustfmt_check_2024 hunks from hosted run 42466514013. No source behavior changes; authenticated submodule indexing remains pending. Refs #4593
|
Refs #4593 — narrow hosted Bazel formatting repair published at Exact-head run 42466514013, repository-gates job 128687191524, failed only HOLD MERGE: this PR remains DRAFT and contains only exclusion/diagnostic/freshness hardening. The requested default-off authenticated submodule inventory, selected historical child-tree capture, dirty child freshness, and graph/indexed/live retrieval parity remain unimplemented. No independent review. #4593 stays open. Please do not merge or change the draft selection to conceal missing behavior, and preserve #4800/#4508, protected #4776 Kev and read-only #4763. |
|
Refs #4593 — new source-review boundary mismatch, regression needed; not a claimed exploit or passing test (reviewed at exact head The default-deny live walker now checks parent stage-0 gitlink roots through Test the concrete scenario with a real parent 160000 gitlink under Current-head hosted workflow 42468887836 has passed repository gates, but Rust native/OS tests and Clippy were skipped for this draft; the Dashboard job remained queued when checked. Do not push a speculative fix that cancels that run. Fix via the one canonical authenticated submodule boundary, not a third independent scanner, and keep DRAFT/HOLD MERGE, |
A parent stage-0 gitlink remains tracked repository source even when its child checkout loses the .git marker. Check every indexed ancestor before an ignored dependency is admitted or revalidated, preserving the existing proof for ordinary ignored sources. Add a real local Git submodule regression that removes the child marker and checks both the admission proof and production source-read refusal. Refs #4593
|
Refs #4593 — published The ignored-dependency gix proof now checks every parent index ancestor for mode-160000 gitlinks, not only the exact candidate path. This closes the omitted-marker admission path discussed in comment 6720823044 for verified ignored dependencies, including the production source read and existing ignored-roster revalidation. Added a real local Git submodule regression source: verified stage-0 gitlink and No Rust regression has executed through Bazel, and there is no independent review. Current-head hosted CI 42482445184 was queued. Previous-head run 42468887836 completed successfully after re-running a Dashboard job blocked by a transient download timeout; draft native Rust/Clippy remained skipped. I independently checked the disposable Git fixture commands locally, but that is not a repository Rust test or acceptance evidence. HOLD MERGE / keep DRAFT / #4593 OPEN. The default-off authenticated submodule inventory, selected child-commit historical capture, child-aware freshness, and indexed/graph/live retrieval parity remain undone. Protected Kev/release PRs, #4800/#4508, and CI protections were not changed. |
Refs #4593. Full authenticated child indexing still incomplete.
|
Refs #4593 — guarded source repair on existing DRAFT #4801, HOLD MERGE. Commit One file changed: The integrated canonical default-off authenticated submodule opt-in, shared inventory across child candidate capture/freshness, pinned historical submodule trees, and parent-relative graph/retrieval are not implemented. Do not mark ready, merge, or close #4593. Protected #4776 Kev, #4763 release and #4800/#4508 peers untouched. |
Refs #4593. Reuse the parent's index-plus-HEAD boundary for ignored sources and cover the real staged-removal checkout scenario. The authenticated opt-in indexing feature remains incomplete.
|
Refs #4593 — source repair published on existing DRAFT PR; HOLD MERGE. Commit Ignored-source proof now calls the same index-plus-HEAD parent gitlink boundary as the source walker. The real-Git fixture commits a parent with a Verification limit: read back the exact commit and diff; no nonzero Bazel Rust test executed locally. Current-head hosted run 42503394287 was queued and chooses |
|
Refs #4593 — exact-head hosted Bazel formatting repair published on existing DRAFT #4801. At Published HOLD MERGE / keep DRAFT / issue #4593 OPEN. Exact-head nonzero Bazel source-walk/authentication, live graph/pinned-history and previously failing Windows child-commit regression, performance measurement of per-entry parent-authentication checks, independent approval and integration onto master remain outstanding. No peer, protected model/release work or CI policy was modified. |
Refs #4593 Reject existing child source walks and graph reads when the child Git administration changes core.worktree without changing the .git marker. Reuse the initial pinned-store topology proof on every bound read. Add a real-Git regression that changes only the child repository worktree configuration after source and graph readers were created. The native Git fixture demonstrates the mismatch; Rust acceptance still needs nonzero Bazel tests on this exact commit.
|
Refs #4593 — published narrow submodule authority repair on existing DRAFT #4801, HOLD MERGE. Commit Concrete issue: after a child source walker or graph reader is constructed, changing the parent-owned child Git repository's Repair: existing bound reads reuse the initial pinned-store topology verification, including child Git/common directory and current checkout identity. The added real-Git Rust regression first proves positive parent/child access, changes only the child repository worktree configuration, checks that fresh admission rejects it, and requires both an already-open graph reader and lazy source walk to refuse further child access with a typed error. Validation hold: these new Rust tests have not executed through Bazel. The prior-head CI passed applicable repository/dashboard gates but skipped native Rust; current-head native Windows (including the prior failed child-HEAD regression), Linux/macOS, performance of per-entry binding checks, and independent approval are still required. Do not mark ready, merge, claim completion or close #4593. No protected/peer refs, CI settings or release work were changed. |
|
Refs #4593 — new actionable capture-time authorization gap; DRAFT / HOLD MERGE (reviewed at exact remote head The existing
Falsifiable stale-inventory scenario: classify an authorized Required narrow repair/test: pass the original authenticated child binding/inventory through the concurrent worktree capture, revalidate it at the actual bounded child read (and when relying on the resulting snapshot), and refuse or retire a revoked child rather than accepting bytes on a stale string prefix. Regression must exercise successful authorized capture, revoke staged parent gitlink, registration, or child
|
Refs #4593 Retain canonical submodule checkout authority alongside classified source paths. Reject stale parent or child binding before/after each child file read and around snapshot reuse, including the clean-HEAD shortcut. Add real-Git staged-gitlink revocation/recovery regression coverage. Native exact-head Bazel verification and independent review remain pending.
|
Refs #4593 — narrow capture-time authorization repair published on existing DRAFT / HOLD MERGE branch. Commit
This addresses the stale-inventory capture finding in comment #6779062117. Not accepted yet: no exact-head Rust behavior has run through Bazel; CI 42904015316 was queued with PR #4801 was restored to DRAFT and remains unmerged; #4593 remains open. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| && self.parent_registration_is_live() | ||
| // The child gitdir marker is unchanged if its repository config | ||
| // redirects core.worktree. A fresh binding would refuse that store; | ||
| // an existing graph reader or lazy walk must do the same. | ||
| && open_pinned_store(&self.git_dir, self.pin) | ||
| .and_then(|child| child.workdir()?.canonicalize().ok()) |
There was a problem hiding this comment.
🔴 Submodule walks repeatedly parse the parent index
For every child entry, checkout_is_live reopens both repositories and parses the parent index. Ordinary scans repeat this work per file and directory, slowing searches and indexing as repositories grow.
Learn more
CheckoutWalk::next calls checkout_is_live before each child directory entry, and graph readers call it for each resolved child file. That check opens the parent repository, loads its index, reads the registration, and reopens the pinned child store. The work therefore grows with both the number of child entries and the size of the parent index, even without a binding change.
Example: A scan of a child with 5,000 files performs at least 5,000 parent index loads and pinned child opens instead of validating a stable binding once per scan.
Recommended fix: Preserve per-entry revocation checks, but separate cheap change detection from the full repository, index, and pinned-object validation. Revalidate on a detected change and on any uncertain metadata state; measure the ordinary scan with a populated parent index.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let Ok(index) = parent.index_or_empty() else { | ||
| return false; | ||
| }; |
There was a problem hiding this comment.
🟡 Parent Git errors silently hide child sources
If the parent index becomes unreadable, parent_registration_is_live returns false without preserving the error. LiveSourceBoundaries::resolve then returns Ok(None) for existing child files, hiding unavailable Git authority as absent source.
Learn more
A graph reader can retain LiveSourceBoundaries after its initial successful open. On later reads, the new parent-registration check reopens the parent and reads its index. An index read error takes the same false path as a revoked registration, and resolve converts both to Ok(None). The repository's production-failure rule requires unavailable authorities to remain typed rather than become successful empty results.
Example: Open boundaries while app/first.rs is valid, then make the parent's index unreadable before resolving that path. The read returns Ok(None) even though the child still exists; an index error is expected.
Recommended fix: Return a typed result from registration validation and propagate Git open, index, and registration read errors through resolve. Preserve a distinct refusal outcome for an actual revoked registration, and adapt the lazy walker to surface its existing typed walk error.
Was this helpful? React with 👍 or 👎 to provide feedback.
Apply the two exact formatting hunks from CI 42904015316 without changing source semantics or weakening the staged-revocation regression. Refs #4593
|
Refs #4593 — exact-head CI formatting repair published; #4801 remains DRAFT / HOLD MERGE. CI 42904015316 ran on Published The newest review has two further unresolved findings: per-entry full parent Git/index revalidation may amplify a 5,000-file child scan; parent index/Git read errors currently collapse into |
Refs #4593 Keep unreadable parent index and malformed registration errors distinct from revoked child identities. Propagate through live graph, lazy source walk, and indexed capture. Add a real-Git regression for index corruption.
|
Refs #4593 — published focused unavailable-Git-authority repair, still DRAFT / HOLD MERGE. Commit Actionable review finding repaired in source: New real-Git Rust regression in Verification: These changed Rust sources/tests have not executed under Bazel on this commit; the new exact-head workflow was queued at publication with native jobs skipped in draft mode. No passing Rust/Clippy/macOS/Windows claim, independent approval, or #4593 acceptance. Do not merge or close the issue. The other unresolved reviewer finding — O(files × parent-index verification) during child walks — remains actionable and needs a safe, measured bounded solution without weakening per-entry revocation. #4800 remains peer-owned: this commit touches |
Apply the pinned formatter's line wrapping and indentation to the submodule checkout and scheduler checks flagged by repository gates.
Refs #4593 Avoid reloading the parent Git index and reopening the child store for all entries of an unchanged authenticated checkout on Unix. Compare native inode, size and change-time identity on the parent, registration, child administration and object pack paths before every source read. Revalidate through the exact Git authority on mutation, with no cache for alternates or uncertain filesystem evidence. Windows retains full validation. Exercise 128 repeated real child reads, staged revocation and same-length working registration changes without weakening existing source refusals.
|
Refs #4593 — published focused per-entry submodule-authentication performance repair on the existing #4801 branch, DRAFT / HOLD MERGE. Commit Code change: On Unix, repeated Falsifiable Rust regressions added: 128 real child-source resolutions must perform exactly one full Git-binding validation on an unchanged checkout, then a staged gitlink removal must immediately refuse and force a fresh validation; a same-length working Evidence limits: The new exact-head CI is 42922412852, queued at readback. The authored Rust tests have not run under Bazel, nor have cross-platform native CI or independent approval. Keep both review threads open pending test evidence, source/security review and any necessary correction. #4593 remains OPEN; no merge, release, model or infrastructure mutation. |
Refs #4593 Track secondary shared-index file identities in the per-entry submodule binding witness. Main index and Git-dir stamps alone cannot detect an in-place corrupted/replaced shared index, allowing stale authenticated sources to be reused. Add a real Git split-index mutation regression.
|
Refs #4593 — published a focused split-index authority repair on existing #4801 DRAFT; HOLD MERGE. Commit New actionable defect fixed: the newly added Unix per-entry binding cache watched Falsifiable regression committed: create a real parent/child Git fixture, enable split-index with New exact-head CI was queued at readback. Draft workflow's native lanes are ordinarily skipped; no exact-head Linux/macOS/Windows Bazel behavioral pass, Clippy, formatter success or independent approving review is claimed. Preserve both unresolved review threads until genuinely verified. #4801 remains OPEN DRAFT and unmerged, #4593 OPEN until full acceptance on |
Refs #4593 A retained child can outlive a change to the parent's effective core.worktree in .git/config.worktree. The old cache watched the shared config and index but not the per-worktree configuration: an in-place edit leaves those stamps unchanged. Include both parent administrative config.worktree witnesses and regress a real Git parent/child checkout with an unchanged index/directory. Keep the original full Git validation and typed revocation, without granting any new source identity. Exact-head nonzero Bazel validation remains owed.
Refs #4593 Use real newline and tab escapes in the new on-disk config.worktree fixture. The parent worktree redirection must be parsed by Git, not written as literal escape sequences. Preserve the same revocation assertions.
|
Refs #4593 — new parent worktree-config authorization gap repaired and pushed on the existing #4801 branch; DRAFT / HOLD MERGE. Published New concrete stale-auth hazard: the Unix cached child-binding witness watched the parent Git index and shared Repair: witness both parent Git/private and common administrative Keep #4801 OPEN DRAFT / unmerged, #4593 OPEN, and preserve #4800's separate progress-code changes when reconciling |
Apply the pinned formatter layout reported by repository gates. Validation: owning Bazel formatter target passed; independent review completed.
Fix strict Clippy type complexity and nested binding checks, and format the reconciled tree-capture guard. Exercise split-index corruption through retained readers and revoke a real staged gitlink during snapshot capture, asserting refusal, preserved publication, and recovery.
|
The historical blanket hold is superseded by the maintainer request to finish this PR. Pushed 6ca67c5: fixed two strict Clippy failures plus reconciled guard formatting, strengthened split-index refusal through retained readers, and added mid-capture staged-gitlink revocation/refused-publication/recovery coverage. Native macOS passed 398 source-index and 714 runtime cases (two existing opt-in cases ignored) before the final behavior-preserving nested-if collapse; final rustfmt and strict Clippy passed and all six focused runtime submodule cases reran with the anti-vacuity helper. Latest-head Windows indexing suites are dispatched: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/43094396000 . The description now states these actual remaining CI/Windows checks rather than an indefinite do-not-merge instruction. No merge or Windows pass is claimed. |
Refresh the existing canonical manifest expectation for the required include_submodules policy field. Preserve all segment-byte and round-trip assertions. Also apply the measured canonical batch setup repair for the 10,001-row global-db overlap fixture without changing its acceptance or timeout. Exact-head CI verification remains required.
Summary
Refs #4593.
Enable direct Git submodules through the default-off project setting
index.include_submodules.v1, followed by a daemon restart. The parent-relative child paths participate in source walking, grep, structural search, unmounted-file audit, and graph indexing. Parent-to-child call edges remain in one graph.Source authority
.gitmodules, and child checkout to the parent-owned Git module store.Repairs and verification
The earlier blanket DRAFT / HOLD MERGE wording is superseded by the maintainer's request to finish and verify this PR. The prerequisite dashboard fixes and scheduler progress work are integrated. The PR is ready for review; this does not authorize bypassing required checks.
Latest commit:
6ca67c5b3669d72a60a1e6dd1379e765e355f6a2.Remaining merge checks
Required hosted CI and the explicitly dispatched native Windows indexing suites must pass on the latest head. The Windows runtime suite includes the formerly failing child-HEAD/add/delete transition, historical pinned graph, and restart journey. No Windows test pass is claimed yet. New actionable review findings must be addressed before merge.
No approval, merge, or completion of #4593 is claimed by these local results.