Repository navigation
fix(skill): pass query values through sidecar files, never substitute them - #5306
danikdanik wants to merge 1 commit into
Conversation
|
Thanks for the pull request, @danikdanik. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.11 and 3.13, and that the change keeps extraction deterministic. |
|
Thanks @danikdanik! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example: Prefixes we use: |
There was a problem hiding this comment.
Graphify code review 
Review completed · b41ee89bf69c
Tip
Merge confidence: 5/5 · Looks safe to merge · 🟠 High impact
No issues reported
1 of 1 earlier findings resolved
What this review covered
- We didn't review 65 files that aren't source code, such as docs, config, images, or generated or third-party code.
- Some lookups asked for files or lines that do not exist; the review was told and carried on.
- 3 of 3 changed files inspected. This is scope, not a correctness guarantee.
Code graph and static checks · 1860 functions depend on this change
Impact & health
Impact — 1860 functions depend on the 1385 functions this change touches.
Health — grade A; 10 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):
dispatch_command()— 8 callers, 156 callees (high)main()— 128 callers, 3 callees (high)_run_hook_guard()— 7 callers, 18 callees (high)_run_cli()— 11 callers, 8 callees (high)render()— 16 callers, 5 callees (high)audit_coverage()— 8 callers, 6 callees (high)_stale_graph_sources()— 7 callers, 6 callees (high)main()— 3 callers, 11 callees (medium)- …and 2 more
Formal checks — 1860 functions this change can affect weren't formally checked in this review. Formal checks are informational only.
Gate & verification
✅ PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1785 function(s) in the blast radius were not formally verified this run
Test selection
594 of 594 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety (
noncode-change,uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/pi.test.ts— full-run-safetytests/test_4874_jvm_resolution.py— full-run-safetytests/test_4909_reference_binding_scale.py— full-run-safetytests/test_4978_reconcile_cost.py— full-run-safetytests/test_4992_ts_import_types.py— full-run-safetytests/test_5072_splittable_under_cap_retry.py— full-run-safetytests/test_affected_cli.py— impact, full-run-safetytests/test_affected_defines.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_affected_overrides.py— full-run-safetytests/test_affected_path_file_node.py— full-run-safetytests/test_affected_seed_reference_forms.py— impact, full-run-safetytests/test_agents_platform.py— impact, full-run-safetytests/test_alias_folding.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_backend_opt_in.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_benchmark_real_query.py— full-run-safetytests/test_binary_media_words.py— full-run-safetytests/test_blade_extractor.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_located_semantic_identity.py— full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_ghost_fold_4955.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_c_include_resolution.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_cache_long_paths.py— full-run-safetytests/test_cache_stale_import_target.py— full-run-safetytests/test_cache_windows_case_4925.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_cargo_missing_manifest.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_only_decl_ids_4904.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safety- … and 544 more
⚠️ non-code file(s) changed (graphify/skill-agents.md,graphify/skill-aider.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
⚠️ changed code file(s) with no mapped test (graphify/skill-agents.md,graphify/skill-aider.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
…itute them The query, path and explain runbooks pasted free-text values into quoted shell arguments and Python string literals — --answer "ANSWER", question = 'QUESTION', a_term = 'NODE_A' — so any question, answer or node name containing a quote, $() or backtick executed as code instead of data. Same injection class as the INPUT_PATH fix (Graphify-Labs#4762), but it fires on ordinary use: answers and labels routinely carry those characters. Every value now travels through a graphify-out/ sidecar the agent writes first (.graphify_question, .graphify_answer, .graphify_nodes, .graphify_expanded, .graphify_node_a/b, .graphify_node_name, .graphify_correction), the answer goes through save-result --answer-file, and the cited-node list through a while-read array. MODE and BUDGET stay literals: fixed enum and number. Adds a fence test that bans the placeholder tokens across all generated bodies and proves a hostile question file reaches argv verbatim.
a3a8719 to
b41ee89
Compare
|
Fixed in b41ee89 — |
Summary
The query/path/explain runbooks still tell the agent to paste free-text values straight into shell commands and Python literals —
QUESTION,ANSWER,ORIGINAL_QUESTION,NODE_A,NODE_B,NODE_NAME, the correction text, and the cited-node list. That is the same paste-the-value class theINPUT_PATHfix (#4762) removed, on the most attacker-adjacent surface in the skill: the user's own question.Why
--answer "ANSWER"was the worst case: the agent's full answer text went into a double-quoted shell argument, so any",$()or backtick in it re-executed as bash — on ordinary, unmalicious input.graphify query "<question>"in the core stub and the Step-1 fast path did the same with the user's question, anda_term = 'NODE_A'put node labels inside single-quoted Python literals. A node label or question containing the delimiter breaks out; a$()in any of them executes.Two follow-on correctness bugs lived at the same boundary and are fixed here rather than deferred:
save-resultuses argparse — a--questionor--correctionvalue starting with-(e.g. "--update vs --cluster-only") was eaten as a flag and rejected. All free-text options now use the--opt=valueform, which argparse treats as data.-undernargs="*"was parsed as an option, and awhile readloop dropped the final line of an unterminated file. A new--nodes-fileflag onsave-result(mirroring the existing--answer-file, shipped in V8 windows skill fix #1502) reads the labels itself.How
Every free-text value now travels through a
graphify-out/sidecar the runbook writes first:.graphify_question,.graphify_answer,.graphify_nodes,.graphify_node_a,.graphify_node_b,.graphify_node_name,.graphify_expanded,.graphify_correction— written by the agent, read back via"$(cat ...)"/--answer-file/--nodes-file/Path(...).read_text().graphify/cli.py—--nodes-filereads labels line-by-line (strip, skip empties, tolerate unterminated last line). No other CLI change;--question=/--correction=needed no flag.tools/skillgen/fragments/query-stub/default.md+core/core.mdfast path — the raw question goes to.graphify_question, thengraphify query "$(cat graphify-out/.graphify_question)".tools/skillgen/gen.py— the bash→PowerShell translator now emits$((Get-Content f -Raw).Trim())for$(cat f)sidecar reads: a plain(Get-Content ...)inside"..."is literal text in PowerShell, and.Trim()matches bash$(...)newline stripping. Monolith fragments (aider.md,devin.md) got the same embedded-query-section edits;_SANCTIONED_MONOLITH_DIFFSrecords the change-class.What changed
gen.py+cli.py; all 134 generated artifacts regenerated (--checkclean).tests/test_skillgen_query_value_transport.py(new) — bans every old placeholder token across all generated bodies, requires the sidecar reads +--question=/--nodes-file/--answer-fileforms, executes a hostile save-result block under bash proving payloads arrive verbatim with no sentinel execution, and runsgraphify save-resultend-to-end with leading-dash and unterminated node lines.Validation
python -m tools.skillgen --check: 134 artifacts byte-match.pytest tests/test_skillgen_query_value_transport.py tests/test_skillgen.py tests/test_skill_no_backtick_substitution.py: 250 passed.upstream/v8(env-only — subprocess tests without the project deps on PYTHONPATH, missing tree-sitter grammars); nothing attributable to this diff.