Skip to content

fix(skill): pass query values through sidecar files, never substitute them - #5306

Open
danikdanik wants to merge 1 commit into
Graphify-Labs:v8from
danikdanik:fix/skill-query-file-transport
Open

danikdanik wants to merge 1 commit into
Graphify-Labs:v8from
danikdanik:fix/skill-query-file-transport

Conversation

@danikdanik

Copy link
Copy Markdown

Summary

The query/path/explain runbooks still tell the agent to paste free-text values straight into shell commands and Python literals — QUESTION, ANSWER, ORIGINAL_QUESTION, NODE_A, NODE_B, NODE_NAME, the correction text, and the cited-node list. That is the same paste-the-value class the INPUT_PATH fix (#4762) removed, on the most attacker-adjacent surface in the skill: the user's own question.

Why

--answer "ANSWER" was the worst case: the agent's full answer text went into a double-quoted shell argument, so any ", $() or backtick in it re-executed as bash — on ordinary, unmalicious input. graphify query "<question>" in the core stub and the Step-1 fast path did the same with the user's question, and a_term = 'NODE_A' put node labels inside single-quoted Python literals. A node label or question containing the delimiter breaks out; a $() in any of them executes.

Two follow-on correctness bugs lived at the same boundary and are fixed here rather than deferred:

  • save-result uses argparse — a --question or --correction value starting with - (e.g. "--update vs --cluster-only") was eaten as a flag and rejected. All free-text options now use the --opt=value form, which argparse treats as data.
  • Node labels are no longer argv at all: a label starting with - under nargs="*" was parsed as an option, and a while read loop dropped the final line of an unterminated file. A new --nodes-file flag on save-result (mirroring the existing --answer-file, shipped in V8 windows skill fix #1502) reads the labels itself.

How

Every free-text value now travels through a graphify-out/ sidecar the runbook writes first:

  • .graphify_question, .graphify_answer, .graphify_nodes, .graphify_node_a, .graphify_node_b, .graphify_node_name, .graphify_expanded, .graphify_correction — written by the agent, read back via "$(cat ...)" / --answer-file / --nodes-file / Path(...).read_text().
  • graphify/cli.py — --nodes-file reads labels line-by-line (strip, skip empties, tolerate unterminated last line). No other CLI change; --question=/--correction= needed no flag.
  • tools/skillgen/fragments/query-stub/default.md + core/core.md fast path — the raw question goes to .graphify_question, then graphify query "$(cat graphify-out/.graphify_question)".
  • tools/skillgen/gen.py — the bash→PowerShell translator now emits $((Get-Content f -Raw).Trim()) for $(cat f) sidecar reads: a plain (Get-Content ...) inside "..." is literal text in PowerShell, and .Trim() matches bash $(...) newline stripping. Monolith fragments (aider.md, devin.md) got the same embedded-query-section edits; _SANCTIONED_MONOLITH_DIFFS records the change-class.

What changed

  • 5 fragments + gen.py + cli.py; all 134 generated artifacts regenerated (--check clean).
  • tests/test_skillgen_query_value_transport.py (new) — bans every old placeholder token across all generated bodies, requires the sidecar reads + --question=/--nodes-file/--answer-file forms, executes a hostile save-result block under bash proving payloads arrive verbatim with no sentinel execution, and runs graphify save-result end-to-end with leading-dash and unterminated node lines.
  • Windows render pinned to the evaluated subexpression form.

Validation

  • python -m tools.skillgen --check: 134 artifacts byte-match.
  • pytest tests/test_skillgen_query_value_transport.py tests/test_skillgen.py tests/test_skill_no_backtick_substitution.py: 250 passed.
  • Full tests/ sweep run: failures identical to pristine upstream/v8 (env-only — subprocess tests without the project deps on PYTHONPATH, missing tree-sitter grammars); nothing attributable to this diff.

@github-actions

Copy link
Copy Markdown

Thanks for the pull request, @danikdanik. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.11 and 3.13, and that the change keeps extraction deterministic.

@github-actions github-actions Bot added the needs:title PR title should follow the conventional-commit format label Nov 18, 2026
@github-actions

Copy link
Copy Markdown

Thanks @danikdanik! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example:

fix(python): resolve a bare stdlib import correctly

Prefixes we use: feat, fix, docs, chore, refactor, test, perf, build, ci. Editing the PR title clears this automatically. This is just a nudge and does not block review.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify code review View in Graphify

Review completed · b41ee89bf69c

Tip

Merge confidence: 5/5 · Looks safe to merge · 🟠 High impact
No issues reported
1 of 1 earlier findings resolved

What this review covered
  • We didn't review 65 files that aren't source code, such as docs, config, images, or generated or third-party code.
  • Some lookups asked for files or lines that do not exist; the review was told and carried on.
  • 3 of 3 changed files inspected. This is scope, not a correctness guarantee.
Code graph and static checks · 1860 functions depend on this change

Impact & health

Impact — 1860 functions depend on the 1385 functions this change touches.

Health — grade A; 10 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):

  • dispatch_command() — 8 callers, 156 callees (high)
  • main() — 128 callers, 3 callees (high)
  • _run_hook_guard() — 7 callers, 18 callees (high)
  • _run_cli() — 11 callers, 8 callees (high)
  • render() — 16 callers, 5 callees (high)
  • audit_coverage() — 8 callers, 6 callees (high)
  • _stale_graph_sources() — 7 callers, 6 callees (high)
  • main() — 3 callers, 11 callees (medium)
  • …and 2 more

Formal checks — 1860 functions this change can affect weren't formally checked in this review. Formal checks are informational only.

Gate & verification

✅ PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1785 function(s) in the blast radius were not formally verified this run

Test selection

594 of 594 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/pi.test.ts — full-run-safety
  • tests/test_4874_jvm_resolution.py — full-run-safety
  • tests/test_4909_reference_binding_scale.py — full-run-safety
  • tests/test_4978_reconcile_cost.py — full-run-safety
  • tests/test_4992_ts_import_types.py — full-run-safety
  • tests/test_5072_splittable_under_cap_retry.py — full-run-safety
  • tests/test_affected_cli.py — impact, full-run-safety
  • tests/test_affected_defines.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_affected_overrides.py — full-run-safety
  • tests/test_affected_path_file_node.py — full-run-safety
  • tests/test_affected_seed_reference_forms.py — impact, full-run-safety
  • tests/test_agents_platform.py — impact, full-run-safety
  • tests/test_alias_folding.py — full-run-safety
  • tests/test_analyze.py — full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — full-run-safety
  • tests/test_astro_import_ids.py — full-run-safety
  • tests/test_atomic_canvas_export.py — full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_backend_opt_in.py — full-run-safety
  • tests/test_benchmark.py — full-run-safety
  • tests/test_benchmark_raw_graph.py — full-run-safety
  • tests/test_benchmark_real_query.py — full-run-safety
  • tests/test_binary_media_words.py — full-run-safety
  • tests/test_blade_extractor.py — full-run-safety
  • tests/test_build.py — full-run-safety
  • tests/test_build_located_semantic_identity.py — full-run-safety
  • tests/test_build_merge_dedup_scope.py — full-run-safety
  • tests/test_build_merge_ghost_fold_4955.py — full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — full-run-safety
  • tests/test_build_merge_shrink_guard.py — full-run-safety
  • tests/test_builtin_global_type_refs.py — full-run-safety
  • tests/test_c_include_resolution.py — full-run-safety
  • tests/test_cache.py — full-run-safety
  • tests/test_cache_long_paths.py — full-run-safety
  • tests/test_cache_stale_import_target.py — full-run-safety
  • tests/test_cache_windows_case_4925.py — full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — full-run-safety
  • tests/test_cargo_missing_manifest.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — full-run-safety
  • tests/test_case_only_decl_ids_4904.py — full-run-safety
  • tests/test_case_sensitive_resolution.py — full-run-safety
  • … and 544 more

⚠️ non-code file(s) changed (graphify/skill-agents.md, graphify/skill-aider.md, graphify/skill-amp.md, graphify/skill-claw.md, graphify/skill-codex.md …) → running the full suite for safety (a code graph can't see config/fixture/data deps)

⚠️ changed code file(s) with no mapped test (graphify/skill-agents.md, graphify/skill-aider.md, graphify/skill-amp.md, graphify/skill-claw.md, graphify/skill-codex.md …) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

…itute them

The query, path and explain runbooks pasted free-text values into quoted
shell arguments and Python string literals — --answer "ANSWER",
question = 'QUESTION', a_term = 'NODE_A' — so any question, answer or
node name containing a quote, $() or backtick executed as code instead of
data. Same injection class as the INPUT_PATH fix (Graphify-Labs#4762), but it fires on
ordinary use: answers and labels routinely carry those characters.

Every value now travels through a graphify-out/ sidecar the agent writes
first (.graphify_question, .graphify_answer, .graphify_nodes,
.graphify_expanded, .graphify_node_a/b, .graphify_node_name,
.graphify_correction), the answer goes through save-result --answer-file,
and the cited-node list through a while-read array. MODE and BUDGET stay
literals: fixed enum and number.

Adds a fence test that bans the placeholder tokens across all generated
bodies and proves a hostile question file reaches argv verbatim.
@danikdanik
danikdanik force-pushed the fix/skill-query-file-transport branch from a3a8719 to b41ee89 Compare November 18, 2026 09:53
@danikdanik danikdanik changed the title security(skill): carry query/answer/node values through sidecar files, never substitute them fix(skill): pass query values through sidecar files, never substitute them Nov 18, 2026
@danikdanik

Copy link
Copy Markdown
Author

Fixed in b41ee89 — _cat_to_get_content now emits .TrimEnd("r","n") instead of .Trim(), so the PowerShell read strips only trailing line endings and preserves intentional boundary spaces, matching bash $(cat) semantics exactly. Verified in pwsh: " padded rn" passes through as " padded ". Regenerated artifacts + updated the translation pin test. (Title also switched to conventional-commits format.)

@github-actions github-actions Bot removed the needs:title PR title should follow the conventional-commit format label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant