Repository navigation
Conversation
Both text channels and the graph channel returned candidates in the order they happened to be visited, and fusion turns that position into a reciprocal-rank term. The resulting ranks were unrelated to relevance, which made every hybrid method's ranking depend on index layout rather than on the query. BM25 and vector truncated by traversal prefix. Querying a candidate limit of 100 scored the first 100 documents in index order - which is chunkId order, because both builders insert in that order - and then sorted only those. Every query therefore returned the alphabetically first matches. On HotpotQA this put 98.5% of BM25 top-10 hits inside the first 100 of 41976 chunkIds, and measured recall sat at chance level (0.0022 for BM25, 0.0000 for vector). Both readers now score the whole matching set and retain the best `candidateLimit` of it. The limit bounds the retained set, not the traversal; scoring work stays visible in the evaluated counters. Graph hits were returned in expansion order, whose second sort key is a hash used for deterministic sampling. `finish()` now orders by association score with a path-based tie-break, so the graph channel's rank means what fusion assumes it means. Separating retained candidates from scored documents broke two accounting assumptions that had silently equated them: the candidate budget was charged per scored document, letting one channel consume a whole request's allocation and starve the graph stage, and the response validator rejected a retained count that exceeded the channel budget. Both now track candidate output, with scoring work reported separately. The graph quota that preserves graph evidence in the fused window demanded every returned position carry graph evidence, evicting higher-scoring text candidates. It is now `retrieval.graph-fusion.graph-quota-share` (default 0.34), clamped below the window so the quota can never outrank the fused order. Exact ranking becomes the benchmark default, since the bounded path exists to cap scoring work and measuring it silently reported text retrieval at chance level. A `textChannelBoundScored` column flags a run that scored exactly its candidate limit, so a capped run cannot be mistaken for a quality measurement again. HotpotQA, 500 queries, topK 10, maxCandidates 100: BM25_ONLY recall 0.0022 -> 0.9673 VECTOR_ONLY recall 0.0000 -> 0.8197 FIXED_HYBRID recall 0.8385 -> 0.9460, MRR 0.7425 -> 0.8008 ADAPTIVE recall 0.9082 -> 0.9525, MRR 0.6886 -> 0.6970 2Wiki is largely unchanged because its graph channel rarely saturates the quota. Six existing assertions encoded the defects and are updated with them: two expected the traversal-prefix winner, two treated scoring as the budgeted quantity, and two expected the graph stage to be starved of budget. New tests pin the fixes: a high-scoring chunk stored under a late chunkId must still be retrieved, the graph channel must rank by association score, and hop-split expansion must return the same order as a single call. Verification: 403 retrieval tests and 114 Python tests pass, Checkstyle clean.
The maintainers asked for three things the previous material could not answer: how to reproduce the numbers without the corpora, what the cost side looks like next to quality, and which part of the adaptive decision actually does anything. Re-measuring first was unavoidable. The earlier 500-query runs were scored against a chunking that no longer exists in the bundle: the same query returns different evidence ids (0009eeb4... vs 2db9ae3b...), and only graph-only kept its value (0.5028 to 0.5030) because it barely touches the text index. Every other method moved by two orders of magnitude once the text channels were scored against the corpus that is actually shipped. The three documents report only the re-measured figures and mark the old ones void. What the new numbers say, including the parts that do not flatter the design: * Adaptive does not beat fixed hybrid. MRR is 6.8% lower on 2Wiki and 13.0% lower on HotpotQA. It beats graph-only by 36-150%, but graph-only is the weakest non-degenerate method. * F1 is 0.002-0.010 and EM is zero for every method. The answer layer is extractive and the gold answers are short entities, so a span extractor rarely reproduces them. The lowest-recall method has the highest F1, which is why F1 cannot rank these methods. * The cost saving exists on 2Wiki only (34% fewer edges); on HotpotQA adaptive spends 65% more edges than graph-only. A component ablation was added to explain the routing gap. Neutralising the graph weighting raises MRR from 0.6796 to 0.7562 (+11.3%), so the policy's defining behaviour is the cause: it weighs the graph eight times higher than text while the text channels retrieve better on both corpora. Three of seven rules never fire, replanning executes 1.21 times per request and changes no outcome, and two more rules cannot fire at their thresholds. Those are recorded as open problems rather than papered over. The benchmark now also reports the graph stage's own token estimate and the run total, because a per-request mean cannot be compared against a per-token price.
The headline tables were retyped by hand from the run CSVs and drifted in the last digit of several cells: 2Wiki BM25 MRR appeared as 0.7367 instead of 0.7368, HotpotQA BM25 as 0.8253 instead of 0.8254, and eight more. The deltas were computed from the correct values, so the two disagreed. Every quality figure in all three documents is now checked against the CSVs it cites: 48 values across six methods and two datasets, plus the ten ablation rows. A script compares the rendered tables to the source and reports zero mismatches.
The guide quoted 455 Java tests and 114 Python tests; the suites now run 463 and 115 after the ablation and token-column guards were added. A reader checking the first example command against a fresh run would have seen a mismatch.
The review was raised in Chinese and the figures need to be checked by reviewers who work in Chinese. The English documents stay canonical; these are translations of the same content, not a separate analysis. Every number was re-checked against the run CSVs after translation: 112 values across the two metric tables and the ablation table, all matching. Wording that carries a conclusion - "adaptive does not beat fixed hybrid", "the graph weighting is the cause" - is translated rather than softened.
…ve setting The maintainer asked whether a configuration exists where the adaptive channel is the best performer with the router in place. The component ablation identified the graph preference as the cause of the gap; this sweep answers the follow-up with measurement instead of argument. A declared 6-cell grid over graph-weight x text-weight, each cell restarting the service with exactly one pair of weights and running the same 200-query slice. The shipped pair (2.0 / 0.25) is the worst cell. The best cell (0.25 / 1.0) reaches MRR 0.7676 against fixed hybrid's 0.7506, a +2.3% improvement, while scanning 13.0% fewer edges. The grid is declared up front so this is a search rather than a cherry-pick, and the result is reported as a candidate to validate, not a new default: it comes from one dataset, one slice, with no held-out confirmation. Also hardens ablate_adaptive.sh against a zero-byte classpath left by an interrupted build, which previously failed later with a confusing NoClassDefFoundError.
Replanning was a no-op by construction: once the replan budget was spent the policy fell back to the pre-execution feature snapshot, so the post-hop boundary re-planned from graphGrowthRate=0 and never stopped a frontier that had already grown past the limit. Measured features are now always merged, a spent budget may only contract the plan, and the post-hop boundary keeps the completed text channels' allocation instead of rolling it back, which turns the previous plan round trip into a real stop. Routing defaults are re-derived from measurement rather than kept as a graph-first prior. A 200-query sweep per dataset puts the optimum at a graph:text ratio of 1:4; the shipped 8:1 was the worst cell measured, costing 7.8% MRR on 2Wiki and 15.0% on HotpotQA against fixed hybrid. Defaults are now graph-weight=0.25, text-weight=1.0. The lexical/semantic branches no longer hard-code a graph weight of 1.0, which would have quadrupled it whenever they fired, and high-average-idf moves from 1.5 - below the measured minimum, hence no discriminating power - to 4.0. max-hop is configurable and defaults to 1: the retriever already stops itself on diffusion growth, so the planned second hop never executed. Multi-hop remains available by setting it back to 2. decouple-text-routing is added behind a flag that stays off: enabling it lowered recall on both datasets, so the lexical/semantic signals are recorded as unhelpful rather than enabled by default. The evaluation pipeline now refuses to score a response against a corpus the service is not serving. Gold resolution is checked before any request, and the first returned evidence id absent from --chunks aborts the run with an actionable message. This is the failure that produced the earlier 0.002 text recall and the wrong fusion prior. Verification: 470 Maven tests and 118 pytest tests pass. Sweeps and attributions are reproducible through tools/graphrag/sweep_fusion.sh and ablate_adaptive.sh, which now takes --max-candidates so its halving ablation can match the 500-query budget.
The offline dependency:build-classpath goal can write a zero-byte file; the service then failed at startup with NoClassDefFoundError. reproduce.sh now reuses GEAFLOW_CLASSPATH when provided and stops with an actionable message otherwise, matching the ablation and sweep runners.
Three documents, one per review point, under tools/graphrag/docs. external-reproduction.md gives the upstream dataset links, versions and SHA-256 checksums instead of shipping corpora, pins the commit the figures were produced from, and lists the checks that distinguish a broken reproduction from a real result, including the enforced corpus-namespace and gold-resolution guards. performance-metrics.md reports cost next to quality: hit rate at three strictness levels, F1 and EM, retained and graph-estimated tokens, an explicitly parameterised LLM cost example, edge scans and latency. Adaptive deltas are given against every single-channel baseline and against fixed hybrid, with negative values reported as measured. adaptive-module-analysis.md contains the full component ablation, a measured attribution for each condition that shows no difference, and the decision module's limitations as open problems for follow-up work. All figures come from the 500-query runs of commit fee1608 and are rendered from the run CSVs rather than transcribed; the eight adaptive MRR deltas were checked arithmetically. Non-adaptive methods are bit-identical to the previous runs, confirming that the measured gains come from the routing changes alone.
|
The core implementation, metrics, ablation experiments, and failure analysis have improved significantly. However, this is not ready for acceptance under the independent reproducibility requirement. reproduce.sh currently uses GEAFLOW_VECTORS for both chunk-vector JSONL and query-vector NPZ files, which are incompatible formats. The query vectors also lack a download URL or complete generation instructions, while --skip-build requires a prebuilt runnable index package. Please address the following:
|
Six review points addressed. Separate the two vector inputs. Chunk embeddings (JSONL, one vector per chunk) build the index; query embeddings (NPZ, ids plus vectors) drive the benchmark. reproduce.sh used GEAFLOW_VECTORS for both, so the publish stage could be handed an NPZ that the bridge reads as JSONL lines. GEAFLOW_CHUNK_VECTORS and GEAFLOW_QUERY_VECTORS are now distinct, with GEAFLOW_VECTORS kept only as a deprecated query-vector alias. Document vector generation end to end: model iic/nlp_gte_sentence-embedding_english-small, vector_version v1.0.0, ModelScope source link, 384 dimensions, and the exact embed_chunks / embed_queries commands. The full pipeline is staged build_release -> embed -> publish -> benchmark, so no prebuilt index package is required. Verified end to end on a 12-query real subset: ingest produced 171 vertices and 52 chunks, the bundle published READY, and every method answered 12/12 queries with the vector channel evaluating all 52 chunk vectors. Complete the Python dependencies: requirements.txt for benchmarking and tests, requirements-ingest.txt for parquet ingestion, requirements-embedding.txt for regeneration. Fix the default build. Run artifacts were landing in module directories because Maven resolves a relative -Dmdep.outputFile against the module, which also left the intended classpath empty. Output directories are now absolute, and the root RAT excludes cover **/tmp/** so generated output no longer fails the licence check. mvn -o -pl geaflow-ai test passes without skipping checkstyle, RAT or jacoco: 470 tests, 0 failures; 118 pytest tests. Resolve dataset licence status. 2WikiMultiHopQA is Apache-2.0 per the upstream repository LICENSE, HotpotQA is CC BY-SA 4.0 per its project page; both manifests now record confirmed-upstream with the evidence, instead of pending-review. Report relationPathHit next to recall and explain why it is near zero while paths are returned: expansion yields entity neighbourhoods, not gold reasoning chains. Complete the HotpotQA component ablation over all eight conditions. It adds a finding 2Wiki does not show: zeroing the graph rank weight improves HotpotQA MRR by 6.5%, whereas the same change costs 0.39% on 2Wiki, so no single global fusion ratio suits both corpora.
|
感谢详细的评审意见,六点均已在 5d58d77 中修复。
数据集仍不随仓库分发;评审者可从文档中的来源下载、核对校验和,并仅凭仓库内的说明完成构建与分数复现。 |
Measured on all 48851 re-embedded chunks: text channels reproduce exactly, but the rebuilt graph has 834797 edges against the reference 640275 because the ingest and graph code changed after the reference bundle was built. Documents that reproducing the reported scores requires the published reference bundle, and that from-scratch graph figures additionally require the same build code revision. Also documents the required GRAPH_RAG_CACHE variable.
A from-scratch rebuild reproduces the text index exactly but not the graph snapshot: the reference 2Wiki bundle has 640275 edges while a rebuild with current code has 834797, because the ingest and graph code changed after the reference was built. Reproducing graph figures therefore requires the same build code, not just the same data and model. publish_bundle.py now records `git rev-parse HEAD` as build_code_revision in READY.json, with an explicit --build-code-revision override for reproducible or offline builds, and the Java bridge writes it into the bundle metadata. A reviewer can check out that revision and rebuild the same graph snapshot instead of a newer, incompatible one. Older bundles without the field remain valid and are still the reference for the earlier reported figures. Verified by publishing a real 12-query subset: READY.json carries the exact HEAD revision. CanonicalJsonlArtifactBridgeTest asserts the field is written; 470 Maven tests pass with checkstyle and RAT enabled, and 118 pytest tests pass.
…sums The reported scores are tied to the reference bundles, which are not in the git repository and whose build revision predates build_code_revision recording. Publishing them is therefore the way to keep those exact figures reproducible without forcing a rebuild that would change the graph snapshot. package_reproduction.sh assembles one archive per dataset containing the READY bundle, the query vectors, the reference six-method CSV and a REPRODUCE.json descriptor, and writes SHA256SUMS.txt plus the gh release command for whoever has release permissions. The reproduction guide now leads with downloading or regenerating these packages and points at the extracted bundle and vectors, while warning that rebuilding them with newer code will not reproduce identical graph figures. Verified by unpacking: READY.json version and every artifact checksum are identical to the source bundle.
… can reproduce The reported figures were tied to a reference bundle that is not in the repository and whose graph snapshot cannot be rebuilt from current code (640275 edges then, 834797 now because the ingest and graph code changed afterwards). That made the numbers impossible to reproduce from the remote contents alone. Both datasets were rebuilt and re-measured with the current code, and the reports now use those figures. Rebuilding twice yields the same bundle version and edge count, so the new baselines are reproducible from the repository contents alone; READY.json also records the build revision. Conclusions are unchanged in direction and small in magnitude. Adaptive MRR is 0.7850 on 2Wiki (+2.3% over fixed hybrid) and 0.8096 on HotpotQA (+1.2%), against 0.7867 and 0.8067 before, i.e. under 1% drift even though the graph edge count moved by 30%. Single-channel methods stay bit-identical. The 1:4 fusion ratio remains optimal on both datasets, and the HotpotQA no-graph-fusion result is literally identical at 0.8397, so the finding that graph evidence is mostly ranking noise there is robust to the snapshot change. The three zero-delta ablation conditions are unchanged as well. Also documents the reproducibility checks: the 12-query end-to-end build, the determinism of the full rebuild, and why graph snapshots drift without affecting the headline conclusions.
…ntents alone Freshly regenerated all query vectors and re-ran both 500-query benchmarks against the rebuilt bundles. 2Wiki matches the report bit for bit on every method; HotpotQA matches except for a 0.04% MRR wobble on the vector channel caused by float-level differences when vectors are regenerated. This closes the last open question on whether the figures can be reproduced from the remote repository alone.
|
@aotenjou
|
Document the missing query-JSONL step with a real command. materialize_queries.py turns the raw dataset (zip or parquet) into the canonical query records the benchmark and embed_queries.py consume; the generated id and question sequences are identical to the files used for the report, so the evaluation is unchanged. The HotpotQA section now names the actual parquet cache and its SHA-256, which is what the manifest references - the earlier JSON path and checksum contradicted it. Angle-bracket placeholders in the build, embed and publish commands are replaced by a copy-pasteable 2Wiki example with a HotpotQA substitution table. Make floating-point assertions tolerant instead of exact. RankingMetrics.ndcgAtK was asserted to sixteen digits, but Math.log is only specified to one ulp, so that assertion can fail on other JDKs even though the computation is correct. All exact double assertions in the touched tests now compare with a 1e-9 tolerance. Verified by running the whole module on a downloaded Temurin JDK 17.0.21 as well as on JDK 21: 470 tests pass with checkstyle and RAT enabled. Explain the adaptive path-hit of zero as a graph-construction limitation rather than a routing tuning issue: edges come from entity co-occurrence and carry no directed relation semantics, so the gold reasoning chain is absent from the graph and no fusion weight can produce it. Improving path-hit requires relation extraction at ingest or relation-constrained traversal, which is now stated as the follow-up work.
|
Thanks for the careful re-review — all three points are addressed in 6a4c379. From-scratch workflow. The missing step is now a real command: tools/graphrag/materialize_queries.py turns the raw dataset (zip or parquet) into the canonical query JSONL, and I verified JDK 17 test failure. The root cause was an exact sixteen-digit assertion on RankingMetrics.ndcgAtK. Math.log is only specified to one ulp, so that assertion can legitimately fail on another Adaptive path-hit of 0. This is a graph-construction limitation, not a routing-tuning issue. Edges come from text-cooccurrence-entity-v2 — "two entities appear in the same chunk" — and |
adaptive graph retrieval
Summary
This PR adds an end-to-end, budget-aware GraphRAG retrieval system to
geaflow-ai. It combines lexical, vector, and graph retrieval behind a unified evidence model and Java API, with adaptive routing, bounded execution, provenance tracing, and extractive answer support.Design
The system covers the full retrieval workflow:
RoutingContextfeeds anAdaptivePolicy, which produces a validatedRetrievalPlanfor the executor. The system supports BM25-only, vector-only, graph-only, fixed hybrid, and adaptive hybrid modes, with sequential, parallel, and cascaded execution.The retrieval policy and budget accounting remain in Java as the behavioral source of truth. The Python tools prepare data and run reproducible evaluation workflows; they are not runtime dependencies of the Java service.
Validation
model/RetrievalDomainModelTest(19)、model/RetrievalProvenanceContractTest(17)、answer/CitationProvenanceTest(6)ingest/IngestionPipelineTest(7)、ingest/CanonicalJsonlArtifactBridgeTest(5)、index/ArtifactPublicationTest(3)、index/graph/GraphArtifactTest(14)、metadata/*(23)channel/graph/VersionedGraphViewTest(11)、channel/graph/VersionedGraphViewLazinessTest(4)channel/graph/GraphRetrieverTest(19)、channel/graph/GraphRetrieverContinuationTest(4)、channel/graph/AssociationScoreFormulaTest(10)、channel/graph/EdgeStrengthTimeDecayTest(13)、channel/graph/EntityAnchorResolverTest(4)、service/EdgeScanBudgetTest(4)execution/ExecutionBudgetTest(9)、service/BudgetMonotonicityTest(4)、execution/RecallPlanAllocationTest(5)、fusion/WeightedRrfFusionTest(5)、service/RetrievalModeContractTest(3)execution/AdaptiveRoutingTest(18)、execution/AdaptivePolicyContractTest(6)、execution/ReplanBoundTest(4)、execution/AdaptiveGraphFeedbackTest(6)、api/RetrievalPropertiesTest(18)service/GraphRagMvpEndToEndTest(44)、service/RetrievalServiceTest(5)、service/ReadinessReportTest(4)、service/RetrievalBatchCliTest(5)、api/RetrievalApiJsonTest(18)evaluation/EvaluationMetricsTest(3)、evaluation/RelationPathGoldenTest(4)、evaluation/AnswerMetricsGoldenTest(1)、service/RetrievalMetricsTest(1)answer/AnswerConsumerTest(10)、service/RetrievalExtractiveAnswerTest(6)、answer/AnswerConsumerStressTest(1)service/ApiUsageExampleTest(5)、service/RetrievalDeterminismTest(4)