Repository navigation
branch-4.2: [refactor](be) Unify MATCH and SEARCH execution across inverted index formats (#68758) - #69206
Merged
Conversation
…verted index formats (apache#68758) Backport the shared MATCH and SEARCH query engine and the CLucene and SNII source adapters. Preserve branch-4.2 scan-schema binding, segment row counts, rowset traversal, and BKD type conversion. Pass the segment delete bitmap to the SEARCH Top-K collector so deleted rows do not consume its result limit. The hunk audit in docs/backport-68758-hunk-audit.md records the source changes and the branch-specific adaptations. Conflicts: - Bind NESTED columns through the branch-4.2 read schema and root UID. - Use the branch's segment and type APIs in collection statistics and index readers. - Resolve moved header dependencies and define the test-only temporary file directory setter where its complete type is available. - Preserve the branch-only RowIdStorageReader test and its direct header include. - Omit Kuromoji-only changes because this branch has no Kuromoji parser. (cherry picked from commit 14fde1c)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Move the bundled corpus beside the relocated test so its default path resolves. Refresh the expected bitmaps and unsupported-query statuses for the shared MATCH engine while keeping the stored index files unchanged. Test: ./run-be-ut.sh --run --filter=SniiGoldenCorpus.WriteOrVerify -j 8
Member
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Oct 18, 2026
This was referenced Oct 18, 2026
Merged
yiguolei
pushed a commit
that referenced
this pull request
Oct 19, 2026
…conjunctions (#69172) (#69239) ### What problem does this PR solve? Issue Number: N/A Related PR: #69172, #69206 Problem Summary: Backport #69172 after #69206 introduced the shared inverted-index query engine to branch-4.2. Dense unscored conjunctions now skip terms held by every row, write a lone term directly to the result bitmap, use branch-free block intersection, and bound streaming-source batches. Query results are unchanged. Source PR: #69172 Source squash commit: `f937d950d45313f20864d287df453e1b90f5c615` (cherry-picked with `-x`). ### Release note None ### Check List (For Author) - Test - [x] Regression test: `test_match_format_parity` passed on an isolated branch-4.2 FE/BE cluster after rebasing onto `ecd322ad31193cda19cf4f3b08e0f3f9d12dd67c` (1 suite, 0 failures). - [x] Unit Test: `ListedTermsTest.*:CursorChainedPostings.*` passed (41 tests in 2 suites); all 9 source-added tests were listed and executed. No sanitizer report. - [x] Manual test: `./build.sh --be -j64` and `./build.sh --fe -j8` passed; isolated backend reported `Alive=true`, and `SELECT 1` succeeded. - Behavior changed: - [x] No. Unscored conjunction execution is faster; query results are unchanged. - Does this need documentation? - [x] No. ### Validation - `build-support/check-format.sh`: PASS. - Complete merged-tree format preflight against `origin/branch-4.2`: PASS. - Complete PR diff English description check: PASS (5 source files, no exceptions). - Branch-4.2 has no `build-support/run-clang-tidy.sh`; the branch's BE build and format gates passed. ### Hunk audit The source squash patch ID matches the final source PR head. The cherry-pick was clean, and every changed file has a byte-identical postimage to the source commit. | Source file | Disposition | | --- | --- | | `be/src/storage/index/inverted/query/engine/boolean_query/listed_terms.cpp` | Ported (verbatim) | | `be/src/storage/index/inverted/query/exec/cursor_chained_postings.cpp` | Ported (verbatim) | | `be/src/storage/index/inverted/query/exec/cursor_chained_postings.h` | Ported (verbatim) | | `be/test/storage/index/inverted/query/engine/boolean_query/listed_terms_test.cpp` | Ported (verbatim) | | `be/test/storage/index/inverted/query/exec/cursor_chained_postings_test.cpp` | Ported (verbatim) | `git range-diff` shows only the `-x` cherry-pick provenance line; the code patch is unchanged.
yiguolei
pushed a commit
that referenced
this pull request
Oct 20, 2026
…ngs (#69192) (#69244) ### What problem does this PR solve? Issue Number: CIR-22263 Related PRs: #69192 (source), #69206 (branch-4.2 prerequisite) Problem Summary: Compatible docs-only (T1) SNII postings currently fall back to rebuilding from raw columns when the destination does not request phrase positions. Backport the source PR's streamed merge support while retaining raw rebuild for incompatible or gram sources. Source squash commit: `8db05407dcf574c90da47579acb0ad7af57e027a`. The backport keeps its cherry-pick provenance and adds one branch-4.2 adaptation commit. The seven SNII implementation and focused test files have byte-identical postimages to the source commit. #### Hunk audit | Source file or hunk | Disposition | | --- | --- | | `be/src/storage/index/inverted/snii/compaction/eligibility.cpp` | Ported verbatim. | | `be/src/storage/index/inverted/snii/compaction/eligibility.h` | Ported verbatim. | | `be/src/storage/index/inverted/snii/compaction/posting_cursor.cpp` | Ported verbatim. | | `be/src/storage/index/inverted/snii/compaction/snii_index_compaction.cpp` | Ported verbatim. | | `be/src/storage/index/inverted/snii/compaction/snii_index_compaction.h` | Ported verbatim. | | `be/test/storage/index/inverted/snii/compaction/snii_compaction_eligibility_test.cpp` | Ported verbatim. | | `be/test/storage/index/inverted/snii/compaction/snii_index_compaction_test.cpp` | Ported verbatim. | | `be/src/storage/compaction/compaction.cpp` `@@ -1338 +1338,2 @@` | N/A: branch-4.2 has no `dest_segment_ids` mapping or `Rowset::segment()` API. Its compaction output and adjacent CLucene writer path use destination ordinals; nonzero start IDs are assigned only to transient partial-update writers. Kept the target's ordinal lookup. | | `be/test/storage/index/inverted/clucene/compaction/index_compaction_test.cpp` `@@ -1816,7 +1816 @@` | Ported: both sibling indexes must merge. | | Same file `@@ -1825,2 +1819,2 @@` | Adapted to the target's local segment numbering. | | Same file `@@ -1829 +1823 @@` | Ported: both index IDs must be eligible. | | Same file `@@ -1833,2 +1827,3 @@` | Adapted because the target helper has no physical segment start ID parameter. | | Same file `@@ -1840,2 +1835,4 @@` | Adapted to `Rowset::segment_path()` because `Rowset::segment()` is unavailable. | | Same file `@@ -1848 +1844,0 @@` | Ported: removed the obsolete raw-rebuild assertion comment. | | Same file `@@ -1850,2 +1846,2 @@` | Ported: checks document counts in each output segment. | | Same file `@@ -1853,0 +1850,10 @@` | Adapted: compares streamed and raw-rebuilt bytes by local segment ordinal. | `git range-diff` matches the source squash to the `-x` cherry-pick and shows one additional adaptation commit. The physical-ID lookup is omitted for the API and output segment model above; the docs-only eligibility and merger changes are preserved. ### Release note Compatible docs-only SNII indexes use streamed compaction on branch-4.2. ### Check List (For Author) - Test - [x] Unit Test: `SniiCompactionEligibilityTest.*:SniiIndexCompactionTest.*:IndexCompactionTest.snii_native_merge*` passed, 38/38 tests. The source-added docs-only cases and the sibling merge case ran. No ASAN or LeakSanitizer report appeared. - [x] Regression test: `test_storage_format_snii` passed, 1 suite, 0 failures, on an isolated local FE/BE cluster. - [x] Manual test: full FE and BE builds passed. The isolated cluster reported BE Alive=true and `SELECT 1` succeeded. - Behavior changed: - [x] Yes. Compatible docs-only postings are streamed rather than rebuilt from raw columns. - Does this need documentation? - [x] No. The complete committed PR diff passes the English description check. The merged target tree passes clang-format 16. The worktree is clean. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #68758, #67180, #67538, #67918
Problem Summary:
Backport #68758 to branch-4.2. MATCH and SEARCH now lower to a shared logical query engine for CLucene and SNII indexes. The backport preserves branch-4.2 APIs for scan-schema column binding, segment row counts, rowset segment traversal, and BKD type conversion. It passes the segment delete bitmap to the SEARCH Top-K collector so deleted rows do not consume the requested result count.
Source squash commit:
14fde1c1dc096f09c17af019c5e0cff3175da6c9(cherry-picked with-x). Branch-4.2 has no Kuromoji implementation, so the source-only Kuromoji code and metadata are omitted.The complete hunk audit accounts for all 2,641 source hunks:
git range-diffagainst the source squash commit was reviewed. The material differences are the branch-4.2 scan-schema and type API adaptations, direct includes required after the source-tree move, preservation of the branch-onlyRowIdStorageReadertest, and omission of Kuromoji and build-hygiene paths that do not exist on branch-4.2. The audit gives the disposition of every source hunk.The 1,036 non-Latin code literals in the PR diff are tokenizer, query, and Unicode test input or expected data. Each is bound to an exact finding ID in the shared pre-push English check. Comments, logs, errors, and diagnostics remain English.
Release note
MATCH and SEARCH use one query engine across V2 and SNII inverted indexes. SEARCH regexp, prefix, wildcard, phrase, scoring, and NULL behavior align across both formats as described in #68758.
Check List (For Author)
SELECT 1 AS readysucceededValidation
./build-support/check-format.sh: PASS../build.sh --be -j64(ASAN): PASS../build.sh --fe -j8(JDK 17, Checkstyle enabled): PASS../run-be-ut.sh --run --filter=... -j64(ASAN): 1,045 tests passed in 74 suites; all 461 enabled source-added tests executed; 16 tests are intentionally disabled by GTest.test_match_format_parity,test_search_format_parity,test_search_inverted_index_profile,test_search_not_null_bitmap,test_search_segment_document_ids). The temporary suite root contains byte-identical copies and uses the original golden data.vsearch.cppis blocked by an unmatchedNOLINTENDin unchangedbe/src/core/types.h:578; branch-4.2 has norun-clang-tidy.sh.origin/branch-4.2: PASS (1,000 changed files).