Repository navigation
branch-4.1: [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #64667 - #70550
Open
airborne12 wants to merge 4 commits into
Open
airborne12 wants to merge 4 commits into
airborne12 wants to merge 4 commits into
Conversation
…ogical analyzer apache#64667 ### What problem does this PR solve? Issue Number: apache#64646 Related PR: None Problem Summary: Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and `MATCH` / `MATCH_PHRASE` on Japanese columns end up with poor recall and precision. This PR adds a built-in `kuromoji` parser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column: ```sql INDEX content_idx (`content`) USING INVERTED PROPERTIES("parser" = "kuromoji", "parser_mode" = "search"); ``` After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms. How it works: - Native C++ under `be/src/storage/index/inverted/analyzer/kuromoji/`, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix. - The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed. - search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties. Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses). ### Release note Support Japanese text tokenization in the inverted index via a new kuromoji parser (`PROPERTIES("parser"="kuromoji")`), with `search/normal/extended` modes. Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) ```sql CREATE TABLE test_jp ( id BIGINT, content TEXT, INDEX idx_content (content) USING INVERTED PROPERTIES("parser" = "kuromoji", "parser_mode" = "search") ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num" = "1"); INSERT INTO test_jp VALUES (1, '東京都に住んでいます'), (2, '日本語の形態素解析エンジン'); -- Search mode also matches the Tokyo component of the compound. SELECT id FROM test_jp WHERE content MATCH '東京'; -- expect: 1 SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2 -- inspect segmentation directly SELECT TOKENIZE('東京都に住んでいます', '"parser"="kuromoji","parser_mode"="search"'); ``` (cherry picked from commit 1117132) Conflicts: be/test/CMakeLists.txt fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java
Backport the Kuromoji-related macOS arm64 linker fix from apache#67451. Use system malloc and supply the required gperftools symbols only for the dictionary converter. The independent SNII CRC change is outside this backport. Adapt the real-dictionary test to the source dictionary location, translate descriptions and diagnostics to English, and preserve Japanese input fixtures. Backport-of: 383be27 (Kuromoji subset; manually adapted)
Use designated aggregate initialization, signed-safe comparisons, ranges algorithms, and explicit reader length casts. Keep dictionary values, sort order, parser behavior, and Japanese fixtures unchanged. Keep the Viterbi traversal, offline generation steps, and shared dictionary fixture checks together, with focused readability annotations that explain their structure.
The target branch does not have the upstream macOS unit-test allocator fallback. Its unit-test source glob also includes testutil, so keep these stubs inactive there and enable them only for the dictionary converter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #64646
Related PR: #64667; Kuromoji portion of #67451 (#67448).
Problem Summary:
Backport the Japanese Kuromoji morphological analyzer to
branch-4.1. This includes dictionary generation and packaging, parser registration, FE property validation, and the original unit and regression tests. The analyzer remains disabled by default throughenable_kuromoji_analyzer; its default mode issearchwhen enabled.Include the follow-up macOS ARM64 dictionary-converter allocator repair from #67451. The independent SNII CRC repair from that PR is outside this backport's selected scope. The release branch lacks the upstream unit-test allocator workaround, so allocator stubs are added with a private converter-only guard.
Source feature squash:
11171329d6d0b56bf5d45b4f212aa311b92da946.Selected follow-up:
383be27d963c6bac3595e3528510348a65da0551(manually adapted Kuromoji subset).Candidate commits:
66e75a7b6f8e471a027c19cc1144de4ecb9bdd78branch-4.1: feature Add Japanese (Kuromoji) morphological analyzer [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #646678f2370e0b4f409cbb242af8cd55835c2581354fdfix Adapt Kuromoji dictionary tool and tests for branch-4.10c6a9ab5ed0562bd66035110850287e85285f4adchore Adapt Kuromoji code to branch static checks058a1d419658a302f658b45afd413961bca423f6fix Scope gperftools stubs to the Kuromoji dictionary toolRelease note
Add the optional Kuromoji Japanese analyzer with
normal,search, andextendedmodes for inverted indexes andTOKENIZE.Check List (For Author)
Validation
All commands ran in this branch's isolated worktree with JDK 17 and Linux ASAN builds.
./build.sh --be --fe./run-be-ut.sh --run --filter="Kuromoji*.*:DartsSmokeTest.*"./run-fe-ut.sh --run org.apache.doris.analysis.InvertedIndexKuromojiValidationTest,org.apache.doris.analysis.InvertedIndexPropertiesTestbash run-regression-test.sh --run -d inverted_index_p0,variant_p0/predefine -s test_japanese_analyzer,test_properties,test_variant_field_pattern_invalid_inverted_index -parallel 1be/src/core/types.h:577unmatchedNOLINTENDdiagnostic was reproduced with the identical target header and separately recorded.The four generated dictionaries are byte-identical before and after the C++ static-check adaptations. BE unit and cluster logs contain no ASAN, UBSan, or LeakSanitizer reports. No sanitizer suppression was added for this change. Three narrow static-check suppressions retain the Viterbi traversal, dictionary generation order, and shared dictionary-test fixture in one function each. The final tool-only compile definition has identical Linux preprocessed converter output with and without the definition. macOS ARM64 execution was not available on this Linux host. CI results are pending.
Backport review
Concrete
git range-diffmapping against the original feature squash:Differences from the source:
InvertedIndexPropertiesconstants/default-mode hunks and the matching utility alias already present from [feat](inverted-index) Backport IK custom analyzers to branch-4.1 #68550; retain the target analyzer-validation API.Original-source hunk audit (a whole-file row is used only when the committed file is byte-identical to the source):
.clang-format-ignore.clang-format-ignore..gitignore@@ -154,6 +154,9 @@ docker/runtime/be/resource/apache-doris/.licenserc.yaml@@ -112,6 +112,7 @@ header:LICENSE.txt@@ -369,6 +369,14 @@ be/src/olap/skiplist.h : BSD-style licenseNOTICE.txtNOTICE.txt.be/CMakeLists.txt@@ -374,6 +374,26 @@ install(DIRECTORYbe/CMakeLists.txt@@ -1099,6 +1119,49 @@ if (BUILD_META_TOOL OR BUILD_INDEX_TOOL)be/dict/kuromoji/README.mdbe/dict/kuromoji/README.md.be/src/common/config.cpp@@ -1420,6 +1420,8 @@ DEFINE_mInt64(snii_forced_spill_min_arena_bytes, "67108864");be/src/common/config.h@@ -1460,6 +1460,8 @@ DECLARE_mInt64(snii_forced_spill_min_arena_bytes);be/src/exprs/function/function_tokenize.cpp@@ -159,7 +159,7 @@ Status FunctionTokenize::execute_impl(FunctionContext* /*context*/, Block& blockbe/src/storage/index/inverted/analyzer/analyzer.cpp@@ -39,6 +39,7 @@be/src/storage/index/inverted/analyzer/analyzer.cpp@@ -88,7 +89,8 @@ bool InvertedIndexAnalyzer::is_builtin_analyzer(const std::string& analyzer_namebe/src/storage/index/inverted/analyzer/analyzer.cpp@@ -126,6 +128,22 @@ AnalyzerPtr InvertedIndexAnalyzer::create_builtin_analyzer(InvertedIndexParserTybe/src/storage/index/inverted/analyzer/kuromoji/KuromojiAnalyzer.hbe/src/storage/index/inverted/analyzer/kuromoji/KuromojiAnalyzer.h.be/src/storage/index/inverted/analyzer/kuromoji/KuromojiMode.hbe/src/storage/index/inverted/analyzer/kuromoji/KuromojiMode.h.be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.cpp@@ -0,0 +1,207 @@be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.hbe/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.h.be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.hbe/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h.be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dict_format.hbe/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dict_format.h.be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary.cpp@@ -0,0 +1,325 @@be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary.h@@ -0,0 +1,140 @@be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.cpp@@ -0,0 +1,263 @@be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.hbe/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h.be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.cpp@@ -0,0 +1,301 @@be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.hbe/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h.be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.hbe/src/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.h.be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.cpp@@ -0,0 +1,301 @@be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.hbe/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.h.be/src/storage/index/inverted/inverted_index_parser.cpp@@ -39,6 +39,8 @@ std::string inverted_index_parser_type_to_string(InvertedIndexParserType parser_be/src/storage/index/inverted/inverted_index_parser.cpp@@ -62,6 +64,8 @@ InvertedIndexParserType get_inverted_index_parser_type_from_string(const std::stbe/src/storage/index/inverted/inverted_index_parser.cpp@@ -89,8 +93,17 @@ std::string get_parser_mode_string_from_properties(be/src/storage/index/inverted/inverted_index_parser.h@@ -44,7 +44,8 @@ enum class InvertedIndexParserType {be/src/storage/index/inverted/inverted_index_parser.h@@ -80,6 +81,10 @@ const std::string INVERTED_INDEX_PARSER_CHINESE = "chinese";be/src/tools/kuromoji_build_dict.cpp@@ -0,0 +1,202 @@be/test/CMakeLists.txt@@ -161,6 +161,12 @@ if (TARGET phdr_cache_test_dso)be/test/storage/index/inverted/analyzer/kuromoji/darts_smoke_test.cpp@@ -0,0 +1,70 @@be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dict_format_test.cppbe/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dict_format_test.cpp.be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dictionary_builder_test.cpp@@ -0,0 +1,74 @@be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dictionary_test.cpp@@ -0,0 +1,155 @@be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_ipadic_parser_test.cpp@@ -0,0 +1,109 @@be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize_test.cpp@@ -0,0 +1,50 @@be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_real_dict_test.cpp@@ -0,0 +1,177 @@be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi_test.cpp@@ -0,0 +1,256 @@be/test/storage/index/inverted/analyzer/kuromoji_analyzer_test.cppbe/test/storage/index/inverted/analyzer/kuromoji_analyzer_test.cpp.be/test/storage/index/inverted/analyzer/kuromoji_tokenizer_test.cpp@@ -0,0 +1,39 @@be/test/storage/index/inverted/kuromoji_parser_type_test.cppbe/test/storage/index/inverted/kuromoji_parser_type_test.cpp.build.sh@@ -606,6 +606,21 @@ update_submodule() {dist/LICENSE-dist.txt@@ -1532,3 +1532,5 @@ Other dependencies:dist/licenses/LICENSE-darts-clone.txtdist/licenses/LICENSE-darts-clone.txt.dist/licenses/LICENSE-ipadic.txt@@ -0,0 +1,71 @@fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java@@ -35,11 +35,15 @@ public class InvertedIndexProperties {InvertedIndexProperties:INVERTED_INDEX_PARSER_KUROMOJI, its mode constants, and the parser/analyzer default ofsearch. The target file has no new diff.fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java@@ -81,13 +85,24 @@ public static String getInvertedIndexParserMode(Map<String, String> properties)InvertedIndexProperties:INVERTED_INDEX_PARSER_KUROMOJI, its mode constants, and the parser/analyzer default ofsearch. The target file has no new diff.fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java@@ -54,6 +54,7 @@ public class InvertedIndexUtil {fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java@@ -147,7 +148,8 @@ public static void checkInvertedIndexParser(String indexColName, PrimitiveType cfe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java@@ -255,9 +257,10 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java@@ -271,8 +274,13 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java@@ -72,7 +72,7 @@ public class IndexPolicy implements Writable, GsonPostProcessable {fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexKuromojiValidationTest.javafe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexKuromojiValidationTest.java.fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java@@ -105,6 +105,40 @@ public void testGetParserModeDefaultForNonIk() {regression-test/suites/inverted_index_p0/analyzer/test_japanese_analyzer.groovy@@ -0,0 +1,114 @@regression-test/suites/inverted_index_p0/test_properties.groovy@@ -84,7 +84,7 @@ suite("test_properties", "p0"){regression-test/suites/variant_p0/predefine/test_variant_field_pattern_invalid_inverted_index.groovyregression-test/suites/variant_p0/predefine/test_variant_field_pattern_invalid_inverted_index.groovy.thirdparty/build-thirdparty.sh@@ -2282,6 +2282,18 @@ build_icu() {thirdparty/build-thirdparty.sh@@ -2568,6 +2580,7 @@ if [[ "${#packages[@]}" -eq 0 ]]; thenthirdparty/build-thirdparty.sh@@ -2675,6 +2688,7 @@ cleanup_package_source() {thirdparty/vars.sh@@ -563,6 +563,12 @@ ICU_NAME=release-69-1.tar.gzthirdparty/vars.sh@@ -590,6 +596,7 @@ LANCE_C_MD5SUM="15ef7cd20a2e1606384251cb2d41d42f"Selected Kuromoji follow-up: #67451 (
383be27d963c6bac3595e3528510348a65da0551).be/CMakeLists.txt@@ -867,7 +867,8 @@ if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR "${CMAKE_BUILD_TYPE}" STREQUAL "REbe/CMakeLists.txt@@ -1100,7 +1101,25 @@ endif()be/CMakeLists.txt@@ -1109,7 +1128,7 @@ if (NOT MAKE_TEST)be/src/storage/index/snii/encoding/crc32c.cpp@@ -34,10 +34,15 @@be/test/testutil/gperftools_stubs.cpp@@ -15,13 +15,14 @@Check List (For Reviewer who merge this PR)