branch-4.2: [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #64667 by airborne12 · Pull Request #70555 · apache/doris

airborne12 · 2026-11-13T07:43:30Z

What problem does this PR solve?

Issue Number: #64646

Related PR: #64667; Kuromoji portion of #67451 (#67448).

Problem Summary:

Backport the Japanese Kuromoji morphological analyzer to branch-4.2. This includes dictionary generation and packaging, parser registration, FE property validation, and the original unit and regression tests. The analyzer remains disabled by default through enable_kuromoji_analyzer; its default mode is search when enabled.

Include the follow-up macOS ARM64 dictionary-converter allocator repair from #67451. The independent SNII CRC repair from that PR is outside this backport's selected scope. The release branch lacks the upstream unit-test allocator workaround, so allocator stubs are added with a private converter-only guard.

The target moved analyzer/parser sources under storage/index/inverted/analysis/. The backport follows that layout and adds Kuromoji width/case normalization to the target query-term normalization API, matching upstream #68758.

Source feature squash: 11171329d6d0b56bf5d45b4f212aa311b92da946.
Selected follow-up: 383be27d963c6bac3595e3528510348a65da0551 (manually adapted Kuromoji subset).

Candidate commits:

97b310e3573fb645eaf9330659f020dc250c2255 branch-4.2: feature Add Japanese (Kuromoji) morphological analyzer [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #64667
3761a2a3b16d3fbb14601e0d7e47145031c6ed0e fix Adapt Kuromoji dictionary tool and tests for branch-4.2
2f54e38f41533487b7fa072ab63175f2f8a60e94 chore Adapt Kuromoji code to branch static checks
fbb6c3cbe8e0db329b85cc11bef0043d6fa50553 fix Scope gperftools stubs to the Kuromoji dictionary tool
ecd2e3a6a1ed831ea099725a1abdf4592af433ac fix Adapt Kuromoji query normalization to branch-4.2

Release note

Add the optional Kuromoji Japanese analyzer with normal, search, and extended modes for inverted indexes and TOKENIZE.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (dictionary byte comparison and static checks below)
Behavior changed:
- Yes. Add an optional Japanese analyzer; existing parser behavior is retained.
Does this need documentation?
- Yes. Upstream documentation: [docs] Document the kuromoji (Japanese) inverted-index analyzer doris-website#3946

Validation

All commands ran in this branch's isolated worktree with JDK 17 and Linux ASAN builds.

Check	Result
`./build.sh --be --fe`	Passed. Dictionary converter and all four packaged dictionary files generated successfully.
`./run-be-ut.sh --run --filter="Kuromoji.:DartsSmokeTest.*"`	40/40 passed, including all 39 original source tests and the three real-dictionary tests; no skips.
`./run-fe-ut.sh --run org.apache.doris.analysis.InvertedIndexKuromojiValidationTest,org.apache.doris.analysis.InvertedIndexPropertiesTest`	34/34 passed, including all six source-added methods; no skips.
`bash run-regression-test.sh --run -d inverted_index_p0,variant_p0/predefine -s test_japanese_analyzer,test_properties,test_variant_field_pattern_invalid_inverted_index -parallel 1`	3/3 suites passed, zero failures/fatal scripts/skips, against the independently built local FE/BE.
clang-format 16 and JDK 17 Maven Checkstyle	Passed for the complete candidate merged with the target branch.
clang-tidy	Changed-line checks passed for changed C++ files. One unchanged baseline `be/src/core/types.h:578` unmatched `NOLINTEND` diagnostic was reproduced with the identical target header and separately recorded.
English descriptions	Passed across the complete PR diff. Ten exact Japanese input/expected-data literals were individually reviewed; comments and diagnostics are English.

All four generated dictionaries are byte-identical to the independently built and tested branch-4.1 backport. BE unit and cluster logs contain no ASAN, UBSan, or LeakSanitizer reports. No sanitizer suppression was added for this change. Three narrow static-check suppressions retain the Viterbi traversal, dictionary generation order, and shared dictionary-test fixture in one function each. The Linux dictionary-tool build passed with the private tool-only compile definition. macOS ARM64 execution was not available on this Linux host. CI results are pending.

Backport review

Concrete git range-diff mapping against the original feature squash:

1:  11171329d6d ! 1:  97b310e3573 [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer (#64667)
-:  ----------- > 2:  3761a2a3b16 [fix](build) Adapt Kuromoji dictionary tool and tests for branch-4.2
-:  ----------- > 3:  2f54e38f415 [chore](be) Adapt Kuromoji code to branch static checks
-:  ----------- > 4:  fbb6c3cbe8e [fix](build) Scope gperftools stubs to the Kuromoji dictionary tool
-:  ----------- > 5:  ecd2e3a6a1e [fix](inverted-index) Adapt Kuromoji query normalization to branch-4.2

Differences from the source:

Move the analyzer/parser sources and tests into the target analysis layout and update include/license/ignore references. Add the Kuromoji width/Unicode-case normalization arm from upstream 14fde1c1dc096f09c17af019c5e0cff3175da6c9 ([refactor](be) Unify MATCH and SEARCH execution across inverted index formats #68758), with a focused parser/analyzer-alias test.
Preserve the target's BE unit-test link configuration and existing FE property tests while adding all source tests.
Omit duplicate InvertedIndexProperties constants/default-mode hunks and the matching utility alias already present from [feat](inverted-index) Backport IK custom analyzers to branch-4.1 #68550; retain the target analyzer-validation API.
Keep target context around configuration declarations, third-party dependency lists, and test placement. These context differences do not change the added feature.
Apply the selected allocator repair, scope its stubs to the dictionary converter, and retain dictionary generation/install guards.
Translate descriptive comments and diagnostic labels into English while preserving Japanese data. Resolve the real-dictionary analyzer test through the same dictionary source path as its other tests.
Use equivalent C++ static-check syntax and remove one trailing blank license line. Original feature behavior and dictionary bytes are retained.

Original-source hunk audit (a whole-file row is used only when the committed file is byte-identical to the source):

Source file / original hunk	Result	Evidence / adaptation
`.clang-format-ignore` `@@ -9,4 +9,5 @@ be/src/util/sse2neon.h`	Adapted	Retain source metadata and vendored exclusions; update Kuromoji references to the target analysis directory.
`.gitignore` `@@ -154,6 +154,9 @@ docker/runtime/be/resource/apache-doris/`	Ported	Original added behavior retained with target context.
`.licenserc.yaml` `@@ -112,6 +112,7 @@ header:`	Adapted	Retain source metadata and vendored exclusions; update Kuromoji references to the target analysis directory.
`LICENSE.txt` `@@ -369,6 +369,14 @@ be/src/olap/skiplist.h : BSD-style license`	Adapted	Retain source metadata and vendored exclusions; update Kuromoji references to the target analysis directory.
`NOTICE.txt` `@@ -73,6 +73,9 @@ This software includes third party software subject to the following copyrights:`	Adapted	Retain source metadata and vendored exclusions; update Kuromoji references to the target analysis directory.
`be/CMakeLists.txt` `@@ -374,6 +374,26 @@ install(DIRECTORY`	Adapted	Retain dictionary generation/packaging; add the selected #67451 allocator fix and scope stubs to the converter with a private compile definition.
`be/CMakeLists.txt` `@@ -1099,6 +1119,49 @@ if (BUILD_META_TOOL OR BUILD_INDEX_TOOL)`	Adapted	Retain dictionary generation/packaging; add the selected #67451 allocator fix and scope stubs to the converter with a private compile definition.
`be/dict/kuromoji/README.md`	Ported(verbatim)	Target: `be/dict/kuromoji/README.md`.
`be/src/common/config.cpp` `@@ -1420,6 +1420,8 @@ DEFINE_mInt64(snii_forced_spill_min_arena_bytes, "67108864");`	Ported	Original added behavior retained with target context.
`be/src/common/config.h` `@@ -1460,6 +1460,8 @@ DECLARE_mInt64(snii_forced_spill_min_arena_bytes);`	Ported	Original added behavior retained with target context.
`be/src/exprs/function/function_tokenize.cpp` `@@ -159,7 +159,7 @@ Status FunctionTokenize::execute_impl(FunctionContext* /context/, Block& block`	Ported	Original added behavior retained with target context.
`be/src/storage/index/inverted/analyzer/analyzer.cpp` `@@ -39,6 +39,7 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/analyzer.cpp` and update include paths. Retain the target normalization API and add the upstream #68758 Kuromoji width/Unicode-case branch.
`be/src/storage/index/inverted/analyzer/analyzer.cpp` `@@ -88,7 +89,8 @@ bool InvertedIndexAnalyzer::is_builtin_analyzer(const std::string& analyzer_name`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/analyzer.cpp` and update include paths. Retain the target normalization API and add the upstream #68758 Kuromoji width/Unicode-case branch.
`be/src/storage/index/inverted/analyzer/analyzer.cpp` `@@ -126,6 +128,22 @@ AnalyzerPtr InvertedIndexAnalyzer::create_builtin_analyzer(InvertedIndexParserTy`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/analyzer.cpp` and update include paths. Retain the target normalization API and add the upstream #68758 Kuromoji width/Unicode-case branch.
`be/src/storage/index/inverted/analyzer/kuromoji/KuromojiAnalyzer.h` `@@ -0,0 +1,75 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/KuromojiAnalyzer.h` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/KuromojiMode.h`	Ported(verbatim)	Target: `be/src/storage/index/inverted/analysis/analyzer/kuromoji/KuromojiMode.h`.
`be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.cpp` `@@ -0,0 +1,207 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/KuromojiTokenizer.cpp` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.h` `@@ -0,0 +1,53 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/KuromojiTokenizer.h` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h`	Ported(verbatim)	Target: `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/darts.h`.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dict_format.h`	Ported(verbatim)	Target: `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_dict_format.h`.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary.cpp` `@@ -0,0 +1,325 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_dictionary.cpp` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary.h` `@@ -0,0 +1,140 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_dictionary.h` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.cpp` `@@ -0,0 +1,263 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_dictionary_builder.cpp` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h` `@@ -0,0 +1,75 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.cpp` `@@ -0,0 +1,301 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_ipadic_parser.cpp` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h` `@@ -0,0 +1,50 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.h`	Ported(verbatim)	Target: `be/src/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_normalize.h`.
`be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.cpp` `@@ -0,0 +1,301 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_viterbi.cpp` and update include paths.
`be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.h` `@@ -0,0 +1,64 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_viterbi.h` and update include paths.
`be/src/storage/index/inverted/inverted_index_parser.cpp` `@@ -39,6 +39,8 @@ std::string inverted_index_parser_type_to_string(InvertedIndexParserType parser_`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/inverted_index_parser.cpp` and update include paths.
`be/src/storage/index/inverted/inverted_index_parser.cpp` `@@ -62,6 +64,8 @@ InvertedIndexParserType get_inverted_index_parser_type_from_string(const std::st`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/inverted_index_parser.cpp` and update include paths.
`be/src/storage/index/inverted/inverted_index_parser.cpp` `@@ -89,8 +93,17 @@ std::string get_parser_mode_string_from_properties(`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/inverted_index_parser.cpp` and update include paths.
`be/src/storage/index/inverted/inverted_index_parser.h` `@@ -44,7 +44,8 @@ enum class InvertedIndexParserType {`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/inverted_index_parser.h` and update include paths.
`be/src/storage/index/inverted/inverted_index_parser.h` `@@ -80,6 +81,10 @@ const std::string INVERTED_INDEX_PARSER_CHINESE = "chinese";`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/src/storage/index/inverted/analysis/inverted_index_parser.h` and update include paths.
`be/src/tools/kuromoji_build_dict.cpp` `@@ -0,0 +1,202 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required.
`be/test/CMakeLists.txt` `@@ -161,6 +161,12 @@ if (TARGET phdr_cache_test_dso)`	Adapted	Add the source dictionary path definition while retaining target unit-test link settings.
`be/test/storage/index/inverted/analyzer/kuromoji/darts_smoke_test.cpp` `@@ -0,0 +1,70 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/darts_smoke_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dict_format_test.cpp` `@@ -0,0 +1,48 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_dict_format_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dictionary_builder_test.cpp` `@@ -0,0 +1,74 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_dictionary_builder_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dictionary_test.cpp` `@@ -0,0 +1,155 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_dictionary_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_ipadic_parser_test.cpp` `@@ -0,0 +1,109 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_ipadic_parser_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize_test.cpp` `@@ -0,0 +1,50 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_normalize_test.cpp` and update include paths. Add QueryTermWidthAndUnicodeCase to check parser/analyzer aliases with default, true, and false lowercasing through the target normalization API.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_real_dict_test.cpp` `@@ -0,0 +1,177 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Resolve the analyzer dictionary through the same source path as the other real-dictionary tests. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_real_dict_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi_test.cpp` `@@ -0,0 +1,256 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji/kuromoji_viterbi_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji_analyzer_test.cpp` `@@ -0,0 +1,66 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji_analyzer_test.cpp` and update include paths.
`be/test/storage/index/inverted/analyzer/kuromoji_tokenizer_test.cpp` `@@ -0,0 +1,39 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/analyzer/kuromoji_tokenizer_test.cpp` and update include paths.
`be/test/storage/index/inverted/kuromoji_parser_type_test.cpp` `@@ -0,0 +1,44 @@`	Adapted	Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Relocate to `be/test/storage/index/inverted/analysis/kuromoji_parser_type_test.cpp` and update include paths.
`build.sh` `@@ -606,6 +606,21 @@ update_submodule() {`	Ported	Original added behavior retained with target context.
`dist/LICENSE-dist.txt` `@@ -1532,3 +1532,5 @@ Other dependencies:`	Ported	Original added behavior retained with target context.
`dist/licenses/LICENSE-darts-clone.txt`	Ported(verbatim)	Target: `dist/licenses/LICENSE-darts-clone.txt`.
`dist/licenses/LICENSE-ipadic.txt` `@@ -0,0 +1,71 @@`	Adapted	License text retained; remove only a trailing blank line required by the whitespace check.
`fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java` `@@ -35,11 +35,15 @@ public class InvertedIndexProperties {`	N-A	Already present from #68550 in target `InvertedIndexProperties`: `INVERTED_INDEX_PARSER_KUROMOJI`, its mode constants, and the parser/analyzer default of `search`. The target file has no new diff.
`fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java` `@@ -81,13 +85,24 @@ public static String getInvertedIndexParserMode(Map<String, String> properties)`	N-A	Already present from #68550 in target `InvertedIndexProperties`: `INVERTED_INDEX_PARSER_KUROMOJI`, its mode constants, and the parser/analyzer default of `search`. The target file has no new diff.
`fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java` `@@ -54,6 +54,7 @@ public class InvertedIndexUtil {`	N-A	The Kuromoji parser alias already exists in the target from #68550; the remaining source validation changes are included below.
`fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java` `@@ -147,7 +148,8 @@ public static void checkInvertedIndexParser(String indexColName, PrimitiveType c`	Ported	Retain Kuromoji parser and mode validation with the existing target analyzer-validation API.
`fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java` `@@ -255,9 +257,10 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,`	Ported	Retain Kuromoji parser and mode validation with the existing target analyzer-validation API.
`fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java` `@@ -271,8 +274,13 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,`	Ported	Retain Kuromoji parser and mode validation with the existing target analyzer-validation API.
`fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java` `@@ -72,7 +72,7 @@ public class IndexPolicy implements Writable, GsonPostProcessable {`	Ported	Original added behavior retained with target context.
`fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexKuromojiValidationTest.java`	Ported(verbatim)	Target: `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexKuromojiValidationTest.java`.
`fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java` `@@ -105,6 +105,40 @@ public void testGetParserModeDefaultForNonIk() {`	Adapted	Add all four source Kuromoji mode tests alongside the existing target tests.
`regression-test/suites/inverted_index_p0/analyzer/test_japanese_analyzer.groovy` `@@ -0,0 +1,114 @@`	Adapted	Preserve the complete test behavior and Japanese inputs; translate descriptive comments into English.
`regression-test/suites/inverted_index_p0/test_properties.groovy` `@@ -84,7 +84,7 @@ suite("test_properties", "p0"){`	Ported	Original added behavior retained with target context.
`regression-test/suites/variant_p0/predefine/test_variant_field_pattern_invalid_inverted_index.groovy`	Ported(verbatim)	Target: `regression-test/suites/variant_p0/predefine/test_variant_field_pattern_invalid_inverted_index.groovy`.
`thirdparty/build-thirdparty.sh` `@@ -2282,6 +2282,18 @@ build_icu() {`	Ported	Original added behavior retained with target context.
`thirdparty/build-thirdparty.sh` `@@ -2568,6 +2580,7 @@ if [[ "${#packages[@]}" -eq 0 ]]; then`	Ported	Original added behavior retained with target context.
`thirdparty/build-thirdparty.sh` `@@ -2675,6 +2688,7 @@ cleanup_package_source() {`	Ported	Original added behavior retained with target context.
`thirdparty/vars.sh` `@@ -563,6 +563,12 @@ ICU_NAME=release-69-1.tar.gz`	Ported	Original added behavior retained with target context.
`thirdparty/vars.sh` `@@ -590,6 +596,7 @@ LANCE_C_MD5SUM="15ef7cd20a2e1606384251cb2d41d42f"`	Ported	Original added behavior retained with target context.

Selected Kuromoji follow-up: #67451 (383be27d963c6bac3595e3528510348a65da0551).

Source file / original hunk	Result	Evidence / adaptation
`be/CMakeLists.txt` `@@ -867,7 +867,8 @@ if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR "${CMAKE_BUILD_TYPE}" STREQUAL "RE`	N-A	This source hunk only updates a comment in the unit-test allocator fallback, which the target does not have.
`be/CMakeLists.txt` `@@ -1100,7 +1101,25 @@ endif()`	Adapted	Apply the macOS ARM64 system-allocator override only to kuromoji_build_dict; retain dictionary generation and install guards. Use a private tool-only definition for the stubs.
`be/CMakeLists.txt` `@@ -1109,7 +1128,7 @@ if (NOT MAKE_TEST)`	Adapted	Apply the macOS ARM64 system-allocator override only to kuromoji_build_dict; retain dictionary generation and install guards. Use a private tool-only definition for the stubs.
`be/src/storage/index/snii/encoding/crc32c.cpp` `@@ -34,10 +34,15 @@`	N-A	Independent SNII CRC repair explicitly excluded by the user; no SNII file is changed.
`be/test/testutil/gperftools_stubs.cpp` `@@ -15,13 +15,14 @@`	Adapted	The target lacks the upstream unit-test allocator fallback. Add its four required stub definitions with a private KUROMOJI_DICT_TOOL guard; keep target unit-test linking unchanged.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…ogical analyzer apache#64667 Issue Number: apache#64646 Related PR: None Problem Summary: Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and `MATCH` / `MATCH_PHRASE` on Japanese columns end up with poor recall and precision. This PR adds a built-in `kuromoji` parser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column: ```sql INDEX content_idx (`content`) USING INVERTED PROPERTIES("parser" = "kuromoji", "parser_mode" = "search"); ``` After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms. How it works: - Native C++ under `be/src/storage/index/inverted/analyzer/kuromoji/`, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix. - The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed. - search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties. Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses). Support Japanese text tokenization in the inverted index via a new kuromoji parser (`PROPERTIES("parser"="kuromoji")`), with `search/normal/extended` modes. Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) ```sql CREATE TABLE test_jp ( id BIGINT, content TEXT, INDEX idx_content (content) USING INVERTED PROPERTIES("parser" = "kuromoji", "parser_mode" = "search") ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num" = "1"); INSERT INTO test_jp VALUES (1, '東京都に住んでいます'), (2, '日本語の形態素解析エンジン'); -- Search mode also matches the Tokyo component of the compound. SELECT id FROM test_jp WHERE content MATCH '東京'; -- expect: 1 SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2 -- inspect segmentation directly SELECT TOKENIZE('東京都に住んでいます', '"parser"="kuromoji","parser_mode"="search"'); ``` (cherry picked from commit 1117132) Conflicts: be/test/CMakeLists.txt fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java (cherry picked from commit 66e75a7)

Backport the Kuromoji-related macOS arm64 linker fix from apache#67451. Use system malloc and supply the required gperftools symbols only for the dictionary converter. The independent SNII CRC change is outside this backport. Adapt the real-dictionary test to the source dictionary location, translate descriptions and diagnostics to English, and preserve Japanese input fixtures. Backport-of: 383be27 (Kuromoji subset; manually adapted) (cherry picked from commit 8f2370e)

Use designated aggregate initialization, signed-safe comparisons, ranges algorithms, and explicit reader length casts. Keep dictionary values, sort order, parser behavior, and Japanese fixtures unchanged. Keep the Viterbi traversal, offline generation steps, and shared dictionary fixture checks together, with focused readability annotations that explain their structure. (cherry picked from commit 0c6a9ab)

The target branch does not have the upstream macOS unit-test allocator fallback. Its unit-test source glob also includes testutil, so keep these stubs inactive there and enable them only for the dictionary converter. (cherry picked from commit 058a1d4)

### What problem does this PR solve? Issue Number: apache#64646 Related PR: apache#64667, apache#68758, apache#69206 Problem Summary: The target branch moved analyzers and added a query-term normalization API. The feature backport must fold Kuromoji width and Unicode case through that API, as the upstream refactor does. Add a focused test for the parser and analyzer alias with default, enabled, and disabled lowercasing. Backport-of: 14fde1c (Kuromoji normalization subset; manually adapted) ### Release note Kuromoji query literals follow the same width and case rules as indexed terms. ### Testing clang-format 16 passed. Build and focused unit/regression results are recorded in the pull request after execution.

hello-stephen · 2026-11-13T23:14:32Z

run buildall

RiseOfPrime and others added 5 commits November 13, 2026 14:18

airborne12 requested a review from yiguolei as a code owner November 13, 2026 07:43

Navigation Menu

Repository navigation

branch-4.2: [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #64667 - #70555

branch-4.2: [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #64667#70555
airborne12 wants to merge 5 commits into
apache:branch-4.2from
airborne12:pick-64667-67451-branch-4.2

airborne12 commented Nov 13, 2026

Uh oh!

hello-stephen commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

airborne12 commented Nov 13, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Validation

Backport review

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants