[Mryange]

Background

Tablet write validation can truncate strings, round legacy decimals, and deduplicate map keys. Expression outputs can share column trees, so validation must preserve copy-on-write ownership before changing them.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

Writing a shared complex column can modify another output column or leave stale views after replacing a nested string. Extract tablet validation into a standalone dispatcher that consumes and returns a column, detaches shared nodes once, and recursively validates writable child slots. Add ownership and nested-value tests; preserve historical DecimalV2 precision/scale behavior.

Details

• Change map: About 550 lines of production code replace the converter's embedded validator; about 530 lines of unit/regression tests plus generated regression output. The validator separates scalar checks, nullable handling, and Array/Map/Struct traversal; the converter retains block-level orchestration.
• Frequency: When writable validation encounters shared column storage, including one map expression reused by multiple output columns.
• Root cause: Mutable access through const casts bypassed COW; nested replacement could invalidate cached column views. Temporary owning pointers can also cause unnecessary cloning.
• Why existing tests missed it: They did not check shared outputs together with nested replacement or verify unique column storage addresses and reference counts.
• Fix: Consume the input owner, perform COW once at the public boundary, and borrow mutable child slots during recursion. Propagate inherited NULLs and original-row mappings. Deduplicate each map once when visited.
• Other paths and branches: The common tablet converter covers cloud and shared-nothing writes. Automatic picks to 4.1 and 4.2 conflict in vtablet_block_convertor.cpp; no release pick is requested.
• Behavior change: Shared source columns remain unchanged during validation. Nested nullable validation preserves outer-row filtering. DecimalV2 retains its historical execution precision/scale (27, 9); declared-schema enforcement and map-key collisions caused by string truncation remain outside this fix.

Testing

On the development machine:

• BUILD_TYPE=ASAN sh build.sh --be -j48: passed for the final revision.
• BUILD_TYPE_UT=ASAN sh run-be-ut.sh --run --filter='TabletColumnValidatorTest.*:ColumnNullableTest.*:ColumnArrayTest.*:ColumnStructTest.*:ColumnMapTest.*' -j48: 107 tests passed, including all 22 new validator tests.
• Repository clang-format 16 scripts, build hygiene, git diff --check, and clang-tidy on both changed source TUs and the new UT: passed.
• The existing shared-map regression and generated output are retained; that regression was not rerun after this refactoring.

Release note

Fix incorrect results or failures when inserting shared complex columns that require tablet validation.

Check List (For Author)

• Behavior changed: Yes (see Details)
• Does this need documentation: No
• Pre-submission status: Partially verified: ASAN build and focused/related UTs passed; end-to-end regression not rerun after refactoring.

AI Disclosure

• Tool: Codex (GPT-6).
• Generated: Code, tests, and this description.
• Human review: The author reviewed the design and an earlier source version; the final changes remain available for review.

[Mryange]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/42774480635

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[HappenLee]

Potential performance regression: validation can clone columns that are already exclusively owned.

The shared-column isolation is needed for correctness, but the new helper introduces two extra owners before the COW decision in _internal_validate_column():

ColumnPtr real_column_ptr =
        column_ptr == nullptr ? column : column_ptr->get_nested_column_ptr();

// Inside mutate_column():
auto mutable_column = IColumn::mutate(column);

For an initially exclusive, non-nullable column, the reference count is 1 in the owning slot, 2 after constructing real_column_ptr, and 3 when the lvalue column is copied into the by-value parameter of IColumn::mutate(Ptr ptr). Consequently, shallow_mutate() takes the clone() branch even though there is no other consumer that needs the old column. The ColumnPtr& validation parameter itself does not add an owner.

static MutablePtr mutate(Ptr ptr) {
    MutablePtr res = ptr->shallow_mutate(); // Clone decision happens here.
    ptr.reset();                          // Too late to avoid that clone.
    res->mutate_subcolumns();
    return res;
}

For nullable columns, the local alias additionally retains the nested column while the by-value parameter retains the wrapper. Also, mutate_subcolumns() recursively detaches descendants. The ARRAY/MAP/STRUCT validation branches then recurse and repeat the same ownership pattern, potentially copying already-detached descendants again. Even read-only validation of an exclusively owned ARRAY<INT> enters mutate_column() eagerly.

This can add column-data copying, allocation, memory bandwidth use, and peak memory, especially for large or deeply nested complex columns. Copying the smart pointers only increments reference counts; the subsequent COW calls cause the column clones.

Suggested changes:

1. Use non-owning, read-only views (for example const IColumn*) for real_column_ptr, and refresh them after replacing the owning column.
2. Acquire mutable ownership by consuming the actual owner through the existing scoped ownership mechanism where applicable, preserving restoration on early returns and exceptions. Changing only the call to IColumn::mutate(std::move(column)) is insufficient while the owning real_column_ptr alias is still alive.
3. Once a column tree has been made mutable, avoid repeating whole-subtree COW at each recursive validation level, and defer mutation for paths that only need to inspect values. Keep the necessary COW isolation for genuinely shared parents and descendants.

Please add ownership/allocation coverage alongside the SQL correctness regression: exclusive columns and descendants should be reused when possible; shared outputs must remain isolated during string truncation; nested ARRAY/MAP/STRUCT inputs should not repeatedly clone the same already-detached payload.

Validation: traced the COW implementation and reproduced the extra-clone pattern in an isolated model using its reference-counting implementation. This is not a full BE benchmark, so no SQL slowdown or speedup percentage is claimed.