[ti-chi-bot]

This is an automated cherry-pick of #11207

What problem does this PR solve?

Issue Number: close #11206

Problem Summary:

Array bulk deserialization could accept overflowing or decreasing offsets, or non-zero offsets with missing element data. For example, position-independent sizes 2^63 and 2^63 + 4 wrap to offsets [2^63, 4]. Checking only the last offset against the nested column does not detect the earlier invalid offset.

These offsets can cause out-of-bounds reads in consumers such as vector casts and distance functions. TiDB VECTOR uses Array(Float32), and the bulk decoder is used by MPP exchange chunks and columnar storage reads. Large element counts can also overflow buffer allocation arithmetic and produce an undersized allocation.

What is changed and how it works?

datatypes: validate array bulk deserialization

Reject corrupt offsets, inconsistent nested columns, and overflowing buffer allocation sizes during bulk decoding.

• Check initial_size + limit before resizing the offsets column, and check cumulative offset addition before storing each offset.
• Reject decreasing absolute offsets.
• Always require the nested column length to equal the last offset. Recursively check Tuple and Nullable component lengths; nested Arrays validate their own elements during decoding.
• Guard PODArray constructor and reserve/resize allocation sizes against overflow in element-byte multiplication, padding addition, and power-of-two rounding. Share the allocation limits with existing prechecks.

Valid empty arrays with zero offsets remain supported. The inherited Nested ALTER exception allowing non-zero offsets with an empty nested column is removed. DeltaMerge handles columns absent after DDL in DMFileReader::readColumn by creating default values directly.

The new offset-count, cumulative-offset, monotonicity, and nested-length checks throw LOGICAL_ERROR. PODArray allocation-size overflow checks throw TOO_LARGE_ARRAY_SIZE. A decoding failure may leave partially decoded data, so callers must discard the destination column. Representable requests can still attempt large allocations.

Position-independent decoding retains the original preallocation and indexed writes, then shrinks the offsets to the number of sizes actually read at EOF. It adds one offset-count overflow check per batch and one cumulative-offset overflow check per row. Absolute offsets receive an additional sequential validation pass.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Unit-test coverage includes both offset encodings, single and separate streams, batch appends, early EOF, offset-count and cumulative-offset overflow, missing or truncated elements and components, valid empty arrays, and PODArray allocation boundaries.

Local validation: compiled DataTypeArray.cpp and gtest_data_type_array.cpp, checked their formatting with clang-format, and ran git diff --check. The full gtests_dbms suite was not run locally.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Fix an out-of-bounds read when decoding Array columns with overflowing or decreasing offsets, or missing elements

Summary by CodeRabbit

• Bug Fixes
  • Prevented oversized array requests from exceeding supported allocation limits, with clearer errors when limits are exceeded.
  • Improved validation when reading array data, including checks for invalid sizes, offsets, and mismatches between array entries and their contents.
  • Added coverage for valid array reads and round trips, as well as malformed and oversized data.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ilovesoup for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

PODArray now checks element counts against allocation and reserve limits. Array deserialization checks size arithmetic, offset ordering, and nested-column sizes. New tests cover allocation boundaries, valid array streams, and malformed or incomplete data.

Changes: Array Allocation and Deserialization Validation

Layer / File(s)	Summary
PODArray allocation limits dbms/src/Common/PODArray.h, dbms/src/Common/tests/gtest_pod_array.cpp, dbms/src/DataTypes/tests/gtest_data_type_array.cpp	PODArray defines limits for unrounded allocations and rounded reserves, and checks element counts before calculating allocation sizes. Tests cover boundary requests and array decode allocation overflow.
Array offset and nested-column validation dbms/src/DataTypes/DataTypeArray.cpp	Array deserialization checks position-independent size accumulation, absolute offset ordering, and nested-column size against the final offset.
Array deserialization tests dbms/src/DataTypes/tests/gtest_data_type_array.cpp	Tests cover stream round trips, appended batches, empty and nested columns, malformed offsets, and incomplete element streams.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: jayson-huang, gengliqi, hongyunyan

Merge Risk: Minimal · up to 4a977

This change hardens array deserialization and buffer allocation against corrupt offsets and oversized allocations, so malformed data is rejected instead of causing out-of-bounds reads. No actionable merge-blocking risk remains after review.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title names the main change: array bulk deserialization now rejects corrupt offsets before the nested column is indexed. The PR also adds PODArray allocation checks, but the title covers the prima…
Description check		The description follows the repository template. It includes the problem summary with the linked issue, a commit-message block and a detailed change list, the checklist with the unit test box checked …
Linked Issues check		The only directly linked issue, #11206, is closed and completed. The issue text labels itself as historical context only. Under the assessment rules, closed issues supply background only, so no active…
Out of Scope Changes check		With no linked-issue requirements, scope is defined by the current PR description. The described changes all serve one goal: prevent corrupt Array offsets, inconsistent nested lengths, and overflowing…

Full details: Title check

Explanation

The title names the main change: array bulk deserialization now rejects corrupt offsets before the nested column is indexed. The PR also adds PODArray allocation checks, but the title covers the primary change and its file area (DataTypes). The title is specific and matches the changeset.

Full details: Description check

Explanation

The description follows the repository template. It includes the problem summary with the linked issue, a commit-message block and a detailed change list, the checklist with the unit test box checked and local validation noted, and a release note. The Side effects and Documentation boxes are left unchecked, and the release note is a specific bugfix entry. The description is thorough, and the only gap is that no side-effect box is marked even though the PR removes a Nested ALTER exception and adds allocation checks that could affect behavior.

Full details: Linked Issues check

Explanation

The only directly linked issue, #11206, is closed and completed. The issue text labels itself as historical context only. Under the assessment rules, closed issues supply background only, so no active directly linked targets remain and no linked-issue coding requirements apply. The PR objectives state the same conclusion.

Full details: Out of Scope Changes check

Explanation

With no linked-issue requirements, scope is defined by the current PR description. The described changes all serve one goal: prevent corrupt Array offsets, inconsistent nested lengths, and overflowing allocation sizes during bulk deserialization. The changes cover: (1) dbms/src/DataTypes/DataTypeArray.cpp, which adds offset-count and cumulative-offset overflow checks, monotonicity checks, and nested-length checks; (2) dbms/src/Common/PODArray.h, which adds TOO_LARGE_ARRAY_SIZE checks for allocation-size arithmetic; (3) dbms/src/Common/tests/gtest_pod_array.cpp and dbms/src/DataTypes/tests/gtest_data_type_array.cpp, which test those paths. The DMFileReader::readColumn change is described in the PR as a required consequence of removing the Nested ALTER exception for non-zero offsets with an empty nested column. Its link to that removal is stated, though it is not verified in this review. No unrelated change appears in the summary.

Finishing Touches

Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit checks each offset’s place,
And bounds each count before the race.
Through streams, the nested values flow,
Bad sums stop before bytes can go.
The burrow rests with columns sound.}

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f31fedff-439c-4faa-925d-401bb9b1030b

Commits

Reviewing files that changed from the base of the PR and between 9089405 and 4a97711.

Files selected for processing: 4

• dbms/src/Common/PODArray.h
• dbms/src/Common/tests/gtest_pod_array.cpp
• dbms/src/DataTypes/DataTypeArray.cpp
• dbms/src/DataTypes/tests/gtest_data_type_array.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.