Skip to content

ColumnArray::AppendAsColumn silently accepts a wrong-typed element column, writes zero data bytes and desynchronizes the native block stream #543

Open

Description

@claude

Description

ColumnArray::AppendAsColumn() (clickhouse/columns/array.cpp:49) relies on data_->Append() throwing when the supplied column type does not match the array's element type:

void ColumnArray::AppendAsColumn(ColumnRef array) {
    // appending data may throw (i.e. due to ype check failure), so do it first to avoid partly modified state.
    data_->Append(array);
    AddOffset(array->Size());
}

But most Column::Append(ColumnRef) implementations silently no-op on a type mismatch instead of throwing:

  • ColumnVector<T>::Append — clickhouse/columns/numeric.cpp:72-76: if (auto col = column->As<ColumnVector<T>>()) { ... }, no else.
  • ColumnString::Append — clickhouse/columns/string.cpp:248-260: same shape.
  • ColumnFixedString::Append — clickhouse/columns/string.cpp:73-79: same shape, and additionally silent when string_size_ differs.

So passing a wrong-typed column to AppendAsColumn appends nothing to data_ while AddOffset(array->Size()) still advances the offsets by array->Size(). The ColumnArray is left internally inconsistent — exactly the state its own constructor rejects with ValidationError("Mismatch between data and offsets: ...") — and SaveBody() then writes offsets promising N elements followed by zero element bytes.

On the wire this desynchronizes the native-protocol block: the server reads the following column's bytes as this array's element data. The user gets no client-side error, only a confusing (and misleading) server-side exception, and the connection is left unusable for the next operation.

This is the C++ analogue of ClickHouse/clickhouse-java#3041, where SerializerUtils.serializeArrayData silently writes zero bytes for a non-null, non-array/non-List value.

Note the same failure class was reported for ColumnNullable in PR #376 (closed, unmerged): nulls_ grows even when the nested Append() silently fails. ColumnArray has the identical problem with offsets_.

ClickHouse server version

26.7.2.59 (native protocol, port 9000), verified against a live server.
Repo at commit 737145d.

Reproduction

Added to ut/column_array_ut.cpp (needs #include <clickhouse/client.h> for the second test):

TEST(ArrayDesync, WrongTypedAppendAsColumn) {
    // Array(String), but we append a UInt64 column as one row's elements
    auto arr = std::make_shared<ColumnArray>(std::make_shared<ColumnString>());
    auto wrong = std::make_shared<ColumnUInt64>();
    wrong->Append(1); wrong->Append(2); wrong->Append(3);

    EXPECT_NO_THROW(arr->AppendAsColumn(wrong));   // currently passes -- the defect

    std::cerr << "arr->Size()=" << arr->Size()
              << " claimed elems row0=" << arr->GetSize(0)
              << " actual data size=" << arr->GetAsColumn(0)->Size() << std::endl;

    Buffer buf;
    BufferOutput out(&buf);
    arr->SaveBody(&out);
    out.Flush();
    std::cerr << "SaveBody bytes = " << buf.size() << std::endl;
}

TEST(ArrayDesync, EndToEndInsert) {
    clickhouse::Client client(clickhouse::ClientOptions().SetHost("localhost").SetPort(9000));
    client.Execute("DROP TABLE IF EXISTS test_arr_desync");
    client.Execute("CREATE TABLE test_arr_desync (id UInt32, val Array(String), tail String) ENGINE = Memory");

    clickhouse::Block b;
    auto id = std::make_shared<ColumnUInt32>(); id->Append(1);

    auto val = std::make_shared<ColumnArray>(std::make_shared<ColumnString>());
    auto bad = std::make_shared<ColumnUInt64>(); bad->Append(7); bad->Append(8);
    val->AppendAsColumn(bad);                     // wrong element type, silently dropped

    auto tail = std::make_shared<ColumnString>(); tail->Append("TAILVALUE");
    b.AppendColumn("id", id);
    b.AppendColumn("val", val);
    b.AppendColumn("tail", tail);

    try { client.Insert("test_arr_desync", b); std::cerr << "INSERT SUCCEEDED" << std::endl; }
    catch (const std::exception& e) { std::cerr << "INSERT threw: " << e.what() << std::endl; }

    try {
        client.Select("SELECT id, val, tail FROM test_arr_desync", [](const clickhouse::Block&) {});
    } catch (const std::exception& e) { std::cerr << "SELECT threw: " << e.what() << std::endl; }
}

Actual output

[ RUN      ] ArrayDesync.WrongTypedAppendAsColumn
arr->Size()=1 claimed elems row0=3 actual data size=0
SaveBody bytes = 8
[       OK ] ArrayDesync.WrongTypedAppendAsColumn

[ RUN      ] ArrayDesync.EndToEndInsert
INSERT threw: DB::Exception: Unknown data type family: TAILVALUE
SELECT threw: cannot execute query while executing another operation
[       OK ] ArrayDesync.EndToEndInsert

Reading that: SaveBody emitted only the 8-byte offset (3) and zero element bytes. End-to-end, the server consumed the tail column's string payload as the type name of the next column — hence Unknown data type family: TAILVALUE. The connection was then left mid-operation, so the follow-up SELECT also failed.

Expected

AppendAsColumn (or the underlying Append) should reject a column whose type does not match the array's element type with a clear client-side ValidationError / std::runtime_error naming both types, leaving the ColumnArray unmodified. Appending a correctly-typed column must keep working, and AppendAsColumn of an empty correctly-typed column must still add a zero-length row.

Suggested fix

Two possible layers, not mutually exclusive:

  1. Narrow — in ColumnArray::AppendAsColumn (clickhouse/columns/array.cpp:49), validate before mutating:

    void ColumnArray::AppendAsColumn(ColumnRef array) {
        if (!data_->Type()->IsEqual(array->Type()))
            throw ValidationError("Cannot append column of type " + array->Type()->GetName()
                                  + " to Array of " + data_->Type()->GetName());
        data_->Append(array);
        AddOffset(array->Size());
    }

    This covers the array path only, but it is where the offsets/data desync is introduced.

  2. General — make the Append(ColumnRef) implementations throw on mismatch instead of silently returning (numeric.cpp:72, string.cpp:73, string.cpp:248, and any sibling with the same if (auto col = column->As<...>())-with-no-else shape). That would also fix the ColumnNullable variant from PR [Bug fix] Append() in Nullable columns causes nulls_ array to be incorrectly appended to if nested column Append() fails #376 and make the existing comment in AppendAsColumn true. It is a behavior change for callers currently relying on the silent no-op, so it may warrant its own discussion.

Regression tests: a wrong-typed AppendAsColumn throws and leaves Size()/GetOffset() unchanged, plus contrast cases that a correctly-typed column and an empty correctly-typed column still behave as today.

Link

Relayed from ClickHouse/clickhouse-java#3041

Activity

  1. polyglotAI-bot commented on Nov 18, 2026

    @polyglotAI-bot
    Collaborator

    Recommendation: Close — already fixed
    Commit 0dac477 ("Validate ColumnArray element types before appending", merged to master on 2026-09-16 in 09613d4) fixes this. Not released yet: the latest release v2.6.2 still shows the defect, so the fix reaches users with the next release.

    Analysis: On master, ColumnArray::AppendAsColumn checks the element type before it changes the column (clickhouse/columns/array.cpp:96-101, helper CanAppendType at array.cpp:31-54) and throws ValidationError. The rows, offsets and data stay unchanged, so SaveBody and Client::Insert cannot send a desynchronized block. The check keeps the conversions that worked before: equal types, Tuple with other field names, LowCardinality(T) from T, and Bool from UInt8; the same commit adds unit tests for them and for the empty-column row (ut/column_array_ut.cpp). In v2.6.2, AppendAsColumn calls data_->Append() with no check (array.cpp:49-53 at v2.6.2), and the run below shows the reported desync. The general layer of the report is still on master: ColumnVector<T>::Append (numeric.cpp:71-75), ColumnFixedString::Append (string.cpp:73-79) and ColumnString::Append (string.cpp:252-265) ignore a column of another type with no error. To make them throw is a separate behavior change; if it is wanted, it needs its own issue.

    Reproduction: complete program and output
    #include <clickhouse/base/output.h>
    #include <clickhouse/client.h>
    
    #include <cstdlib>
    #include <iostream>
    #include <memory>
    #include <string>
    
    using namespace clickhouse;
    
    static ClientOptions Options() {
        const char* host = std::getenv("CLICKHOUSE_HOST");
        const char* port = std::getenv("CLICKHOUSE_PORT");
        return ClientOptions()
            .SetHost(host ? host : "localhost")
            .SetPort(port ? static_cast<uint16_t>(std::stoi(port)) : 9000);
    }
    
    static void PrintState(const char* name, ColumnArray& arr) {
        Buffer buf;
        BufferOutput out(&buf);
        arr.SaveBody(&out);
        out.Flush();
        std::cout << name << ": rows=" << arr.Size()
                  << " elements_by_offsets=" << (arr.Size() ? arr.GetOffset(arr.Size() - 1) + arr.GetSize(arr.Size() - 1) : 0)
                  << " elements_in_data=" << arr.GetData()->Size()
                  << " SaveBody_bytes=" << buf.size() << "\n";
    }
    
    int main() {
        // Case 1: Array(String) gets a UInt64 column as one row.
        {
            ColumnArray arr(std::make_shared<ColumnString>());
            auto wrong = std::make_shared<ColumnUInt64>();
            wrong->Append(1); wrong->Append(2); wrong->Append(3);
            try {
                arr.AppendAsColumn(wrong);
                std::cout << "case1 wrong type: no exception\n";
            } catch (const std::exception& e) {
                std::cout << "case1 wrong type: exception: " << e.what() << "\n";
            }
            PrintState("case1 state", arr);
        }
    
        // Case 2: a correct-typed column and an empty correct-typed column.
        {
            ColumnArray arr(std::make_shared<ColumnString>());
            auto ok = std::make_shared<ColumnString>();
            ok->Append("a"); ok->Append("b");
            arr.AppendAsColumn(ok);
            arr.AppendAsColumn(std::make_shared<ColumnString>());
            PrintState("case2 correct + empty", arr);
        }
    
        // Case 3: end-to-end INSERT of a block with the wrong-typed row.
        Client client(Options());
        client.Execute("DROP TABLE IF EXISTS test_issue543");
        client.Execute("CREATE TABLE test_issue543 (id UInt32, val Array(String), tail String) ENGINE = Memory");
        {
            auto id = std::make_shared<ColumnUInt32>();
            id->Append(1);
            auto val = std::make_shared<ColumnArray>(std::make_shared<ColumnString>());
            auto bad = std::make_shared<ColumnUInt64>();
            bad->Append(7); bad->Append(8);
            auto tail = std::make_shared<ColumnString>();
            tail->Append("TAILVALUE");
            bool built = true;
            try {
                val->AppendAsColumn(bad);
            } catch (const std::exception& e) {
                built = false;
                std::cout << "case3 build block: exception: " << e.what() << " (INSERT not sent)\n";
            }
            if (built) {
                Block b;
                b.AppendColumn("id", id);
                b.AppendColumn("val", val);
                b.AppendColumn("tail", tail);
                try {
                    client.Insert("test_issue543", b);
                    std::cout << "case3 INSERT: no exception\n";
                } catch (const std::exception& e) {
                    std::cout << "case3 INSERT: exception: " << e.what() << "\n";
                }
            }
            try {
                size_t rows = 0;
                client.Select("SELECT count() FROM test_issue543", [&](const Block& blk) {
                    if (blk.GetRowCount()) rows = blk[0]->As<ColumnUInt64>()->At(0);
                });
                std::cout << "case3 next SELECT on same client: count=" << rows << "\n";
            } catch (const std::exception& e) {
                std::cout << "case3 next SELECT on same client: exception: " << e.what() << "\n";
            }
        }
    
        Client(Options()).Execute("DROP TABLE IF EXISTS test_issue543");
        return 0;
    }

    Output on master (58453c2), ClickHouse 26.10.4.207:

    case1 wrong type: exception: can't append column of type UInt64 to column type String
    case1 state: rows=0 elements_by_offsets=0 elements_in_data=0 SaveBody_bytes=0
    case2 correct + empty: rows=2 elements_by_offsets=2 elements_in_data=2 SaveBody_bytes=20
    case3 build block: exception: can't append column of type UInt64 to column type String (INSERT not sent)
    case3 next SELECT on same client: count=0
    

    Output on v2.6.2, ClickHouse 26.10.4.207:

    case1 wrong type: no exception
    case1 state: rows=1 elements_by_offsets=3 elements_in_data=0 SaveBody_bytes=8
    case2 correct + empty: rows=2 elements_by_offsets=2 elements_in_data=2 SaveBody_bytes=20
    case3 INSERT: exception: DB::Exception: Unknown data type family: TAILVALUE
    case3 next SELECT on same client: exception: cannot execute query while executing another operation
    

    Automated triage by Polyglot AI. Classification added on 2026-11-19. A maintainer makes the final decision.

  2. added
    data-typesArea: type names, value encoding and decoding, row mapping, data formats
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdata-typesArea: type names, value encoding and decoding, row mapping, data formats

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions