Skip to content

Array(Nested(...)) columns silently decode as Array(Nothing) and desynchronize the native block stream #571

Description

@claude

Description

The type parser has no entry for Nested, so a column whose server-reported type is Array(Nested(key String, value String)) is parsed as Array(<unknown terminal>) with Type::Void, and CreateTerminalColumn() maps Type::Void to ColumnNothing. The result is that CreateColumnByType() returns a perfectly valid-looking ColumnArray(ColumnNothing) — no error, no nullptr — and Client::Select() happily accepts it when reading the block header.

This matters because Nested inside Array(...) is not flattened by the server. Unlike a top-level Nested column (which the server splits into col.key Array(String), col.value Array(String) — the case discussed in #40), Array(Nested(...)) is reported over the native protocol with the literal type name:

$ clickhouse-client -q "SELECT name, type FROM system.columns WHERE table='some_data'"
id      UInt64
data    Array(Nested(key String, value String))

On the wire Array(Nested(k String, v String)) is serialized as Array(Array(Tuple(String, String))). But the client builds Array(Nothing), and ColumnNothing::LoadBody() (clickhouse/columns/nothing.h:61) just does input->Skip(rows) — 1 byte per element. So after reading the outer offsets the client skips N bytes where the server actually wrote 8*N inner-offset bytes plus all the string data. The input stream is desynchronized from that point on: the remaining block (and every block after it) is decoded as garbage, typically surfacing much later as an unrelated parse failure or a corrupted/empty result rather than a clear "unsupported type" error.

This is the C++ analogue of ClickHouse/clickhouse-java#3178, where the JDBC driver also failed on Array(Nested(...)) because its conversion layer did not account for Nested producing an extra list level.

Related but distinct from #40, which asks for a convenience API over flattened top-level Nested columns (those already work, since the server hands them over as plain Array(T)). Here the client receives a literal Nested(...) type name and silently produces wrong data.

ClickHouse server version

26.9.8.3 — used to confirm the type name the server reports for Array(Nested(...)) (shown above).

Code analysis only; the C++ repro below was written but not executed — binary execution was unavailable in the environment I investigated from. The static trace through TypeParser::Parse → CreateColumnFromAst → CreateTerminalColumn is given in full under "Suggested fix" below, and the first assertion (Array(Nothing)) is a pure parser/factory fact requiring no server.

Reproduction

Pure client-side, no server needed — this already demonstrates the root cause:

#include <clickhouse/columns/factory.h>
#include <gtest/gtest.h>

using namespace clickhouse;

TEST(CreateColumnByType, ArrayOfNested) {
    auto col = CreateColumnByType("Array(Nested(key String, value String))");
    ASSERT_NE(nullptr, col);
    // Expected: Array(Array(Tuple(String, String)))
    // Actual:   Array(Nothing)
    EXPECT_EQ("Array(Array(Tuple(String, String)))", col->Type()->GetName());
}

End-to-end, against a server:

CREATE TABLE some_data (
    `id`   UInt64,
    `data` Array(Nested(`key` String, `value` String))
) ENGINE = Memory;

INSERT INTO some_data VALUES
(1, [[('key1','test'), ('key2','another-test')], [('key1','more-data')]]);
#include <clickhouse/client.h>
#include <iostream>

using namespace clickhouse;

int main() {
    Client client(ClientOptions().SetHost("localhost").SetPort(9000));
    client.Select("SELECT id, data FROM some_data", [](const Block& block) {
        for (size_t c = 0; c < block.GetColumnCount(); ++c) {
            std::cout << block.GetColumnName(c) << " -> "
                      << block[c]->Type()->GetName() << "\n";
        }
    });
}

Expected: the data column comes back as Array(Array(Tuple(String, String))) with the two inner arrays intact (or, failing that, a clear UnimplementedError naming the unsupported type).

Actual: data is reported as Array(Nothing) carrying no values, and because ColumnNothing::LoadBody under-consumes the stream by the full size of the inner offsets and strings, decoding of the rest of the response is corrupted.

Suggested fix

Trace of the current behaviour:

  • clickhouse/types/type_parser.cpp:116 — GetTypeMeta() has no Nested branch, so the Nested(...) node falls through to TypeAst::Terminal.
  • clickhouse/types/type_parser.cpp:107 — GetTypeCode("Nested") misses kTypeCode and returns Type::Void.
  • clickhouse/types/type_parser.cpp:152 — ValidateAST() does reject unknown Terminal + Void nodes, but it is only ever called on the root AST node (type_parser.cpp:238). The root here is Array, so the bad child is never validated.
  • clickhouse/columns/factory.cpp:49 — CreateTerminalColumn() maps Type::Void to ColumnNothing, turning the unknown type into a silently-wrong column instead of a nullptr.

Two things worth doing, independently useful:

  1. Support Nested. Nested(a T1, b T2) is exactly Array(Tuple(a T1, b T2)). Adding a Nested meta that desugars to that in CreateColumnFromAst would make Array(Nested(...)) decode correctly, and would also let named-tuple element names (already supported via TypeAst::element_name) carry through.

  2. Fail loudly on unknown nested types. Apply ValidateAST recursively, or have CreateTerminalColumn return nullptr for Type::Void when ast.name is not literally Nothing/void. Right now any unrecognized type nested inside a container degrades to ColumnNothing and desynchronizes the stream rather than raising UnimplementedError. (Same silent-desync failure mode as ColumnArray::AppendAsColumn silently accepts a wrong-typed element column, writes zero data bytes and desynchronizes the native block stream #543.)

Link

Original client issue: ClickHouse/clickhouse-java#3178

Activity

  1. Hakaou commented on Oct 17, 2026

    @Hakaou

    I can reproduce this on master (fc5d98f) against ClickHouse 26.11: CreateColumnByType("Array(Nested(key String, value String))") returns Array(Void). A Select on such a column fails: without compression the callback gets an Array(Void) column, whose elements throw Can't cast from Void when read as arrays, and the select then fails with an empty server exception, with LZ4 it fails with some data was not read. With flatten_nested = 0 a top-level Nested column fails earlier with unsupported column type: Nested(...).

    I'm taking it: the parser will know Nested as its own node and the column factory will build an Array of the named Tuple from it, which is how the server defines it (src/DataTypes/DataTypeNested.cpp builds a DataTypeArray of a named DataTypeTuple under a custom name). Opened #575 for it. The second point of the suggested fix, unknown type names inside a container, is tracked separately in #589, with a draft in #590.

  2. added 9 commits that reference this issue on Oct 18, 2026
    c3bdd9a
    713e8b2
    1ef19aa
    feb0f8e
    10aa714
    b414b18
    71399b4
    272ddd0
    9d61241
  3. polyglotAI-bot commented on Nov 18, 2026

    @polyglotAI-bot
    Collaborator

    Recommendation: Keep open — fix needed: review PR #575
    PR #575 fixes this report: with its diff on current master, the program below reads all rows correctly, with and without compression. The second point of the report (unknown type names inside a container) is tracked in #589, with PR #590.

    Analysis: GetTypeMeta has no Nested entry (clickhouse/types/type_parser.cpp:123-157), so Nested(...) becomes a terminal, and GetTypeCode gives it Type::Void (type_parser.cpp:114-121). ValidateAST rejects such a terminal only at the root of the type (type_parser.cpp:251), so inside Array(...) it passes, and CreateTerminalColumn builds a ColumnNothing for it (clickhouse/columns/factory.cpp:61-62). ColumnNothing::LoadBody skips one byte per row (clickhouse/columns/nothing.h:61-62), but the server sends the inner offsets and the strings, so the client reads the rest of the stream at the wrong position. The callback gets an Array(Void) column with no data, then Select throws an unknown packet type error (unimplemented 107) without compression, or some data was not read with LZ4. The server reports the column type as Array(Nested(key String, value String)): it does not flatten a Nested inside an Array.

    Reproduction: complete program and output
    #include <clickhouse/client.h>
    #include <clickhouse/columns/factory.h>
    
    #include <cstdlib>
    #include <iostream>
    #include <string>
    
    using namespace clickhouse;
    
    static ClientOptions Options(CompressionMethod compression) {
        const char* host = std::getenv("CLICKHOUSE_HOST");
        const char* port = std::getenv("CLICKHOUSE_PORT");
        return ClientOptions()
            .SetHost(host ? host : "localhost")
            .SetPort(port ? std::stoi(port) : 9000)
            .SetCompressionMethod(compression);
    }
    
    static std::string Render(const ColumnRef& data, size_t row) {
        // data is Array(Nested(key String, value String)) = Array(Array(Tuple(String, String)))
        auto outer = data->As<ColumnArray>();
        if (!outer)
            return "<not an array>";
        auto nested = outer->GetAsColumn(row)->As<ColumnArray>();
        if (!nested)
            return "<element column is " + outer->GetAsColumn(row)->Type()->GetName() + ">";
        std::string out = "[";
        for (size_t i = 0; i < nested->Size(); ++i) {
            auto tuple = nested->GetAsColumn(i)->As<ColumnTuple>();
            out += i ? ", [" : "[";
            for (size_t j = 0; j < tuple->Size(); ++j) {
                out += j ? ", (" : "(";
                out += std::string((*tuple)[0]->As<ColumnString>()->At(j)) + ", " +
                       std::string((*tuple)[1]->As<ColumnString>()->At(j)) + ")";
            }
            out += "]";
        }
        return out + "]";
    }
    
    static void SelectCase(const std::string& name, CompressionMethod compression) {
        try {
            Client client(Options(compression));
            size_t rows = 0;
            client.Select("SELECT id, data FROM issue571 ORDER BY id", [&](const Block& block) {
                if (block.GetRowCount() == 0)
                    return;
                rows += block.GetRowCount();
                std::cout << name << ": block of " << block.GetRowCount() << " rows:";
                for (size_t c = 0; c < block.GetColumnCount(); ++c)
                    std::cout << " " << block.GetColumnName(c) << "=" << block[c]->Type()->GetName();
                std::cout << "\n";
                for (size_t r = 0; r < block.GetRowCount(); ++r)
                    std::cout << name << ":   id=" << block[0]->As<ColumnUInt64>()->At(r)
                              << " data=" << Render(block[1], r) << "\n";
            });
            std::cout << name << ": Select returned, " << rows << " rows (expected 2)\n";
        } catch (const std::exception& e) {
            std::cout << name << ": exception: " << e.what() << "\n";
        }
    }
    
    int main() {
        auto col = CreateColumnByType("Array(Nested(key String, value String))");
        std::cout << "CreateColumnByType: "
                  << (col ? col->Type()->GetName() : std::string("nullptr")) << "\n";
    
        {
            Client client(Options(CompressionMethod::None));
            client.Execute("DROP TABLE IF EXISTS issue571");
            client.Execute("CREATE TABLE issue571 (id UInt64, data Array(Nested(key String, value String))) ENGINE = Memory");
            client.Execute("INSERT INTO issue571 VALUES "
                           "(1, [[('key1', 'test'), ('key2', 'another-test')], [('key1', 'more-data')]]), "
                           "(2, [[('key3', 'x')]])");
            client.Select("SELECT toTypeName(data) FROM issue571 LIMIT 1", [](const Block& block) {
                if (block.GetRowCount() > 0)
                    std::cout << "server type: " << block[0]->As<ColumnString>()->At(0) << "\n";
            });
        }
    
        SelectCase("Select, no compression", CompressionMethod::None);
        SelectCase("Select, LZ4", CompressionMethod::LZ4);
    
        Client client(Options(CompressionMethod::None));
        client.Execute("DROP TABLE issue571");
        return 0;
    }

    Output on master (58453c2), ClickHouse 26.10.4.207:

    CreateColumnByType: Array(Void)
    server type: Array(Nested(key String, value String))
    Select, no compression: block of 2 rows: id=UInt64 data=Array(Void)
    Select, no compression:   id=1 data=<element column is Void>
    Select, no compression:   id=2 data=<element column is Void>
    Select, no compression: exception: unimplemented 107
    Select, LZ4: exception: some data was not read
    exit=0
    

    Output on master (58453c2) with the diff of PR #575 (2a438751) applied, ClickHouse 26.10.4.207:

    CreateColumnByType: Array(Array(Tuple(key String, value String)))
    server type: Array(Nested(key String, value String))
    Select, no compression: block of 2 rows: id=UInt64 data=Array(Array(Tuple(key String, value String)))
    Select, no compression:   id=1 data=[[(key1, test), (key2, another-test)], [(key1, more-data)]]
    Select, no compression:   id=2 data=[[(key3, x)]]
    Select, no compression: Select returned, 2 rows (expected 2)
    Select, LZ4: block of 2 rows: id=UInt64 data=Array(Array(Tuple(key String, value String)))
    Select, LZ4:   id=1 data=[[(key1, test), (key2, another-test)], [(key1, more-data)]]
    Select, LZ4:   id=2 data=[[(key3, x)]]
    Select, LZ4: Select returned, 2 rows (expected 2)
    exit=0
    

    Automated triage by Polyglot AI. Classification added on 2026-11-19. A maintainer makes the final decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions