Skip to content

ColumnEnum writes undeclared Enum8/Enum16 ordinals by default; server stores them and cannot read them back #602

Open

Description

@claude

Description

ColumnEnum<T> sends an Enum8/Enum16 ordinal to the server without checking it against the members the column type declares, on every path that is reachable by default. The server does not convert a Native block whose column type already is Enum8(...) — it stores the bytes as they are. The insert succeeds and the row then cannot be read back: SELECT e / SELECT toString(e) fail with Code: 691 UNKNOWN_ELEMENT_OF_ENUM, and the bad value stays in the table.

The label path is correct and asymmetric with the ordinal path:

Path File Validates?
Append(const std::string& name) clickhouse/columns/enum.cpp:36 yes — EnumType::GetEnumValue uses name_to_value_.at(), throws std::out_of_range for an unknown label
SetNameAt(n, name) clickhouse/columns/enum.cpp:64 yes, same mechanism
Append(const T& value, bool checkValue = false) clickhouse/columns/enum.cpp:30 no, by default — checkValue defaults to false
SetAt(n, value, bool checkValue = false) clickhouse/columns/enum.cpp:58 no, by default
ColumnEnum(TypeRef, const std::vector<T>&) clickhouse/columns/enum.cpp:15 no, and no way to ask for it — bulk data is stored verbatim
ColumnEnum(TypeRef, std::vector<T>&&) clickhouse/columns/enum.cpp:22 no, and no way to ask for it
Append(ColumnRef) clickhouse/columns/enum.cpp:81 no — splices col->data_ in without comparing the two types' members
Slice(begin, len) clickhouse/columns/enum.cpp:99 n/a — constructs via the unchecked vector ctor

SaveBody (clickhouse/columns/enum.cpp:88) then writes those ordinals straight to the wire:

template <typename T>
void ColumnEnum<T>::SaveBody(OutputStream* output) {
    WireFormat::WriteBytes(*output, data_.data(), data_.size() * sizeof(T));
}

EnumType::HasEnumValue (clickhouse/types/types.cpp:403) is available and is exactly the check needed, but nothing on the write path calls it unless the caller opts in per value.

Relation to #551

PR #551 added the checkValue parameter to Append/SetAt and states "Keep the default unchecked behavior unchanged." So the mechanism exists but is off by default, and it was never extended to the two vector constructors or to Append(ColumnRef). This issue is about that remaining default-unsafe gap, not about re-doing #551.

Why the default matters

A caller who builds a column from a std::vector<T> — the natural bulk-insert idiom, and the one Slice itself uses — has no validating option at all. A caller who uses Append(value) gets silent corruption unless they know to pass a second argument. The failure is not reported at insert time; it surfaces later as an unreadable row, and it survives OPTIMIZE TABLE ... FINAL on MergeTree.

Affected ordinals include 0, which is a very common accidental value (a default-constructed integer, an unset field) and which many ClickHouse enums do not declare.

ClickHouse server version

26.10.3.7 is the server available in this environment, but this finding is code analysis only; the reproduction below was compiled against the prebuilt libclickhouse-cpp-lib.a in this repo but could not be executed, because binary execution is blocked in the sandbox I investigated from. The equivalent behaviour was verified against a live server (26.9.1.1629) in the linked clickhouse-cs issue, including with a hand-built Native block posted over HTTP, which is what establishes that the server does not validate this path.

Reproduction

#include <clickhouse/client.h>
#include <iostream>

using namespace clickhouse;

int main() {
    Client client(ClientOptions().SetHost("localhost").SetPort(9000));

    client.Execute("DROP TABLE IF EXISTS t_enum_repro");
    client.Execute("CREATE TABLE t_enum_repro (e Enum8('a' = 1)) ENGINE = Memory");

    // 0 and 5 are NOT declared members of Enum8('a' = 1)
    auto col = std::make_shared<ColumnEnum8>(Type::CreateEnum8({{"a", 1}}));
    col->Append(1);
    col->Append(0);   // undeclared, default checkValue = false
    col->Append(5);   // undeclared

    Block block;
    block.AppendColumn("e", col);

    client.Insert("t_enum_repro", block);        // expected: throws. actual: succeeds.

    client.Select("SELECT CAST(e AS Int16) FROM t_enum_repro", [](const Block& b) {
        for (size_t i = 0; i < b.GetRowCount(); ++i)
            std::cout << "stored ordinal: " << b[0]->As<ColumnInt16>()->At(i) << "\n";
    });

    // expected: returns 'a' for every row. actual: Code 691 UNKNOWN_ELEMENT_OF_ENUM.
    client.Select("SELECT toString(e) FROM t_enum_repro", [](const Block&) {});

    // The vector constructor has no validation option at all:
    auto col2 = std::make_shared<ColumnEnum8>(Type::CreateEnum8({{"a", 1}}),
                                              std::vector<int8_t>{0, 5});
    Block b2;
    b2.AppendColumn("e", col2);
    client.Insert("t_enum_repro", b2);           // expected: throws. actual: succeeds.
}

Expected: both Insert calls throw before any byte of the block is written, naming the offending ordinal and the enum type — matching what the label path already does, and what the server does on every path where it does convert the value:

INSERT INTO t VALUES (0)
Code: 691. DB::Exception: Unexpected value 0 in enum (UNKNOWN_ELEMENT_OF_ENUM)

SELECT CAST(0 AS Enum8('a' = 1))
Code: 691. DB::Exception: Unexpected value 0 in enum (UNKNOWN_ELEMENT_OF_ENUM)

Actual: both inserts are accepted. SELECT CAST(e AS Int16) returns 1, 0, 5, 0, 5. SELECT toString(e) and SELECT e fail with Code: 691. The rows are permanently unreadable.

Suggested fix

Validate on the write path rather than only at the opt-in call site, so that no construction route can produce a column that serializes an undeclared ordinal:

  • clickhouse/columns/enum.cpp:88 — in SaveBody, check every element against the declared members before writing the first byte, and throw ValidationError naming the ordinal and type_->GetName(). This is the one chokepoint all paths pass through, so it closes the vector-constructor and Append(ColumnRef) holes too.
  • Keep the cost down on this hot path: EnumType::HasEnumValue is a std::map::find per value. A precomputed membership bitset (256 entries for Enum8) built once per type would be cheaper than a lookup per row.
  • Alternatively or additionally, validate in the two std::vector<T> constructors (clickhouse/columns/enum.cpp:15,22) and in Append(ColumnRef) (:81), and flip the checkValue default to true on Append/SetAt.
  • Behaviour that must not change: declared ordinals, including 0 when the enum declares it, and the int8_t/int16_t boundary values; label writes; Slice/CloneEmpty of an already-valid column (these can keep a plain copy, since the source was validated).

Link

Same contract gap reported for the .NET client's native TCP path: ClickHouse/clickhouse-cs#721

Activity

  1. polyglotAI-bot commented on Nov 17, 2026

    @polyglotAI-bot
    Collaborator

    Recommendation: Keep open — fix needed
    The defect shows on master, and no open PR covers it. One check in ColumnEnum<T>::SaveBody (clickhouse/columns/enum.cpp:89) closes all the unchecked paths; PR #609 fixes a different Enum defect.

    Analysis: Append(const T&, bool checkValue = false) (clickhouse/columns/enum.h:18, enum.cpp:30) and SetAt (enum.cpp:58) check the value only on request. The vector constructors (enum.cpp:16, enum.cpp:23) and Append(ColumnRef) (enum.cpp:76) never check, and Append(ColumnRef) also copies the ordinals of a column with another Enum type. SaveBody (enum.cpp:89) writes the bytes unchanged, and the server stores them. After that, SELECT toString(e) fails on the server; SELECT e over the native protocol returns the rows (the report says that it fails), but ColumnEnum8::NameAt throws std::out_of_range (map::at) for each undeclared ordinal. The opt-in check from commit a779cc3 (first in v2.6.0) works: Append(0, true) throws ValidationError.

    Reproduction: complete program and output
    #include <clickhouse/client.h>
    
    #include <cstdlib>
    #include <iostream>
    #include <string>
    #include <vector>
    
    using namespace clickhouse;
    
    static ClientOptions Options() {
        const char* host = std::getenv("CLICKHOUSE_HOST");
        const char* port = std::getenv("CLICKHOUSE_PORT");
        return ClientOptions()
            .SetHost(host ? host : "localhost")
            .SetPort(port ? static_cast<uint16_t>(std::stoi(port)) : 9000);
    }
    
    static TypeRef EnumA() { return Type::CreateEnum8({{"a", 1}}); }
    
    // Inserts one block with column e, then prints the result.
    static void InsertCase(const std::string& name, ColumnRef col) {
        try {
            Client client(Options());
            Block block;
            block.AppendColumn("e", col);
            client.Insert("t_enum_602", block);
            std::cout << name << ": insert OK, " << col->Size() << " rows" << std::endl;
        } catch (const std::exception& e) {
            std::cout << name << ": insert threw: " << e.what() << std::endl;
        }
    }
    
    int main() {
        {
            Client client(Options());
            client.Execute("DROP TABLE IF EXISTS t_enum_602");
            client.Execute("CREATE TABLE t_enum_602 (e Enum8('a' = 1)) ENGINE = Memory");
        }
    
        // 1. Append(value) with the default checkValue = false. 0 and 5 are not declared.
        {
            auto col = std::make_shared<ColumnEnum8>(EnumA());
            col->Append(int8_t(1));
            col->Append(int8_t(0));
            col->Append(int8_t(5));
            InsertCase("Append(value), default", col);
        }
    
        // 2. The vector constructor (no check option).
        InsertCase("ColumnEnum8(type, vector{0, 5})",
                   std::make_shared<ColumnEnum8>(EnumA(), std::vector<int8_t>{0, 5}));
    
        // 3. Append(ColumnRef) from a column of another Enum8 type.
        {
            auto other = std::make_shared<ColumnEnum8>(Type::CreateEnum8({{"b", 7}}));
            other->Append("b");
            auto col = std::make_shared<ColumnEnum8>(EnumA());
            col->Append(other);
            InsertCase("Append(ColumnRef) from Enum8('b' = 7)", col);
        }
    
        // 4. Opt-in check: Append(value, true) rejects the undeclared value.
        try {
            auto col = std::make_shared<ColumnEnum8>(EnumA());
            col->Append(int8_t(0), true);
            std::cout << "Append(0, true): accepted" << std::endl;
        } catch (const std::exception& e) {
            std::cout << "Append(0, true): threw: " << e.what() << std::endl;
        }
    
        // 5. The label path rejects an unknown label.
        try {
            auto col = std::make_shared<ColumnEnum8>(EnumA());
            col->Append(std::string("zzz"));
            std::cout << "Append(\"zzz\"): accepted" << std::endl;
        } catch (const std::exception& e) {
            std::cout << "Append(\"zzz\"): threw: " << e.what() << std::endl;
        }
    
        // Read back.
        try {
            Client client(Options());
            std::string ordinals;
            client.Select("SELECT CAST(e AS Int16) FROM t_enum_602", [&](const Block& b) {
                for (size_t i = 0; i < b.GetRowCount(); ++i)
                    ordinals += std::to_string(b[0]->As<ColumnInt16>()->At(i)) + " ";
            });
            std::cout << "stored ordinals: " << ordinals << std::endl;
        } catch (const std::exception& e) {
            std::cout << "SELECT CAST(e AS Int16) threw: " << e.what() << std::endl;
        }
        try {
            Client client(Options());
            size_t rows = 0;
            client.Select("SELECT toString(e) FROM t_enum_602", [&](const Block& b) { rows += b.GetRowCount(); });
            std::cout << "SELECT toString(e): " << rows << " rows" << std::endl;
        } catch (const std::exception& e) {
            std::cout << "SELECT toString(e) threw: " << e.what() << std::endl;
        }
        // SELECT e over the native protocol, then the label of each row in the client.
        try {
            Client client(Options());
            client.Select("SELECT e FROM t_enum_602", [&](const Block& b) {
                for (size_t i = 0; i < b.GetRowCount(); ++i) {
                    auto col = b[0]->As<ColumnEnum8>();
                    std::string label;
                    try {
                        label = "NameAt = " + std::string(col->NameAt(i));
                    } catch (const std::exception& e) {
                        label = std::string("NameAt threw: ") + e.what();
                    }
                    std::cout << "SELECT e: ordinal " << int(col->At(i)) << ", " << label << std::endl;
                }
            });
        } catch (const std::exception& e) {
            std::cout << "SELECT e threw: " << e.what() << std::endl;
        }
    
        Client(Options()).Execute("DROP TABLE IF EXISTS t_enum_602");
        return 0;
    }

    Output on master (58453c2), ClickHouse 26.10.4.207:

    Append(value), default: insert OK, 3 rows
    ColumnEnum8(type, vector{0, 5}): insert OK, 2 rows
    Append(ColumnRef) from Enum8('b' = 7): insert OK, 1 rows
    Append(0, true): threw: Enum type doesn't have value 0
    Append("zzz"): threw: map::at
    stored ordinals: 1 0 5 7 0 5 
    SELECT toString(e) threw: DB::Exception: Unexpected value 0 in enum: while executing 'FUNCTION toString(__table1.e :: 1) -> toString(__table1.e) String : 0'
    SELECT e: ordinal 1, NameAt = a
    SELECT e: ordinal 0, NameAt threw: map::at
    SELECT e: ordinal 5, NameAt threw: map::at
    SELECT e: ordinal 0, NameAt threw: map::at
    SELECT e: ordinal 5, NameAt threw: map::at
    SELECT e: ordinal 7, NameAt threw: map::at
    exit=0
    

    Automated triage by Polyglot AI. A maintainer makes the final decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions