Skip to content

Type parser: escapes in a quoted Enum label / backtick identifier are not decoded like the server #606

Open

Description

@claude

Describe the bug

TypeParser::NextToken (clickhouse/types/type_parser.cpp:276-298) decodes escapes inside a single-quoted Enum label with kUnescapeMap (type_parser.cpp:87-99), which only covers \0 \b \f \n \r \t \\ plus a special case for \'. Every other escape falls into the else arm on line 293 and keeps its backslash.

The server (parseComplexEscapeSequence, src/IO/ReadHelpers.cpp) additionally:

  • decodes \N to no characters at all;
  • drops the backslash when the escaped character is one of \ ' " ` / = or an ASCII control character (<= 0x1F).

So the client and the server disagree on these labels:

Escape in a quoted label Server clickhouse-cpp
\N removed (empty) \N (backslash kept)
\= = \=
\/ / \/
\ + control char 0x00-0x1F (for example a literal TAB) the control character \ + the control character

The same applies to the backtick / double-quote branch (type_parser.cpp:299-327), which decodes only \<quote> and <quote><quote> and leaves every other escape untouched. The server applies the full escape table to back-quoted identifiers too, so Tuple(`a\Nb` Int8, c Int8) is Tuple(ab Int8, c Int8) on the server but gives the element name a\Nb in the client.

Impact is low, the same as in the C# client issue linked below: the server never emits these escapes in a type name that it sends, so the read path and server-driven insert schemas are not affected. It matters only for type strings that the application writes, for example CreateColumnByType(...) or TypeParser with a DDL type copied from a CREATE TABLE. There the client builds an Enum whose label is a\Nb while the server's label for the same DDL is ab, so ColumnEnum::Append("ab") throws, and labels can collide or fail to collide differently from the server (Enum8('a\Nb' = 1, 'ab' = 2) is a duplicate-name error on the server but two distinct labels in the client).

ClickHouse server version

26.10.3.7. Server behaviour verified over HTTP:

SELECT toTypeName(CAST(1 AS Enum8('a\Nb' = 1)))              -- Enum8('ab' = 1)
SELECT toTypeName(CAST(1 AS Enum8('a\=b' = 1)))              -- Enum8('a=b' = 1)
SELECT hex(toString(CAST(1 AS Enum8('a\<TAB>b' = 1))))       -- 610962   (a<TAB>b)
SELECT toTypeName(CAST((1,2) AS Tuple(`a\Nb` Int8, c Int8))) -- Tuple(ab Int8, c Int8)

The client side comes from analysis of the tokenizer. The test below compiles against the library, but my sandbox could not execute the built binary, so the "clickhouse-cpp" column is the behaviour that the code produces by inspection, not a captured run.

Reproduction

Add to ut/type_parser_ut.cpp, in the style of the existing TypeParserCase.ParseEnum_AllEscapedBytes:

TEST(TypeParserCase, ParseEnum_ServerEscapes) {
    const char kTab = '\t';
    // {fragment as seen by the parser, expected decoded label}
    const std::vector<std::pair<std::string, std::string>> cases = {
        {"a\\Nb", "ab"},                                     // \N decodes to nothing
        {"a\\=b", "a=b"},                                    // \= decodes to '='
        {"a\\/b", "a/b"},                                    // \/ decodes to '/'
        {std::string("a\\") + kTab + "b",                    // backslash + literal TAB
         std::string("a") + kTab + "b"},
    };

    for (const auto& [label, expected] : cases) {
        TypeAst ast;
        ASSERT_TRUE(TypeParser("Enum8('" + label + "' = 1)").Parse(&ast));
        ASSERT_EQ(ast.meta, TypeAst::Enum);
        EXPECT_EQ(ast.elements[0].value_string, expected) << "label: " << label;
    }

    // Back-quoted element names use the same server escape table.
    TypeAst tuple;
    ASSERT_TRUE(TypeParser("Tuple(`a\\Nb` Int8, c Int8)").Parse(&tuple));
    EXPECT_EQ(tuple.elements[0].name, "ab");

    // An unknown escape keeps its backslash, like the server.
    TypeAst unknown;
    ASSERT_TRUE(TypeParser("Enum8('a\\zb' = 1)").Parse(&unknown));
    EXPECT_EQ(unknown.elements[0].value_string, "a\\zb");
}

Expected: every EXPECT_EQ passes (labels ab, a=b, a/b, a<TAB>b, tuple element name ab).

Actual: the first five fail and give a\Nb, a\=b, a\/b, a\<TAB>b and tuple element name a\Nb. Only the last case (\z keeps its backslash) passes.

Public-API form of the same problem:

auto col = CreateColumnByType("Enum8('a\\Nb' = 1)");   // the server label for this DDL is "ab"
col->As<ColumnEnum8>()->Append(std::string("ab"));     // throws: unknown enum name

Suggested fix

In clickhouse/types/type_parser.cpp:

  • handle \N in the single-quote branch: consume both characters and append nothing;
  • drop the backslash also for ", `, / and = (kUnescapeMap has only 0 b f n r t \ today, and ' is special-cased at line 285);
  • drop the backslash when the escaped byte is an ASCII control character (<= 0x1F);
  • apply the same table in the backtick / double-quote branch (lines 299-327), which today decodes only the quote character itself.

Keep the current behaviour for everything else: an unknown escape such as \z keeps its backslash, and \ + DEL (0x7F) keeps its backslash.

Link

Relayed from the equivalent C# client issue: ClickHouse/clickhouse-cs#730

Activity

  1. eminfikret commented on Nov 16, 2026

    @eminfikret

    Hello, is this issue ready for contribution? I'd like to pick it up if it's available. Thanks.

  2. polyglotAI-bot commented on Nov 17, 2026

    @polyglotAI-bot
    Collaborator

    Recommendation: Keep open — fix needed
    No open PR fixes this: PR #608 (for #605) changes the same branch, but only adds the '' form. The fix must also decode \xHH, which the server decodes and the suggested fix in the issue does not list.

    Analysis: The single-quote branch of TypeParser::NextToken (clickhouse/types/type_parser.cpp:276-297) decodes only \' and the escapes in kUnescapeMap (type_parser.cpp:87-99: \0 \b \f \n \r \t \\); every other escape keeps its backslash (type_parser.cpp:293-295). The back-quote and double-quote branch (type_parser.cpp:299-327) decodes only \<quote> and the doubled quote, so a Tuple element name keeps a\Nb. The server decodes \N to nothing, drops the backslash before =, /, ", ` and a control character, and also decodes \x41 to A; an unknown escape such as \z keeps its backslash on both sides. With the client label a\Nb, Append("ab") throws std::out_of_range (map::at), and Enum8('a\Nb' = 1, 'ab' = 2), which the server rejects as duplicate names, is accepted. Type names that the server sends do not contain these escapes, so only type names that the application writes are affected.

    Reproduction: complete program and output
    // clickhouse-cpp#606: backslash escapes in a quoted Enum label and in a
    // back-quoted Tuple element name. Prints, for each type name, the type that
    // the client builds and the type that the server resolves for the same text.
    #include <clickhouse/client.h>
    #include <clickhouse/columns/factory.h>
    
    #include <cstdlib>
    #include <iostream>
    #include <string>
    
    using namespace clickhouse;
    
    static ClientOptions Options() {
        const char* host = std::getenv("CLICKHOUSE_HOST");
        const char* port = std::getenv("CLICKHOUSE_PORT");
        return ClientOptions()
            .SetHost(host ? host : "localhost")
            .SetPort(port ? std::atoi(port) : 9000);
    }
    
    // SQL string literal for any text.
    static std::string Quote(const std::string& s) {
        std::string r = "'";
        for (char c : s) {
            if (c == '\\' || c == '\'') r += '\\';
            r += c;
        }
        return r + "'";
    }
    
    static std::string ClientType(const std::string& t) {
        try {
            auto col = CreateColumnByType(t);
            return col ? col->GetType().GetName() : "nullptr";
        } catch (const std::exception& e) {
            return std::string("EXCEPTION: ") + e.what();
        }
    }
    
    static std::string ServerType(const std::string& t) {
        try {
            Client client(Options());
            std::string r;
            client.Select("SELECT toTypeName(defaultValueOfTypeName(" + Quote(t) + "))",
                [&](const Block& b) {
                    for (size_t i = 0; i < b.GetRowCount(); ++i)
                        r = std::string(b[0]->As<ColumnString>()->At(i));
                });
            return r;
        } catch (const std::exception& e) {
            std::string m = e.what();
            return "SERVER ERROR: " + m.substr(0, m.find('\n')).substr(0, 90);
        }
    }
    
    int main() {
        // Shown as C++ literals: "\\N" is the two characters \N in the type name.
        for (const std::string t : {
                 "Enum8('a\\Nb' = 1)",
                 "Enum8('a\\=b' = 1)",
                 "Enum8('a\\/b' = 1)",
                 "Enum8('a\\\tb' = 1)",          // backslash + literal TAB
                 "Enum8('a\\\"b' = 1)",
                 "Enum8('a\\`b' = 1)",
                 "Enum8('a\\x41b' = 1)",
                 "Enum8('a\\zb' = 1)",
                 "Enum8('a\\Nb' = 1, 'ab' = 2)",
                 "Tuple(`a\\Nb` Int8, c Int8)",
                 "Tuple(`a\\=b` Int8, c Int8)",
             }) {
            std::cout << t << "\n  client: " << ClientType(t) << "\n  server: " << ServerType(t) << "\n";
        }
    
        try {
            auto col = CreateColumnByType("Enum8('a\\Nb' = 1)")->As<ColumnEnum8>();
            col->Append(std::string("ab"));
            std::cout << "append \"ab\": ok\n";
        } catch (const std::exception& e) {
            std::cout << "append \"ab\": EXCEPTION: " << e.what() << "\n";
        }
        return 0;
    }

    Build (in the source tree, after cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DWITH_OPENSSL=ON && cmake --build build):

    clang++ -std=c++17 $(grep -o -- "-DCH_[A-Z0-9_]*=[01]" build/build.ninja | sort -u) -I. -Icontrib/absl main.cpp -o repro \
      -Wl,--start-group $(find build -name "*.a") -Wl,--end-group -lssl -lcrypto
    

    Output on master (58453c2), ClickHouse 26.10.4.207 (the fourth type name has a literal TAB after the backslash):

    Enum8('a\Nb' = 1)
      client: Enum8('a\\Nb' = 1)
      server: Enum8('ab' = 1)
    Enum8('a\=b' = 1)
      client: Enum8('a\\=b' = 1)
      server: Enum8('a=b' = 1)
    Enum8('a\/b' = 1)
      client: Enum8('a\\/b' = 1)
      server: Enum8('a/b' = 1)
    Enum8('a\	b' = 1)
      client: Enum8('a\\\tb' = 1)
      server: Enum8('a\tb' = 1)
    Enum8('a\"b' = 1)
      client: Enum8('a\\"b' = 1)
      server: Enum8('a"b' = 1)
    Enum8('a\`b' = 1)
      client: Enum8('a\\`b' = 1)
      server: Enum8('a`b' = 1)
    Enum8('a\x41b' = 1)
      client: Enum8('a\\x41b' = 1)
      server: Enum8('aAb' = 1)
    Enum8('a\zb' = 1)
      client: Enum8('a\\zb' = 1)
      server: Enum8('a\\zb' = 1)
    Enum8('a\Nb' = 1, 'ab' = 2)
      client: Enum8('a\\Nb' = 1, 'ab' = 2)
      server: SERVER ERROR: DB::Exception: Duplicate names in enum: 'ab' = 1 and 2: In scope SELECT toTypeName(default
    Tuple(`a\Nb` Int8, c Int8)
      client: Tuple(`a\Nb` Int8, c Int8)
      server: Tuple(
        ab Int8,
        c Int8)
    Tuple(`a\=b` Int8, c Int8)
      client: Tuple(`a\=b` Int8, c Int8)
      server: Tuple(
        `a=b` Int8,
        c Int8)
    append "ab": EXCEPTION: map::at
    

    Automated triage by Polyglot AI. Classification added on 2026-11-19. A maintainer makes the final decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions