Skip to content

Decimal(P, S) with P > 38 (Decimal256) is accepted and read as 16-byte values: wrong data with no error, Insert blocks #620

Open

Description

@polyglotAI-bot

Symptom

ClickHouse sends a Decimal256(S) column as Decimal(76, S), and a Decimal(P, S) column with 39 <= P <= 76 as Decimal(P, S). The server writes 32 bytes per value for these types. The library does not support Decimal256 (#377), but it does not reject these type names. It creates a ColumnDecimal that stores 16 bytes per value, the same as Decimal128.

  • Select() reads 16 of the 32 bytes per value. The rest of the block is then read as values and as the next packet.
    • Without compression, the callback can get wrong values with no error. For SELECT toDecimal256(arrayElement([1, 2, 0, 0], number + 1), 2) FROM numbers(4) the server returns 1, 2, 0, 0. The client returns 1.00, 0.00, 2.00, 0.00, and Select() returns normally.
    • In other cases, the callback gets wrong values first, and then Select() throws an error that does not name the type: unimplemented 72, can't read packet type from input stream, unsupported custom serialization.
    • With LZ4, Select() throws some data was not read.
  • Insert() of a ColumnDecimal(76, 2) into a Decimal256(2) column sends 16 bytes per value. The server waits for the remaining bytes. Insert() blocks until the server receive_timeout (300 s by default), and then throws DB::NetException: Timeout exceeded while reading from socket.

The explicit name is rejected: CreateColumnByType("Decimal256(2)") returns nullptr. Only the generic name Decimal(P, S), which the server sends, is accepted.

Reproduction

master at 2f8583f, default 2.x build (also with -DCH_USE_3X_API=ON), clang 18.1, Linux x86-64, ClickHouse server 26.10.4.207 on localhost:9000.

The server types and sizes:

$ curl -s localhost:8123 --data-binary "SELECT toTypeName(toDecimal256(1, 2)), toTypeName(CAST(1, 'Decimal(39, 2)')), byteSize(CAST(1, 'Decimal(39, 2)')), byteSize(CAST(1, 'Decimal(38, 2)'))"
Decimal(76, 2)	Decimal(39, 2)	32	16
$ curl -s localhost:8123 --data-binary "SELECT toDecimal256(arrayElement([1, 2, 0, 0], number + 1), 2) FROM numbers(4) FORMAT TSV"
1
2
0
0

Program:

#include <clickhouse/client.h>
#include <clickhouse/columns/factory.h>
#include <iostream>
using namespace clickhouse;

int main() {
    std::cout << CreateColumnByType("Decimal(76, 2)")->Type()->GetName() << "\n";  // accepted
    std::cout << (CreateColumnByType("Decimal256(2)") ? "column" : "nullptr") << "\n";

    Client client(ClientOptions().SetHost("localhost").SetPort(9000));

    // Select: wrong values, no error
    client.Select("SELECT toDecimal256(arrayElement([1, 2, 0, 0], number + 1), 2) AS d FROM numbers(4)",
        [](const Block& b) {
            if (b.GetRowCount() == 0) return;
            if (auto d = b[0]->As<ColumnDecimal>())
                for (size_t r = 0; r < d->Size(); ++r) std::cout << d->StringAt(r) << "\n";
        });

    // Insert: blocks for 300 s, then throws
    client.Execute("DROP TABLE IF EXISTS t_dec256");
    client.Execute("CREATE TABLE t_dec256 (d Decimal256(2)) ENGINE = Memory");
    auto col = std::make_shared<ColumnDecimal>(76, 2);
    col->Append(std::string("1.50"));
    col->Append(std::string("-2.25"));
    Block block;
    block.AppendColumn("d", col);
    client.Insert("t_dec256", block);
}

Output:

Decimal(76,2)
nullptr
1.00
0.00
2.00
0.00

Then Insert() blocks. During the wait, system.processes shows the INSERT INTO t_dec256 query with elapsed 15 s and more. After 5 min, Insert() throws:

DB::NetException: Timeout exceeded while reading from socket (peer: [::1]:38554, local: [::1]:9000, 300000 ms)

With SetConnectionRecvTimeout(8s), Insert() throws can't receive string data: Resource temporarily unavailable instead. In both cases the table stays empty.

Other results with the same build and server (callback values, then the result of Select()):

Query Compression Callback values Select()
SELECT toDecimal256(number + 1, 2) FROM numbers(2) none 1.00, 0.00 (server: 1, 2) unimplemented 72
SELECT CAST(number + 1, 'Decimal(39, 2)') FROM numbers(2) none 1.00, 0.00 (server: 1, 2) unimplemented 72
SELECT toDecimal256(-1, 2) none -1.00 can't read packet type from input stream
SELECT toDecimal256(1, 2) none 1.00 returns normally
SELECT CAST(number + 1, 'Decimal(39, 2)') AS d, toUInt8(7) AS x FROM numbers(2) none none unsupported custom serialization
SELECT CAST(number + 1, 'Nullable(Decimal256(2))') FROM numbers(2) none column Nullable(Decimal(76,2)) unimplemented 72
SELECT [toDecimal256(number + 1, 2)] FROM numbers(2) none column Array(Decimal(76,2)) unimplemented 72
all of the above LZ4 none some data was not read

Contrast: Decimal(38, 2) and toDecimal128(...) give the correct values for Select(), and Insert() of a ColumnDecimal(38, 2) into a Decimal128(2) column stores 1.5 and -2.25.

Expected

The library rejects a Decimal precision that it does not support, before it reads or sends data. Select() throws unsupported column type: Decimal(76, 2), as it does for Decimal256(2) and other unsupported type names at the root. ColumnDecimal(76, 2) throws a ValidationError instead of creating a column with the wrong value size.

Root cause

  • clickhouse/types/type_parser.cpp:72 maps the name Decimal to Type::Decimal for every precision.
  • clickhouse/columns/factory.cpp:93-94 creates ColumnDecimal(precision, scale) from the AST and does not check the precision.
  • clickhouse/columns/decimal.cpp:64-73: the constructor uses ColumnInt128 (16 bytes) for every precision above 18. ColumnDecimal::LoadBody and SaveBody then read and write 16 bytes per value.
  • clickhouse/types/types.cpp:320: the precision check in the DecimalType constructor is a comment, // TODO: assert(precision <= 38 && precision > 0);. This code has been the same since Decimal support was added (9ed5193), so the latest release v2.6.2 has the same behavior.

Suggested fix

  • In CreateColumnFromAst (factory.cpp), return nullptr for Decimal(P, S) when P is not in 1..38. Client then throws unsupported column type: Decimal(76, 2). Inside Array, Nullable and the other wrappers, the factory already gives nullptr for a nested type that it cannot build (Reject wrapper types built over unsupported nested types #578).
  • In the DecimalType constructor (or in ColumnDecimal(size_t, size_t)), replace the TODO with a ValidationError for a precision that is not in 1..38. Then a caller cannot build a column that Insert() sends with the wrong size.
  • Keep: Decimal(1..38, S), Decimal32(S), Decimal64(S) and Decimal128(S) work as now. Support for Decimal256 stays the subject of Support Decimal256 #377.

Related

Provenance

Found by automated analysis of the client (Polyglot AI) during the issue triage of #377. Verified with the program above against a live ClickHouse server, not by code inspection only.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions