Skip to content

Reject undeclared Enum8 and Enum16 values on insert - #617

Open
Indian-boult wants to merge 2 commits into
ClickHouse:masterfrom
Indian-boult:fix/enum-undeclared-values
Open

Indian-boult wants to merge 2 commits into
ClickHouse:masterfrom
Indian-boult:fix/enum-undeclared-values

Conversation

@Indian-boult

@Indian-boult Indian-boult commented Nov 18, 2026 •

Copy link
Copy Markdown

ColumnEnum<T> sends a value that its Enum type does not declare, and the server stores it. After that, the server cannot read the row.

On master 58453c2 with ClickHouse 26.10.4.207, each of these inserts goes through without an error.

  • Append(0) and Append(5) on a column of Enum8('a' = 1)
  • the vector constructor with {0, 5}
  • Append(ColumnRef) from a column of Enum8('b' = 7)
  • SetAt(0, 9) with the default flag
  • Append(0) and Append(-300) on a column of Enum16('a' = 1000)
  • a Nullable(Enum8('a' = 1)) row with 3, and an Array(Enum8('a' = 1)) element 4

SELECT CAST(e AS Int16) returns 1 0 5 0 5 7 9 for the Enum8 table. The server then fails to read the rows.

SELECT toString(e) FROM t_cpp602_e8
Code: 691. DB::Exception: Unexpected value 0 in enum: while executing 'FUNCTION toString(__table1.e :: 1) -> toString(__table1.e) String : 0'. (UNKNOWN_ELEMENT_OF_ENUM) (version 26.10.4.207 (official build))

SELECT e FROM t_cpp602_e8
Code: 691. DB::Exception: Unexpected value 0 in enum: While executing ParallelFormattingOutputFormat. (UNKNOWN_ELEMENT_OF_ENUM) (version 26.10.4.207 (official build))

The column checks a value only when the caller passes checkValue = true to Append or SetAt. The vector constructors and Append(ColumnRef) have no check. SaveBody writes the bytes as they are. The server does not convert a Native column whose type is the same as the table column, so it stores the bytes.

ColumnEnum<T>::SaveBody now checks every value before it writes the column. For a value that the type does not declare, it throws ValidationError, for example Enum type Enum8('a' = 1) doesn't have value 0. All write paths end in SaveBody, and Array, Tuple, Map and Nullable call the SaveBody of the nested column. So this one check covers the default Append and SetAt, both vector constructors, Append(ColumnRef) and the containers. The triage on the issue names the same place. LoadBody does not change, so reads from the server do not change.

The check makes a table of all values of T once for each call, 256 bits for Enum8 and 65536 bits for Enum16. After that, each row costs one lookup. In a Release build, Save of 10 million rows took 3.0 ms for Enum8 and 3.2 ms for Enum16, against 0.17 ms and 0.37 ms on master. One Save call of an Enum16 column costs about 0.12 µs more for the table.

The nested value of a NULL row has no meaning. ColumnNullableT::Append(std::nullopt) appends 0 there, and the server sends 0 there when it returns a NULL. A check without the null map would reject a NULL in Nullable(Enum8('a' = 1)), and it would reject a block that came from the server. So ColumnNullable::SaveBody gives its null map to a nested ColumnEnum8 or ColumnEnum16, and the check skips the NULL rows. This uses a private overload of SaveBody and a friend declaration, so the public API does not change.

The check throws after the client has written a part of the block. On master, the client then stays in the Inserting state, as after other failed inserts, and the README says to call ResetConnection() after an exception. If the application does not do that, the Client destructor calls EndInsert(), and the server waits for the rest of the block. In my runs with 100 and 100000 rows and no compression, the destructor did not return within 20 seconds. #611 closes the connection after a failure in the middle of an exchange, and it covers this case. With #611 applied on top of this change, the next query worked without ResetConnection(), and the destructor returned in 4 ms and 11 ms. This change does not touch client.cpp, so it does not conflict with #611. The server test calls ResetConnection() after the error, so it passes with and without #611.

An insert with an undeclared value now throws ValidationError on the client, and the server stores nothing. Before, the insert went through and made rows that the server cannot read. Declared values, NULL rows, appends by label and reads from the server do not change. Append(value, true) and SetAt(n, value, true) still throw at the call. An unknown label in Append(name) still throws std::out_of_range, as before.

For comparison, clickhouse-go checks an Enum ordinal when the application appends it, and returns unknown element from lib/column/enum8.go. The Java client-v2 writes the number without a check.

This change adds these tests. All of them fail on master and pass with this change.

  • ColumnsCase.Enum8SaveRejectsUndeclaredValue saves declared values by number and by label, with -128 and 127, and then rejects 0 from the default Append.
  • ColumnsCase.Enum16SaveRejectsUndeclaredValue does the same for Enum16 with -32768, 1000 and 32767, and rejects -1 from the default SetAt.
  • ColumnsCase.EnumSaveRejectsUndeclaredValueFromAnyPath covers the vector constructor, Append(ColumnRef) from another Enum type, and Array(Enum8).
  • ColumnsCase.EnumSaveDoesNotCheckNullRows saves NULL rows of Nullable(Enum8) and Nullable(Enum16) whose nested value is 0, and then rejects an undeclared value in a row that is not NULL.
  • ClientCase.EnumUndeclaredValueIsNotInserted runs with both ClientCase parameter sets. It inserts a block with an undeclared value and expects the exact ValidationError. Then it checks that the table is empty, inserts the block with a declared value, and reads back the labels and the NULL rows.

With this change, the full clickhouse-cpp-ut binary ran 1131 tests against ClickHouse 26.10.4.207, in a Debug build with Apple clang on macOS. 1081 passed, 50 were skipped, and none failed. Master skips the same 50 tests. They are generic column tests that do not apply to some types, performance tests that skip in Debug builds, and one test that reads /proc/self/status. On master with the new tests, 1075 passed and the 6 new test instances failed. With #611 applied on top, 1145 ran, 1095 passed, 50 were skipped, and none failed.

This change does not touch the Enum type parser or EnumType, which #604, #607 and #609 are about.

Fixes #602


Note

Medium Risk
Touches the hot insert/serialize path for all Enum8/Enum16 writes (including nested nullable) with a small per-save lookup table and stricter failure behavior mid-block.

Overview
Enum columns now fail on the client before serializing data when a row holds a numeric value that is not declared on the Enum8/Enum16 type, throwing ValidationError from ColumnEnum::SaveBody instead of sending bytes the server would store but cannot read.

ColumnNullable passes its null bitmap into a private enum SaveBody overload so NULL rows are not validated (nested placeholder 0 is ignored), covering Nullable(Enum*) inserts and round-tripped blocks.

Unit and client tests assert rejection across vector construction, Append(ColumnRef), arrays, and that failed inserts leave the table empty.

Reviewed by Cursor Bugbot for commit 5481412. Bugbot is set up for automated code reviews on this repo. Configure here.

ColumnEnum<T> checks a value only when the caller passes checkValue to
Append or SetAt. The vector constructors and Append(ColumnRef) do not
check at all. SaveBody then writes the values as they are. The server
stores a Native column whose type is the same as the table column
without a check, and then it cannot read the row. SELECT toString(e)
fails with UNKNOWN_ELEMENT_OF_ENUM.

SaveBody now checks every value before it writes the column, and it
throws ValidationError for a value that the type does not declare. All
write paths end in SaveBody, so this one check covers all of them.
LoadBody does not change, so reads from the server do not change. The
check uses a table of all values of T, so each row costs one lookup.

The nested value of a NULL row has no meaning. ColumnNullableT appends
0 there, and the server sends 0 there too. ColumnNullable::SaveBody now
gives its null map to a nested ColumnEnum8 or ColumnEnum16, and the
check skips the NULL rows. Without this, the check would reject a NULL
in Nullable(Enum8('a' = 1)) and a block that was read from the server.

Fixes ClickHouse#602
MSVC treats int16_t(0) as a null pointer constant, so it could also convert
the value to ColumnRef and reported the Append call as ambiguous.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ColumnEnum writes undeclared Enum8/Enum16 ordinals by default; server stores them and cannot read them back

1 participant