Repository navigation
Reject undeclared Enum8 and Enum16 values on insert - #617
Open
Indian-boult wants to merge 2 commits into
Open
Indian-boult wants to merge 2 commits into
Indian-boult wants to merge 2 commits into
Conversation
ColumnEnum<T> checks a value only when the caller passes checkValue to
Append or SetAt. The vector constructors and Append(ColumnRef) do not
check at all. SaveBody then writes the values as they are. The server
stores a Native column whose type is the same as the table column
without a check, and then it cannot read the row. SELECT toString(e)
fails with UNKNOWN_ELEMENT_OF_ENUM.
SaveBody now checks every value before it writes the column, and it
throws ValidationError for a value that the type does not declare. All
write paths end in SaveBody, so this one check covers all of them.
LoadBody does not change, so reads from the server do not change. The
check uses a table of all values of T, so each row costs one lookup.
The nested value of a NULL row has no meaning. ColumnNullableT appends
0 there, and the server sends 0 there too. ColumnNullable::SaveBody now
gives its null map to a nested ColumnEnum8 or ColumnEnum16, and the
check skips the NULL rows. Without this, the check would reject a NULL
in Nullable(Enum8('a' = 1)) and a block that was read from the server.
Fixes ClickHouse#602
MSVC treats int16_t(0) as a null pointer constant, so it could also convert the value to ColumnRef and reported the Append call as ambiguous.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ColumnEnum<T>sends a value that its Enum type does not declare, and the server stores it. After that, the server cannot read the row.On master 58453c2 with ClickHouse 26.10.4.207, each of these inserts goes through without an error.
Append(0)andAppend(5)on a column ofEnum8('a' = 1){0, 5}Append(ColumnRef)from a column ofEnum8('b' = 7)SetAt(0, 9)with the default flagAppend(0)andAppend(-300)on a column ofEnum16('a' = 1000)Nullable(Enum8('a' = 1))row with 3, and anArray(Enum8('a' = 1))element 4SELECT CAST(e AS Int16)returns1 0 5 0 5 7 9for theEnum8table. The server then fails to read the rows.The column checks a value only when the caller passes
checkValue = truetoAppendorSetAt. The vector constructors andAppend(ColumnRef)have no check.SaveBodywrites the bytes as they are. The server does not convert a Native column whose type is the same as the table column, so it stores the bytes.ColumnEnum<T>::SaveBodynow checks every value before it writes the column. For a value that the type does not declare, it throwsValidationError, for exampleEnum type Enum8('a' = 1) doesn't have value 0. All write paths end inSaveBody, andArray,Tuple,MapandNullablecall theSaveBodyof the nested column. So this one check covers the defaultAppendandSetAt, both vector constructors,Append(ColumnRef)and the containers. The triage on the issue names the same place.LoadBodydoes not change, so reads from the server do not change.The check makes a table of all values of
Tonce for each call, 256 bits forEnum8and 65536 bits forEnum16. After that, each row costs one lookup. In a Release build,Saveof 10 million rows took 3.0 ms forEnum8and 3.2 ms forEnum16, against 0.17 ms and 0.37 ms on master. OneSavecall of anEnum16column costs about 0.12 µs more for the table.The nested value of a NULL row has no meaning.
ColumnNullableT::Append(std::nullopt)appends 0 there, and the server sends 0 there when it returns a NULL. A check without the null map would reject a NULL inNullable(Enum8('a' = 1)), and it would reject a block that came from the server. SoColumnNullable::SaveBodygives its null map to a nestedColumnEnum8orColumnEnum16, and the check skips the NULL rows. This uses a private overload ofSaveBodyand a friend declaration, so the public API does not change.The check throws after the client has written a part of the block. On master, the client then stays in the
Insertingstate, as after other failed inserts, and the README says to callResetConnection()after an exception. If the application does not do that, theClientdestructor callsEndInsert(), and the server waits for the rest of the block. In my runs with 100 and 100000 rows and no compression, the destructor did not return within 20 seconds. #611 closes the connection after a failure in the middle of an exchange, and it covers this case. With #611 applied on top of this change, the next query worked withoutResetConnection(), and the destructor returned in 4 ms and 11 ms. This change does not touchclient.cpp, so it does not conflict with #611. The server test callsResetConnection()after the error, so it passes with and without #611.An insert with an undeclared value now throws
ValidationErroron the client, and the server stores nothing. Before, the insert went through and made rows that the server cannot read. Declared values, NULL rows, appends by label and reads from the server do not change.Append(value, true)andSetAt(n, value, true)still throw at the call. An unknown label inAppend(name)still throwsstd::out_of_range, as before.For comparison, clickhouse-go checks an Enum ordinal when the application appends it, and returns
unknown elementfromlib/column/enum8.go. The Java client-v2 writes the number without a check.This change adds these tests. All of them fail on master and pass with this change.
ColumnsCase.Enum8SaveRejectsUndeclaredValuesaves declared values by number and by label, with -128 and 127, and then rejects 0 from the defaultAppend.ColumnsCase.Enum16SaveRejectsUndeclaredValuedoes the same forEnum16with -32768, 1000 and 32767, and rejects -1 from the defaultSetAt.ColumnsCase.EnumSaveRejectsUndeclaredValueFromAnyPathcovers the vector constructor,Append(ColumnRef)from another Enum type, andArray(Enum8).ColumnsCase.EnumSaveDoesNotCheckNullRowssaves NULL rows ofNullable(Enum8)andNullable(Enum16)whose nested value is 0, and then rejects an undeclared value in a row that is not NULL.ClientCase.EnumUndeclaredValueIsNotInsertedruns with bothClientCaseparameter sets. It inserts a block with an undeclared value and expects the exactValidationError. Then it checks that the table is empty, inserts the block with a declared value, and reads back the labels and the NULL rows.With this change, the full
clickhouse-cpp-utbinary ran 1131 tests against ClickHouse 26.10.4.207, in a Debug build with Apple clang on macOS. 1081 passed, 50 were skipped, and none failed. Master skips the same 50 tests. They are generic column tests that do not apply to some types, performance tests that skip in Debug builds, and one test that reads/proc/self/status. On master with the new tests, 1075 passed and the 6 new test instances failed. With #611 applied on top, 1145 ran, 1095 passed, 50 were skipped, and none failed.This change does not touch the Enum type parser or
EnumType, which #604, #607 and #609 are about.Fixes #602
Note
Medium Risk
Touches the hot insert/serialize path for all Enum8/Enum16 writes (including nested nullable) with a small per-save lookup table and stricter failure behavior mid-block.
Overview
Enum columns now fail on the client before serializing data when a row holds a numeric value that is not declared on the
Enum8/Enum16type, throwingValidationErrorfromColumnEnum::SaveBodyinstead of sending bytes the server would store but cannot read.ColumnNullablepasses its null bitmap into a private enumSaveBodyoverload so NULL rows are not validated (nested placeholder0is ignored), coveringNullable(Enum*)inserts and round-tripped blocks.Unit and client tests assert rejection across vector construction,
Append(ColumnRef), arrays, and that failed inserts leave the table empty.Reviewed by Cursor Bugbot for commit 5481412. Bugbot is set up for automated code reviews on this repo. Configure here.