Repository navigation
Run the ClickHouse core SQL tests through clickhouse-cpp - #621
Open
polyglotAI-bot wants to merge 1 commit into
Open
polyglotAI-bot wants to merge 1 commit into
polyglotAI-bot wants to merge 1 commit into
Conversation
Add tests/clickhouse-test-runner/: a small C++ program that imitates clickhouse-client well enough for the upstream runner (tests/clickhouse-test) to drive it without changes. It sends each statement through clickhouse::Client over the native protocol, reads each value with the typed column classes, and writes the values as TabSeparated text itself. Thus the upstream .reference files check how clickhouse-cpp decodes the values. - The shim, its unit tests and live tests (clickhouse-test-runner-ut), behind the CMake option BUILD_CLICKHOUSE_TEST_RUNNER (OFF by default). - allowlist.txt: 10,004 tests that run, and 981 known failures in 23 groups (19 client groups, 4 shim groups). - clickhouse-version (26.10.4.207) and scripts: fetch the upstream tests of that version, run the allowlist (shards, KNOWN_FAILURES), and run the same tests through the real clickhouse-client. - The upstream-sql-tests workflow: nightly, manual, and on pull requests that change the harness, the library or the workflow; one build job and 8 shards. - README.md, AGENTS.md (triage rules), CLAUDE.md, and a pointer in the root AGENTS.md. It is a port of the Go harness (ClickHouse/clickhouse-go#2141). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI review gate — AI review passedClean review from: Cursor Bugbot. |
polyglotAI-bot
marked this pull request as ready for review
November 19, 2026 05:17
AI review gate — auto-approval withheldThe AI review gate is green, but the approver did not approve What to do: ask someone with write access who is not the PR author to approve this pull request. History (latest 5 heads, newest first) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds
tests/clickhouse-test-runner/: a harness that runs the ClickHouse core SQL tests (tests/queries/0_stateless) through clickhouse-cpp, and a workflow that runs them in CI.clickhouse-clientwell enough for the upstream runner (tests/clickhouse-test) to drive it without changes: the same flags, stdin and--querybehavior, test hints,echo, and theextract-from-configshortcut.clickhouse::Client, over the native protocol (the only protocol of clickhouse-cpp). OneClientis one TCP connection, so the session (SET,USE, temporary tables) stays for the whole invocation. Each statement runs withClient::Execute(Query)andQuery::OnData. The command-line settings go with each query (Query::SetSetting), not asSETstatements; query parameters go withQuery::SetParam. Server errors come asServerException.ColumnInt32::At,ColumnNullable::IsNull,ColumnArray,ColumnTuple,ColumnMap,ColumnLowCardinalityT<...>,ColumnDecimal, ...). A type-directed renderer then writes the values asTabSeparatedtext. Thus the upstream.referencefiles are the oracle for the client's type parsing and decoding. The program does not ask the server to format the text.CMakeLists.txtadds the harness only with the new optionBUILD_CLICKHOUSE_TEST_RUNNER(OFFby default, top-level builds only). Bazel does not see it. No file underclickhouse/changes.Tracking issue: ClickHouse/integrations#813
Contents
split,hintandrendertests) and 6 live tests that compare its rendering with the server's ownFORMAT TSVoutput for many expressions (clickhouse-test-runner-ut, gtest, separate fromclickhouse-cpp-ut).allowlist.txt: the tests that run, and the known failures in groups. Each group header gives the cause, a reproduction, andIssue: not filed yet.clickhouse-version: the server version and the upstream tests are pinned together (26.10.4.207, tagv26.10.4.207-stable, as in the Go harness).scripts/:fetch-upstream.sh(sparse shallow checkout; also generateserror_codes_generated.incfromsrc/Common/ErrorCodes.cppwithsed),build.sh,run-upstream-tests.sh(shards,KNOWN_FAILURES=<group>, version check, missing-test warning),run-with-real-client.sh(the same tests through the realclickhouse-client, for triage),selectors.sh,pinned-version.sh,gen-error-codes.sh..github/workflows/upstream-sql-tests.yml: nightly, manual (inputclickhouse_version), on pushes tomasterand on pull requests that change the harness, the workflow or the library. Abuildjob (the shim and its unit tests, once) and 8 shards at-j 4. The server is the stock image of the pinned version withCLICKHOUSE_SKIP_USER_SETUP=1and the host name of the runner.permissions: contents: read. Artifacts on failure (test output, program log, server error log).README.md(local runs, how the shim imitatesclickhouse-client, the allowlist format, how to grow it, CI, the version bump),AGENTS.md(triage rules, environment causes),CLAUDE.md(@AGENTS.md), and a pointer in the rootAGENTS.md.Results (server 26.10.4.207, native protocol)
.sql/.sql.j2tests: 9,982 OK, 2,053 FAIL, 926 skipped by the runner; 1,050 s at-j 6.[untriaged]group): environment 1,035, client 682, shim 298; 38 failures were fixed in the shim. (One more client failure, 03024, came from the allowlist checks.)only=lines (one protocol).-j 6: 10,004 OK and 0 FAIL each time (692 s, 676 s, 682 s).-j 4. One shard job takes 138-173 s on this PR, with about 2 minutes of tests. NoRETRY_FAILED: no known client bug makes random tests fail.clickhouse-clientalso fails these tests on the same server): no cluster / ZooKeeper / S3, upstream CI server configuration, features ofclickhouse-clientitself.AGENTS.mdlists them. 15 tests that pass in upstream CI are left out on purpose, because in a full run on an 8 GiB server they fail also through the real client:EXPLAIN PIPELINEwidths and thread counts that depend on the server load (13), a test that needs more free server memory (1), and an asynchronousDROPin anAtomicdatabase (1).Client bugs found
These are not fixed in this PR, and no issues are filed. The maintainers decide when to file and fix them. All are over the native protocol (the only protocol).
The reproductions use only the public API:
"→" is the actual result, "want" is the server's
TabSeparatedoutput.Wrong values (silent data corruption):
Voidfor a type name that the client does not know when it is nested inArray,Nullable,Map,Tuple,LowCardinalityorSimpleAggregateFunction(Int256,UInt256,BFloat16,Interval*,Dynamic,Variant,AggregateFunction, ...).ColumnNothingreads 0 bytes per value, so the rest of the block is read from the wrong bytes.SELECT toNullable(toInt256(1)) AS x→ typeNullable(Void),IsNull(0)= 0, no error; want1.SELECT map(1, INTERVAL 1 DAY) AS m→ typeMap(UInt8, Void), the value is lost, no error.SELECT [1::Variant(UInt8)] AS a→[], thenUnimplementedError; a nextSELECT 42on the sameClientreturns a 39-rowProfileEventsblock as its result. Other results: a garbled column type in a later block, "can't read packet type", aServerExceptionwith code 0 and no text. At the top level, the same types give a clean "unsupported column type" error (groups below).Decimal(P, S)with P > 38 is read as 16-byte values; the server sends 32 bytes.SELECT toDecimal256(number + 1, 2) AS d FROM numbers(3)→1.00,0.00,2.00, thenUnimplementedError "unimplemented 44"; want1.00,2.00,3.00.SELECT CAST([0.1, NULL] AS Array(Nullable(Decimal256(76))))→ a wrong value (-0.0000...154262283864711497...) and no error.Execute, the socket stays open with the unread packets of that response. A caller that catches the exception and runs the next query on the sameClientgets the old data as the result, without an error.SELECT toDecimal256(number + 1, 2) AS d FROM numbers(3)(an exception), thenSELECT toDecimal128(number + 1, 2) AS d, 'x' AS s, [1,2] AS a FROM numbers(2)→ no block and no error, thenSELECT 42 AS after→ the 2 rows(d, s, a)of the statement before; want42.session_timezoneis ignored. The client's protocol revision (54459) is older thanTimezoneUpdate(54464), so the server never sends the session time zone.SET session_timezone = 'Asia/Tokyo', thenSELECT toDateTime('2020-01-01 00:00:00') AS t→ typeDateTimewithout a zone, and only the zone of the handshake (ServerInfo::timezone, UTC) is known: rendered2019-12-31 15:00:00; want2020-01-01 00:00:00.Packets that the client does not implement:
SELECT number % 2 AS k, count() FROM numbers(4) GROUP BY k WITH TOTALS→ the 2 data rows, thenUnimplementedError "unimplemented 7", and the connection is out of sync; want the rows, an empty line, then the totals row.SELECT number FROM numbers(3) SETTINGS extremes = 1→ the 3 data rows, thenUnimplementedError "unimplemented 8"; want the 3 rows, an empty line, then the extremes rows (0,2).Types that the client cannot read (a clean
UnimplementedError "unsupported column type: <type>"at the top level):SELECT 1::Dynamic; want1. AlsoDynamic(max_types=N).SELECT 1::Variant(UInt8); want1.SELECT toInt256(1),SELECT toUInt256(1); want1. (03024 readsgenerateRandom()without a structure, so it fails only when the server picksInt256orUInt256.)SELECT toIntervalSecond(1); want1. All units,Nanosecond..Year.SELECT CAST((1., 2.)::Point AS Geometry),SELECT [(1., 2.)]::LineString, alsoMultiPointandMultiLineString; want(1,2),[(1,2)].Point,Ring,PolygonandMultiPolygonwork.SELECT [1]::QBit(Float32, 1); want[1].SELECT toBFloat16(1); want1.SELECT sumState(1)→ "unsupported column type: AggregateFunction(sum, UInt8)"; want the state bytes.flatten_nested = 0,CREATE TABLE tn (n Nested(a UInt8, b String)) ENGINE = Memory; INSERT INTO tn VALUES ([(1, 'x')]); SELECT n FROM tn→ "unsupported column type: Nested(a UInt8, b String)"; want[(1,'x')].Other type errors:
SELECT '{"a": 1}'::JSON AS j→ProtocolError "Unsupported JSON serialization version. Make sure output_format_native_write_json_as_string=1 is set."; want{"a":1}. The client readsJSONonly as aStringwith that setting. The shim does not set it, because it changes the query and shows inquery_log. (With the setting,JSON(SHARED a UInt64)also parses.)SELECT tuple() AS t→ValidationError "AST child element index out of bounds: 0"; want().SELECT 1withQuery::SetSetting("output_format_native_encode_types_in_binary_format", "1")→UnimplementedError "unsupported custom serialization"; want1. The client reads the binary type encoding as a type name.SELECT toLowCardinality(true) AS b→AssertionError "ItemView value size mismatch for Bool expected: 1, got: 0"; wanttrue.FixedSizeForDictionaryTypeinclickhouse/columns/lowcardinality.cpphas noType::Boolcase.LowCardinality(UInt8)works.Connection behavior:
Executeof anINSERTthat needs data from the client never returns. ForINSERT INTO t VALUES(no data),INSERT ... SELECT ... FROM input(...)andINSERT ... FROM INFILE, the server sends the header block and waits for data;Executedoes not send data and does not refuse. With the defaultClientOptions(no receive timeout) it waits for ever; with a receive timeout it throwsstd::system_error, and the next query fails with "Unexpected packet Query received from client".CREATE TEMPORARY TABLE t (x UInt8), thenINSERT INTO t VALUES→OnDatagets the header block, thenExecutewaits; wantNO_DATA_TO_INSERT(asclickhouse-client).ClientOptions().SetPingBeforeQuery(true), the ping before each query runs inRetryGuard, which callsResetConnection()on astd::system_error. A broken connection is then replaced by a new connection with a new session, without an error.SETvalues,USEand temporary tables are lost. WithSetConnectionRecvTimeout(2 s):CREATE TEMPORARY TABLE t (x UInt8); INSERT INTO t VALUES (1); SET max_threads = 3, thenINSERT INTO t VALUES(waits for data:std::system_errorafter 2 s), thenSELECT count() FROM t→ServerException101, thenSELECT count() FROM t→ServerException60 (UNKNOWN_TABLE), andSELECT getSetting('max_threads')→32(the default) with no error; want an error for each query after the broken connection (as withoutSetPingBeforeQuery). (The same class as the Gonative-pool-race-new-connection: a second connection losesSETand temporary tables without an error.)Go bug classes, checked on purpose, no bug found:
DateTime64outside 1677-09-21 .. 2262-04-11 (ColumnDateTime64::Atgives the raw ticks; read and write are exact),Time64precision (all digits for P = 1 .. 9),Nullablenull-map bytes other than 1 (IsNullis!= 0), NULL in aLowCardinality(Nullable(...))element of a named tuple, a statement without a result block, tuples in tuples /Nullable(Tuple)/Int128in a tuple, tuple element names in backticks with spaces,Mapkeys of typeArray/Map/Tuple. With the default options, a broken connection is never replaced silently. The server-info query in the user's session (HTTP) does not apply.Shim gaps (known failures of the program, not of the client)
TabSeparatedfamily,LineAsString,ValuesandNull. The shim readsNativeblocks and writes the text itself; it never passesFORMATthrough to the server.SELECT 1 FORMAT JSONEachRow.SET dialect = 'kusto'; let x = 1; print x;.SELECT toDecimal32(1.5, 4) SETTINGS output_format_decimal_trailing_zeros = 1→1.5; want1.5000.;thatclickhouse-clientcannot parse ends at the end of its line there; the shim has no SQL parser and splits only at a top-level;.How to run locally
Details:
tests/clickhouse-test-runner/README.md.Checks
-DBUILD_TESTS=ON): the option isOFF, no harness target;clickhouse-cpp-utunchanged.-Wall -Wextra -Werror): 43 unit tests and 6 live tests pass.clang-format(the repository's.clang-format) is clean on the shim sources.actionlintandshellcheckare clean.🤖 Generated with Claude Code
Note
Medium Risk
Introduces a large CI workflow and new optional build target, but does not change default library builds or
clickhouse/runtime code; main risk is CI cost/flakiness and maintenance of the allowlist and pinned server version.Overview
Adds
tests/clickhouse-test-runner, a clickhouse-client shim that drives ClickHouse’s upstreamtests/clickhouse-testsuite through clickhouse-cpp (native protocol, client-side TabSeparated rendering), plus scripts to fetch pinned upstream tests, build, shard runs, and triage against the real client.Wires it in with an opt-in
BUILD_CLICKHOUSE_TEST_RUNNERCMake flag (default OFF, so normal library builds are unchanged) and a new.github/workflows/upstream-sql-tests.ymljob: build shim + unit tests, then 8 parallel shards of the allowlist (~10k tests) against a Docker ClickHouse image, with nightly/manual/PR triggers. Docs and rootAGENTS.mdpoint maintainers at harness triage rules.Known failures are tracked in
allowlist.txt(client vs shim groups); server version is pinned inclickhouse-versionwith generatederror_codes_generated.inc.Reviewed by Cursor Bugbot for commit 0e9e4c8. Bugbot is set up for automated code reviews on this repo. Configure here.