[fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early by morningman · Pull Request #69945 · apache/doris

morningman · 2026-11-02T03:09:45Z

What problem does this PR solve?

Issue Number: None

Problem Summary:

In short

This ports the BE half of branch-4.x's #68630 to master. When an Arrow Flight SQL client cancels or closes a result stream before its end, master keeps the query running: the BE thread serving the stream waits on the result buffer without ever checking whether the client is still there, and the unread result blocks stay in memory, holding back the query, until the query times out or the delayed buffer cleanup runs minutes later. With this PR the BE notices the cancelled stream within 20 ms, and when the stream is served by the BE that holds its result (the usual case), that BE cancels the query and frees all of the query's Flight result buffers there. The FE route that also reaches the query's other BEs (cancelFlightQuery, which changes FrontendService.thrift) comes in a follow-up PR.

Background

A Flight SQL query leaves its result on the BEs. GetFlightInfo returns one endpoint per result buffer, and the client reads each endpoint with a DoGet stream. Since [fix](arrow-flight) Release a finished Flight query's coordinator instead of holding it until wait_timeout #67504 the FE closes the coordinator as soon as it has returned the FlightInfo, so it no longer cancels anything on behalf of an open stream.
DoGet is served by ArrowFlightBatchLocalReader when the result buffer is on the same BE, or by ArrowFlightBatchRemoteReader, which pulls the blocks from the result BE with the fetch_arrow_data BRPC.
The result buffer (ArrowFlightResultBlockBuffer) is keyed by the query id when parallel result sink is on (the default) and by the fragment instance id when it is off. When it is full, the result sink stops and the query waits for the reader.
ResultBufferMgr drops a buffer that nobody removed at execution_timeout + 5 s, or result_buffer_cancelled_interval_time (300 s by default) after the fragments writing it have finished, whichever comes first. Finishing with an error does not clear the unread blocks. cancel_plan_fragment is the BRPC the FE uses to cancel a query on a BE.

The problem, and what it cost

The local reader never looks at the client. get_arrow_batch loops on 20 ms waits until a block arrives, the buffer is closed or it fails. It has no access to ServerCallContext::is_cancelled(), so a DoGet cancelled while the buffer is empty keeps a Flight server thread waiting until the query produces a block or ends.
Closing a stream cancels nothing. The readers do not override RecordBatchReader::Close(), and their destructor only logs. A client that stops reading (a LIMIT reached on the client side, a cancelled cursor, a dropped connection) leaves the query running and its unread blocks in the buffer. When the result does not fit in the buffer, the full buffer back-pressures the result sink and the fragments, with their memory, wait until the query times out; the unread blocks go only when the buffer is dropped.
The remote reader cannot be interrupted. It waits for fetch_arrow_data with brpc::Join, which returns only when the result BE answers or arrow_flight_reader_brpc_controller_timeout_ms (300 s by default) passes.
An FE cancel leaves the results behind. cancel_plan_fragment cancels the fragments but not the query's Flight buffers, so their unread blocks stay until the delayed cleanup 300 s later.
A fetch for a removed buffer never completes. When fetch_arrow_data cannot find the buffer, it logs and returns without running done, so the caller waits for its BRPC timeout instead of getting an error. [fix](be) Return missing-buffer errors for MySQL and Arrow result fetches #67644 fixes the same line.

How this PR fixes it

Bounded waits, cancellation checks in the reader. get_arrow_batch(result, eos) waits at most 20 ms and then returns OK with no block and eos = false; eos now marks the real end. The local reader retries in a loop and checks is_cancelled() before each wait. DoGetStatement passes context.is_cancelled() to both readers.
The reader closes itself. ArrowFlightBatchReaderBase now implements ReadNext (which calls each reader's ReadNextImpl) and Close(). A stream counts as aborted when it is cancelled, closed before EOF, fails while reading, or is destroyed before EOF; a stream that reached EOF is never treated as aborted.
Aborting a local stream cancels the query on this BE. ResultBufferMgr keeps an index from query id to the query's Flight buffers, maintained under the buffer-map lock together with the map. cancel_arrow_flight_buffers removes every Flight buffer of the query from the map, then, outside the lock, fails each buffer (which wakes its waiting fetches and releases the back-pressure) and cancels the QueryContext it holds a weak reference to. Sibling buffers of the same query are cleared even after their QueryContext has expired.
Remote waits can be interrupted. The remote reader waits for its BRPC with a callback that checks is_cancelled() every 20 ms and calls brpc::StartCancel when the stream is gone.
cancel_plan_fragment drops the Flight buffers of an aborted query. It calls cancel_arrow_flight_buffers unless the reason is LIMIT_REACH or FINISHED, which can leave valid unread results that the client still has to read.
fetch_arrow_data answers when the buffer is gone. It completes the RPC with the lookup error. The line is identical to the one [fix](be) Return missing-buffer errors for MySQL and Arrow result fetches #67644 adds, so the two PRs merge in either order.

Differences from branch-4.x: the two consistency checks between the index and the buffer map are DORIS_CHECK (an if and a DCHECK in 4.x), a null check on the reader's memory tracker that both constructors make unnecessary is not carried over, and the ArrowFlightResultBlockBuffer constructor moved to the .cpp so that varrow_flight_result_writer.h, which 13 TUs include, does not include runtime/query_context.h; only the .cpp needs its definition. The new lines also take a few clang-tidy fixes (the destructor calls ArrowFlightBatchReaderBase::Close() explicitly instead of a virtual call, std::scoped_lock), and the test starts its BRPC server inside ScopedLeakCheckDisabler, like the other BE tests that start one, because brpc registers process-lifetime built-in services on its first start.

Not in this PR: the FE route. A remote reader's abort, and an abort on one result BE, still do not reach the query's other BEs; their fragments and buffers wait for the timeout as before. The follow-up adds FrontendService.cancelFlightQuery (same signature as branch-4.x), the BE broadcast to the FEs and the FE routing table, together with the FE unit test and the test_flight_cancel_cleanup regression test.

Situation	Before	After
Client cancels `DoGet` while the result buffer is empty (local reader)	Server thread waits until the query produces data or ends	Returns within 20 ms; the query is cancelled on this BE
Client closes a local stream before EOF	Query keeps running, back-pressured until it times out if the result does not fit; unread blocks kept for minutes	Query cancelled on this BE; all of its Flight buffers there freed at once
Client cancels while a remote fetch is pending	Waits for the result BE, up to 300 s	BRPC cancelled within 20 ms
FE cancels the query (`cancel_plan_fragment`, abort reason)	Unread blocks kept until the delayed cleanup, 300 s later	Flight buffers freed at once; `LIMIT_REACH` / `FINISHED` still keep them
`fetch_arrow_data` for a removed buffer	RPC never completes; caller waits for its timeout	Completes at once with an error
Stream read to EOF, then closed	Nothing cancelled	Nothing cancelled
New `DoGet` with the ticket of a stream aborted on its result BE	Reads the remaining blocks, or EOS	Fails: the buffer is gone

Results

ArrowFlightCancelTest (9 tests, new): early close and destruction free a back-pressured buffer and cancel the query; sibling buffers are freed after their QueryContext expired; EOF followed by close is not a cancellation; cancellation interrupts an empty-buffer wait; empty waits return for retry and do not end the stream; a conversion error cancels the query; a real Flight server sees a client-side Cancel() through ServerCallContext and cleans up.
ArrowFlightRemoteCancelTest (3 tests, new, against an in-process BRPC PInternalService): cancellation interrupts a pending fetch_arrow_data; an aborting cancel_plan_fragment frees the buffer and a following fetch completes with an error inside its 1 s timeout; LIMIT_REACH / FINISHED keep the unread results.
On a local single-BE cluster, branch-4.x's regression test_flight_cancel_cleanup (it aborts one endpoint after the first batch, by closing it and by an explicit cancel, and requires the query's pipelines to disappear within 20 s) fails on master with Cancelled Flight query retained pipelines: [...: 1 pipeline fragment contexts in query ...] and passes with this PR. The suite comes with the follow-up, because on a multi-BE cluster it also needs the FE route. The 20 arrow_flight_sql_p0 suites and the 14 external_table_p0/remote_doris suites, which read results over Arrow Flight, pass before and after.
The 12 new tests plus ResultBufferMgrTest (6) and ArrowResultBlockBufferTest (4) pass 10 times in a row in one ASAN process. Ten single-point mutations of the change (dropping the buffer cleanup in cancel_plan_fragment, the on_failure call, the local cancel in close(), the cancellation check in the local read loop or in the BRPC wait, the sibling loop, the EOF flag, the QueryContext cancel, the cancel in the destructor, the eos = false on an empty wait) each turn their target tests red.

Release note

Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on the BE that holds the stream's results and frees its unread results there right away, instead of keeping them for minutes. A stream aborted this way cannot be read again with the same ticket.

Check List (For Author)

Test: Unit Test / Manual test
- BE UT: ArrowFlightCancelTest (9, new), ArrowFlightRemoteCancelTest (3, new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4): pass 10 times in a row (ASAN, macOS); 10 mutations each caught
- build-support/check-build-hygiene.sh passes; clang-format 16 clean; clang-tidy reports nothing on the changed lines except readability-inconsistent-ifelse-braces inside the DORIS_CHECK macro expansion
- Manual test on a local single-BE cluster: branch-4.x's test_flight_cancel_cleanup fails before and passes after; arrow_flight_sql_p0 (20 suites) and external_table_p0/remote_doris (14 suites) pass
- No regression test in this PR: test_flight_cancel_cleanup needs the FE route of the follow-up to pass on a multi-BE cluster, so it comes with it
Behavior changed: Yes. A Flight result stream served by the BE that holds its results and cancelled, closed or destroyed before EOF now cancels its query on that BE and frees the query's Flight buffers there; a later DoGet with the same ticket fails instead of resuming. On a stream served by another BE, the abort only interrupts the pending fetch.
Does this need documentation: No

Original author: @Gabriel39

…stream closes early ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#67644 Problem Summary: Port the BE-local half of branch-4.x apache#68630 to master. An Arrow Flight DoGet stream that the client cancels or closes before EOF left its query running. The local reader waited on an empty result buffer without looking at the RPC's cancellation state, and closing the stream did not cancel anything: the abandoned buffer kept its unread blocks until ResultBufferMgr dropped it, at execution_timeout + 5 s or result_buffer_cancelled_interval_time (300 s) after its fragments finished, and when the result did not fit it back-pressured the result sink until the query timed out. A remote reader blocked in a BRPC join that nothing could interrupt, an aborting cancel_plan_fragment left the query's unread Flight buffers in place until that delayed cleanup, and fetch_arrow_data for a buffer that was already gone never ran its done closure, so the caller waited for the BRPC timeout. The reader now checks ServerCallContext::is_cancelled between bounded 20 ms buffer waits and while a BRPC fetch is pending (cancelling it with brpc::StartCancel), and closes itself on Close(), on a read error and in its destructor. Closing a local reader before EOF cancels the query on this BE: ResultBufferMgr keeps a query -> Flight buffers index, removes every Flight buffer of the query and cancels its QueryContext outside the buffer-map lock, even after that context has expired. cancel_plan_fragment with an abort reason (not LIMIT_REACH or FINISHED) also drops the query's Flight buffers, and fetch_arrow_data completes the RPC with an error when the buffer is gone (the same line as apache#67644). The FE route that reaches the query's other BEs (cancelFlightQuery and the BE broadcast) is left to a follow-up PR. Differences from branch-4.x: the two index/map consistency checks are DORIS_CHECK, and the ArrowFlightResultBlockBuffer constructor moved to the .cpp so that varrow_flight_result_writer.h, which 13 TUs include, does not include runtime/query_context.h; only the .cpp needs its definition. ### Release note Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on the BE that holds the stream's results and releases its unread results right away, instead of keeping them for minutes. ### Check List (For Author) - Test: Unit Test / Manual test - BE UT: ArrowFlightCancelTest (9, new), ArrowFlightRemoteCancelTest (3, new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4) - Manual test on a local single-BE cluster: branch-4.x's test_flight_cancel_cleanup fails before and passes after; arrow_flight_sql_p0 and external_table_p0/remote_doris pass - Behavior changed: Yes. A Flight result stream closed before EOF on the BE that holds its results cancels its query on that BE, and a later DoGet with the same ticket fails instead of resuming. - Does this need documentation: No Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hello-stephen · 2026-11-02T03:09:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-11-02T03:10:18Z

run buildall

morningman · 2026-11-02T03:10:20Z

/review

github-actions

Static review of the full PR diff found one verified issue.

The new Arrow Flight reader state machine preserves the important local behaviors: bounded empty-buffer waits remain distinct from EOS, client cancellation interrupts local waits and outbound BRPC joins, and Close/destruction is idempotent. ResultBufferMgr's query-to-buffer index is updated under the buffer-map lock, sibling buffers are removed before cancellation runs outside that lock, and cancel_plan_fragment preserves unread results for LIMIT_REACH and FINISHED while clearing aborts. Missing-buffer fetches now complete their RPC with an error. I found no additional production issue in those paths.

[P2] The new ArrowFlightCancelTest fixture leaves a global result_buffer_block_count metric hook capturing a destroyed ResultBufferMgr after TearDown. Later metric collection can dereference freed storage, and later fixtures cannot replace the hook because registration uses emplace. This needs scoped hook cleanup or a fixture design that does not leave an unregistered manager behind.

The user supplied no additional focus points. The PR explicitly documents the FE query-wide cancellation route as a follow-up; remote DoGet cancellation currently interrupts the proxy's BRPC fetch but does not cancel result BEs reached through that route, so that remains an acknowledged scope limit.

Review convergence: false. This delivery is at the three-round cap. The required delegated subagent results were unavailable because each invocation hit a model-capacity error, so the formal all-subagents convergence condition was not met; the verified finding is still delivered. Static review only; no builds or tests were executed by this review.

Review did not converge after 3 rounds. This automated review cannot pass the review check.

Convergence was not reached at the three-round cap: all three assigned subagent invocations failed with model-capacity errors, so the required latest-round NO_NEW_VALUABLE_FINDINGS results are unavailable. The cross-BE FE cancellation route is also explicitly outside this BE-only PR and remains a documented follow-up scope limit.

github-actions · 2026-11-02T04:19:26Z

+        return _mgr.find_buffer(_id, found).ok();
+    }
+    MockRuntimeState _state;
+    ResultBufferMgr _mgr;


[P2] Unregister the result-buffer metric hook before this fixture's manager is destroyed. ResultBufferMgr registers a global result_buffer_block_count callback that captures this, while TearDown() restores only ExecEnv::_result_mgr; the _mgr member is then destroyed, leaving metric scrapes with a dangling callback. Subsequent fixtures also cannot replace the hook because registration uses emplace. Scope cleanup to the hook owned by this fixture (preserving any prior hook) or avoid constructing an unregistered manager per test.

Fixed in e5736f5. ArrowFlightCancelTest::TearDown() now calls _mgr.stop() after restoring the previous ExecEnv::_result_mgr, so the result_buffer_block_count hook and its gauge are deregistered before the fixture destroys the manager, the same way LoadChannelMgrTest::TearDown() stops its LoadChannelMgr. The fixture never calls init(), so stop() has no cleaner thread to join. Each fixture now registers its own hook and removes it again, so a later fixture no longer finds a stale one; ArrowFlightRemoteCancelTest gets the same cleanup through ArrowFlightCancelTest::TearDown().

On the impact: nothing in doris_be_test runs the server entity's hooks today. enable_metric_calculator defaults to true, so to_prometheus() and to_json() skip the hooks, and the daemon thread that calls trigger_all_hooks(true) is not started in UT. So this was latent, not an observed use-after-free. ResultBufferMgrTest on master builds six managers on the stack in the same way and never stops them; that file is outside this PR.

hello-stephen · 2026-11-02T04:20:46Z

TPC-H: Total hot run time: 28498 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0e2959a42425883e6370fba76408e6f2f8d1b4e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17612	5192	5084	5084
q2	2173	298	243	243
q3	10252	1500	842	842
q4	4675	452	338	338
q5	7442	817	530	530
q6	188	187	154	154
q7	755	793	606	606
q8	9251	1378	1085	1085
q9	5616	4189	4187	4187
q10	6821	1296	1052	1052
q11	441	254	230	230
q12	644	392	276	276
q13	18141	3663	3072	3072
q14	279	273	245	245
q15	q16	722	709	655	655
q17	1213	762	577	577
q18	7028	6198	6076	6076
q19	1102	972	603	603
q20	392	342	227	227
q21	4954	2154	2317	2154
q22	350	299	262	262
Total cold run time: 100051 ms
Total hot run time: 28498 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5610	5602	5630	5602
q2	680	609	528	528
q3	5236	5351	4916	4916
q4	2065	2136	1684	1684
q5	5253	5127	5734	5127
q6	261	192	150	150
q7	2378	2123	1855	1855
q8	3068	2962	2754	2754
q9	8032	8136	8034	8034
q10	3736	3707	3253	3253
q11	619	453	422	422
q12	670	689	528	528
q13	3488	4020	3328	3328
q14	319	311	292	292
q15	q16	701	740	636	636
q17	8464	7590	7182	7182
q18	12933	12221	12999	12221
q19	912	819	823	819
q20	2120	2136	1930	1930
q21	5643	4631	4655	4631
q22	516	458	425	425
Total cold run time: 72704 ms
Total hot run time: 66317 ms

hello-stephen · 2026-11-02T04:30:36Z

TPC-DS: Total hot run time: 151416 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0e2959a42425883e6370fba76408e6f2f8d1b4e, data reload: false

query5	4303	569	446	446
query6	430	208	185	185
query7	4793	456	230	230
query8	317	164	147	147
query9	8775	3161	3202	3161
query10	483	320	252	252
query11	5975	3876	3563	3563
query12	145	101	89	89
query13	1255	482	335	335
query14	6097	3955	3684	3684
query14_1	3410	3453	3417	3417
query15	217	205	186	186
query16	949	415	420	415
query17	872	681	542	542
query18	2415	425	314	314
query19	207	162	131	131
query20	85	85	84	84
query21	212	132	108	108
query22	12916	13170	12782	12782
query23	13502	12920	12399	12399
query23_1	12611	12500	12620	12500
query24	7362	820	494	494
query24_1	496	478	484	478
query25	531	430	360	360
query26	1268	261	139	139
query27	2761	438	282	282
query28	4628	2050	2055	2050
query29	1570	603	459	459
query30	314	228	187	187
query31	1020	939	814	814
query32	146	101	91	91
query33	530	305	248	248
query34	899	870	514	514
query35	744	756	636	636
query36	815	834	736	736
query37	128	100	85	85
query38	1791	1738	1646	1646
query39	800	793	745	745
query39_1	741	733	757	733
query40	221	120	99	99
query41	70	73	66	66
query42	105	86	84	84
query43	346	365	312	312
query44	1309	713	711	711
query45	183	181	165	165
query46	846	968	578	578
query47	2928	2950	2890	2890
query48	296	306	212	212
query49	573	389	282	282
query50	675	280	211	211
query51	10456	10577	10601	10577
query52	80	79	74	74
query53	184	208	151	151
query54	231	206	185	185
query55	76	66	60	60
query56	278	205	214	205
query57	1684	1625	1546	1546
query58	280	265	257	257
query59	2154	2173	1967	1967
query60	291	246	210	210
query61	157	165	146	146
query62	384	336	288	288
query63	187	156	160	156
query64	2744	963	768	768
query65	3341	3243	3243	3243
query66	1778	423	295	295
query67	20496	20166	20199	20166
query68	3308	1003	609	609
query69	392	288	255	255
query70	825	766	753	753
query71	305	237	200	200
query72	2878	2641	2315	2315
query73	535	525	296	296
query74	4651	4513	4338	4338
query75	2352	2285	1915	1915
query76	2416	1013	582	582
query77	349	376	275	275
query78	9744	9700	9164	9164
query79	990	859	547	547
query80	1208	458	343	343
query81	586	332	282	282
query82	590	130	109	109
query83	299	191	176	176
query84	340	119	101	101
query85	860	435	373	373
query86	394	241	231	231
query87	1999	1921	1799	1799
query88	3749	2750	2726	2726
query89	327	282	249	249
query90	1917	159	165	159
query91	156	150	123	123
query92	97	87	90	87
query93	1028	994	582	582
query94	648	301	271	271
query95	560	374	301	301
query96	684	482	238	238
query97	2454	2416	2322	2322
query98	165	151	150	150
query99	784	762	659	659
Total cold run time: 236397 ms
Total hot run time: 151416 ms

hello-stephen · 2026-11-02T04:35:26Z

ClickBench: Total hot run time: 24.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0e2959a42425883e6370fba76408e6f2f8d1b4e, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.09	0.08
query4	1.60	0.09	0.10
query5	0.23	0.25	0.24
query6	1.17	0.69	0.68
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.34	0.36
query10	0.57	0.57	0.57
query11	0.21	0.15	0.14
query12	0.18	0.14	0.15
query13	0.50	0.51	0.49
query14	1.02	1.01	1.02
query15	0.63	0.62	0.61
query16	0.30	0.35	0.35
query17	1.11	1.16	1.13
query18	0.19	0.20	0.19
query19	2.07	1.91	1.95
query20	0.02	0.02	0.01
query21	15.49	0.18	0.16
query22	4.96	0.05	0.05
query23	16.28	0.32	0.12
query24	3.15	0.59	0.28
query25	0.12	0.05	0.05
query26	0.82	0.22	0.16
query27	0.05	0.04	0.05
query28	3.07	0.75	0.37
query29	12.49	4.16	3.28
query30	0.28	0.13	0.09
query31	2.78	0.55	0.33
query32	3.22	0.59	0.51
query33	3.34	3.17	3.16
query34	15.37	3.98	3.37
query35	3.34	3.29	3.30
query36	0.56	0.45	0.43
query37	0.09	0.06	0.07
query38	0.05	0.03	0.03
query39	0.03	0.03	0.03
query40	0.17	0.15	0.15
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.48 s
Total hot run time: 24.11 s

hello-stephen · 2026-11-02T06:59:26Z

BE UT Coverage Report

Increment line coverage 91.40% (85/93) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	67.27% (32076/47682)
Line Coverage	52.11% (337382/647397)
Region Coverage	47.60% (273981/575550)
Branch Coverage	49.43% (129127/261255)

hello-stephen · 2026-11-02T07:07:57Z

BE Regression && UT Coverage Report

Increment line coverage 91.40% (85/93) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.51% (35795/46184)
Line Coverage	62.82% (402198/640191)
Region Coverage	59.41% (340299/572812)
Branch Coverage	60.51% (157500/260268)

…ght cancel test fixture ends ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: `ResultBufferMgr`'s constructor registers the process-wide `result_buffer_block_count` hook, and the hook captures the manager. Only `ResultBufferMgr::stop()` deregisters it. `ArrowFlightCancelTest` builds a manager per test and never stopped it, so after each test the global metric entity kept a hook that pointed at a destroyed manager. A later metric collection with hooks enabled would read freed memory. Because `MetricEntity::register_hook` uses `emplace`, a later fixture could not replace a stale hook either. `TearDown()` now calls `_mgr.stop()` after restoring the previous manager, the same way `LoadChannelMgrTest` stops its `LoadChannelMgr`. The manager is never `init()`ed here, so `stop()` only removes the hook and its gauge; no cleaner thread is joined. ### Release note None ### Check List (For Author) - Test: Unit Test - `ArrowFlightCancelTest.*`, `ArrowFlightRemoteCancelTest.*`, `ResultBufferMgrTest.*` - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

morningman · 2026-11-02T12:55:07Z

Addressed review round 1 in e5736f55c0f.

4438874982: ArrowFlightCancelTest::TearDown() now stops the fixture's ResultBufferMgr, which deregisters the process-wide result_buffer_block_count hook that the manager registered, before the fixture destroys it (as LoadChannelMgrTest does for LoadChannelMgr). Test-only. Nothing in UT ran that hook, so the dangling hook was latent.

ArrowFlightCancelTest.*, ArrowFlightRemoteCancelTest.*, ResultBufferMgrTest.* and ArrowResultBlockBufferTest.* pass locally 10 times in a row in one ASAN process (run with the FE-route follow-up stacked on this commit).

morningman · 2026-11-02T12:55:11Z

run buildall

morningman · 2026-11-02T12:55:16Z

/review

hello-stephen · 2026-11-02T13:40:23Z

TPC-H: Total hot run time: 28697 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5736f55c0f9db012ea956f0dfb66a841bd6da30, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17575	5110	5016	5016
q2	2091	314	261	261
q3	10248	1376	828	828
q4	4682	460	334	334
q5	7449	804	538	538
q6	181	181	149	149
q7	760	809	615	615
q8	9249	1326	1070	1070
q9	5550	4216	4161	4161
q10	6819	1276	1038	1038
q11	427	252	223	223
q12	628	396	269	269
q13	18144	3557	3025	3025
q14	277	273	254	254
q15	q16	714	721	677	677
q17	1196	745	566	566
q18	6955	6159	6108	6108
q19	1113	956	589	589
q20	376	323	229	229
q21	4919	2551	2464	2464
q22	378	332	283	283
Total cold run time: 99731 ms
Total hot run time: 28697 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5910	5892	5881	5881
q2	731	616	587	587
q3	5596	5767	5219	5219
q4	2208	2142	1559	1559
q5	5748	5742	5732	5732
q6	236	184	136	136
q7	2255	1989	1812	1812
q8	2870	2471	2470	2470
q9	7560	7469	7507	7469
q10	3629	3567	3157	3157
q11	573	415	394	394
q12	658	659	502	502
q13	3233	3626	3030	3030
q14	281	292	273	273
q15	q16	655	687	605	605
q17	7749	7018	6831	6831
q18	12978	12187	12765	12187
q19	897	760	766	760
q20	2144	2178	1926	1926
q21	5508	4521	4692	4521
q22	507	481	412	412
Total cold run time: 71926 ms
Total hot run time: 65463 ms

hello-stephen · 2026-11-02T13:50:11Z

TPC-DS: Total hot run time: 151515 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5736f55c0f9db012ea956f0dfb66a841bd6da30, data reload: false

query5	4294	587	445	445
query6	424	212	198	198
query7	4801	452	232	232
query8	320	170	147	147
query9	8767	3158	3170	3158
query10	473	308	255	255
query11	5944	3876	3570	3570
query12	144	91	86	86
query13	1263	444	338	338
query14	6134	3869	3669	3669
query14_1	3459	3418	3436	3418
query15	210	205	188	188
query16	942	412	414	412
query17	869	659	531	531
query18	2433	422	312	312
query19	204	166	135	135
query20	85	83	83	83
query21	208	125	109	109
query22	12956	13045	12816	12816
query23	13409	12763	12504	12504
query23_1	12511	12479	12538	12479
query24	7556	833	479	479
query24_1	485	487	531	487
query25	509	395	343	343
query26	1262	254	141	141
query27	2805	436	288	288
query28	4689	2050	2022	2022
query29	1558	571	440	440
query30	312	215	185	185
query31	1007	926	818	818
query32	131	90	85	85
query33	517	293	228	228
query34	899	836	503	503
query35	737	745	646	646
query36	837	801	725	725
query37	125	95	84	84
query38	1869	1739	1691	1691
query39	789	792	734	734
query39_1	740	730	742	730
query40	217	114	93	93
query41	65	64	64	64
query42	85	86	83	83
query43	347	357	313	313
query44	1317	696	695	695
query45	180	179	163	163
query46	829	965	554	554
query47	2729	2830	2828	2828
query48	302	298	235	235
query49	573	407	281	281
query50	677	283	214	214
query51	10622	10662	10520	10520
query52	79	83	70	70
query53	186	211	158	158
query54	237	197	189	189
query55	76	70	63	63
query56	236	229	210	210
query57	1661	1615	1549	1549
query58	286	264	263	263
query59	2144	2231	1945	1945
query60	288	240	230	230
query61	148	152	155	152
query62	386	337	284	284
query63	184	154	161	154
query64	2802	1085	839	839
query65	3303	3259	3291	3259
query66	1772	417	329	329
query67	20105	20000	19999	19999
query68	3011	1025	615	615
query69	392	311	263	263
query70	853	805	795	795
query71	296	233	220	220
query72	2714	2727	2270	2270
query73	508	519	304	304
query74	4641	4526	4359	4359
query75	2315	2306	1900	1900
query76	2326	1016	572	572
query77	346	383	269	269
query78	9682	9675	9141	9141
query79	1024	850	548	548
query80	525	416	335	335
query81	524	314	283	283
query82	577	133	108	108
query83	301	207	185	185
query84	316	117	106	106
query85	805	447	404	404
query86	310	238	236	236
query87	1965	1950	1809	1809
query88	3779	2785	2759	2759
query89	332	278	246	246
query90	1896	170	168	168
query91	159	152	131	131
query92	96	88	92	88
query93	979	991	570	570
query94	455	307	280	280
query95	580	337	311	311
query96	686	486	241	241
query97	2450	2405	2325	2325
query98	159	157	143	143
query99	771	770	645	645
Total cold run time: 234157 ms
Total hot run time: 151515 ms

hello-stephen · 2026-11-02T13:55:02Z

ClickBench: Total hot run time: 24.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5736f55c0f9db012ea956f0dfb66a841bd6da30, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.27	0.09	0.10
query4	1.61	0.10	0.11
query5	0.23	0.25	0.24
query6	1.16	0.69	0.67
query7	0.04	0.01	0.00
query8	0.05	0.04	0.03
query9	0.40	0.36	0.36
query10	0.57	0.61	0.60
query11	0.20	0.15	0.15
query12	0.18	0.15	0.15
query13	0.50	0.50	0.50
query14	1.02	1.01	1.01
query15	0.62	0.61	0.60
query16	0.36	0.36	0.36
query17	1.18	1.15	1.14
query18	0.23	0.20	0.20
query19	2.00	1.90	1.97
query20	0.02	0.01	0.01
query21	15.50	0.19	0.13
query22	4.92	0.06	0.05
query23	16.27	0.30	0.13
query24	2.93	0.50	0.39
query25	0.10	0.05	0.05
query26	1.06	0.22	0.17
query27	0.04	0.05	0.03
query28	3.29	0.74	0.39
query29	12.51	4.17	3.34
query30	0.28	0.14	0.10
query31	2.77	0.56	0.32
query32	3.23	0.60	0.50
query33	3.30	3.25	3.19
query34	15.39	3.96	3.37
query35	3.31	3.35	3.34
query36	0.55	0.45	0.46
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.72 s
Total hot run time: 24.48 s

github-actions · 2026-11-02T13:56:28Z

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41448611756

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

…s next request ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: The FE dropped a Flight query's cancellation route when the session's next request began and when the session was torn down (FlightSqlChannel.reset). The client reads the results from the BEs, so it can still hold a stream of the query open at that point, for example through a second statement on the same connection, and close that stream early afterwards. The BE's cancelFlightQuery then found no route, and the query's other BEs kept their pipelines and result buffers until the query timeout. branch-4.x (apache#68630) drops the route at the same points. A route now belongs to no session. It goes when its cancellation succeeds, when its query's execution timeout plus 5 s has passed, and, oldest first and as a whole, when the registry would hold more than 100,000 result ids, a few hundred bytes each. Without the session as a bound, that cap is what keeps the registry from growing with the query rate times the query timeout; a query whose route was pushed out ends at its timeout, as every Flight query did before this PR. The registry is an insertion-ordered map under a lock, with the RPCs sent outside it, instead of a Caffeine cache, whose size eviction may reject a new entry instead of evicting the oldest one. StmtExecutor registers a query through FlightSqlQueryCancellation.register(ConnectContext, Coordinator), and FlightSqlChannel is back to its master form. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5): a route survives the session's next request and its teardown and still cancels; the bound drops whole routes, oldest first; an expired route leaves at the next registration. With the 16 other Flight session and coordinator test classes. - Regression: test_flight_cancel_cleanup gains a mode that runs the session's next request before it aborts the stream. - Behavior changed: Yes. A stream aborted after its session moved on to another request, or was closed, still cancels its query on every BE. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hello-stephen · 2026-11-02T16:11:08Z

BE Regression && UT Coverage Report

Increment line coverage 91.40% (85/93) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.74% (35922/46206)
Line Coverage	63.09% (404059/640492)
Region Coverage	59.75% (342355/572989)
Branch Coverage	60.82% (158356/260360)

morningman · 2026-11-03T07:28:23Z

/review

github-actions · 2026-11-03T08:09:33Z

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41596538268

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

morningman · 2026-11-03T09:11:37Z

/review

github-actions · 2026-11-03T09:48:57Z

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41609026680

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

morningman · 2026-11-03T10:16:14Z

/review

github-actions

Static review of the full PR diff found no new actionable issue at the current head.

The cancellation state machine preserves bounded empty-buffer waits as distinct from EOS, polls client cancellation while reading local results, and makes Close/destruction idempotent. Local cancellation wakes the result-buffer wait and clears the query's Arrow Flight siblings; map/index updates are protected by the buffer-map lock, while query cancellation runs after the lock is released. The BRPC callback releases its mutex before StartCancel and join. cancel_plan_fragment preserves unread results for LIMIT_REACH and FINISHED and clears buffers for abort statuses, including both legacy and serialized status forms. Missing-buffer Arrow fetches now complete the RPC with a serialized error.

The existing P2 thread 4438874982 was revalidated as fixed by the fixture's _mgr.stop() teardown; it is not reposted. The PR's remote cross-BE cancellation limitation is explicitly documented as a follow-up, so it is not reported as a new finding. No additional user focus was supplied.

This was a static review only; no builds or tests were run. Review convergence: true after two bounded rounds, with all risk scopes and replacement/final-sweep scopes dispositioned and no accepted inline findings.

Review converged after 2 round(s).

github-actions · 2026-11-04T02:20:44Z

PR approved by anyone and no changes requested.

github-actions · 2026-11-04T02:20:46Z

PR approved by at least one committer and no changes requested.

…t stream closes early ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: Port the FE route of branch-4.x apache#68630 to master, on top of apache#69945 (its BE-local half). With apache#69945 alone, a DoGet stream closed before EOF cancels its query only on the BE that holds the stream's results. A query whose results are spread over several BEs (parallel result sink, one endpoint per BE) kept running on the other BEs: master closes the coordinator when GetFlightInfo returns (apache#67504), so nothing sends them a cancel, and a single-fragment query has no exchange that would carry the cancellation between BEs. Their pipelines ran until the query finished or timed out. A remote reader (a DoGet served by a BE other than the result BE) could not cancel the result BE's query at all. The FE now keeps a route for each Flight query that leaves its results on the BEs: from each result id (the fragment instance id of an endpoint) to the query id and the BRPC addresses of all its BEs (FlightSqlQueryCancellation, registered by StmtExecutor before the coordinator is closed or deferred). The route belongs to the session's FlightSqlChannel and is dropped when the session's next request begins (FlightProtocolAdapter.beginRequest resets the channel), when the session is torn down (releaseSession closes the channel), when a cancellation succeeds, and at the latest execution_timeout + 5 s after registration. The new FrontendService RPC cancelFlightQuery(result_id) sends the existing cancel_plan_fragment, which older BEs understand and which does not use the Flight read pool, to every BE of the query and waits at most 1 s for all of them; when any BE fails, the route stays so that the caller can retry. A reader closed before EOF now asks the running FEs in turn: an FE without the route answers NOT_FOUND and the BE moves on to the next one, a failed attempt is retried once, and the transport is reopened after a thrift error. This runs on the DoGet thread, so with an unreachable FE one early close can take a few seconds (two attempts with a 2 s RPC timeout per FE). Compatibility: cancelFlightQuery has the same name and signature as on branch-4.1 and branch-4.2. An FE without it fails the call; the BE logs a warning for that FE and only the local cleanup of apache#69945 takes effect. A result BE without apache#69945 cancels the query but keeps its result buffer until its usual timeout. Differences from branch-4.x: the route is registered in master's `!isReturnResultFromLocal()` branch (apache#67900), before the `mustOutliveDispatch()` deferral, so deferred queries are routed as well; FlightProtocolAdapter needs no change because the route stays in FlightSqlChannel; the BE reader includes runtime/frontend_info.h itself (exec_env.h no longer includes it, apache#66400); the regression suite keeps its table at the end (AGENTS.md). ### Release note Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on every BE that runs it, not only on the BE that serves the stream. The FE and the BEs must both be upgraded. ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5, new), StmtExecutorTest and 15 other Flight session / coordinator test classes - BE UT: ArrowFlightCancelTest (9), ArrowFlightRemoteCancelTest (9, 6 new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4) - Regression: arrow_flight_sql_p0/test_flight_cancel_cleanup (new) and the rest of arrow_flight_sql_p0 on a local single-BE cluster; the cross-BE path is covered by the BE UT's in-process FE and remote BE - Behavior changed: Yes. A Flight result stream closed before EOF cancels its query on every BE of the query. - Does this need documentation: No Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s next request ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: The FE dropped a Flight query's cancellation route when the session's next request began and when the session was torn down (FlightSqlChannel.reset). The client reads the results from the BEs, so it can still hold a stream of the query open at that point, for example through a second statement on the same connection, and close that stream early afterwards. The BE's cancelFlightQuery then found no route, and the query's other BEs kept their pipelines and result buffers until the query timeout. branch-4.x (apache#68630) drops the route at the same points. A route now belongs to no session. It goes when its cancellation succeeds, when its query's execution timeout plus 5 s has passed, and, oldest first and as a whole, when the registry would hold more than 100,000 result ids, a few hundred bytes each. Without the session as a bound, that cap is what keeps the registry from growing with the query rate times the query timeout; a query whose route was pushed out ends at its timeout, as every Flight query did before this PR. The registry is an insertion-ordered map under a lock, with the RPCs sent outside it, instead of a Caffeine cache, whose size eviction may reject a new entry instead of evicting the oldest one. StmtExecutor registers a query through FlightSqlQueryCancellation.register(ConnectContext, Coordinator), and FlightSqlChannel is back to its master form. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5): a route survives the session's next request and its teardown and still cancels; the bound drops whole routes, oldest first; an expired route leaves at the next registration. With the 16 other Flight session and coordinator test classes. - Regression: test_flight_cancel_cleanup gains a mode that runs the session's next request before it aborts the stream. - Behavior changed: Yes. A stream aborted after its session moved on to another request, or was closed, still cancels its query on every BE. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t stream closes early ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: Port the FE route of branch-4.x apache#68630 to master, on top of apache#69945 (its BE-local half). With apache#69945 alone, a DoGet stream closed before EOF cancels its query only on the BE that holds the stream's results. A query whose results are spread over several BEs (parallel result sink, one endpoint per BE) kept running on the other BEs: master closes the coordinator when GetFlightInfo returns (apache#67504), so nothing sends them a cancel, and a single-fragment query has no exchange that would carry the cancellation between BEs. Their pipelines ran until the query finished or timed out. A remote reader (a DoGet served by a BE other than the result BE) could not cancel the result BE's query at all. The FE now keeps a route for each Flight query that leaves its results on the BEs: from each result id (the fragment instance id of an endpoint) to the query id and the BRPC addresses of all its BEs (FlightSqlQueryCancellation, registered by StmtExecutor before the coordinator is closed or deferred). The route belongs to the session's FlightSqlChannel and is dropped when the session's next request begins (FlightProtocolAdapter.beginRequest resets the channel), when the session is torn down (releaseSession closes the channel), when a cancellation succeeds, and at the latest execution_timeout + 5 s after registration. The new FrontendService RPC cancelFlightQuery(result_id) sends the existing cancel_plan_fragment, which older BEs understand and which does not use the Flight read pool, to every BE of the query and waits at most 1 s for all of them; when any BE fails, the route stays so that the caller can retry. A reader closed before EOF now asks the running FEs in turn: an FE without the route answers NOT_FOUND and the BE moves on to the next one, a failed attempt is retried once, and the transport is reopened after a thrift error. This runs on the DoGet thread, so with an unreachable FE one early close can take a few seconds (two attempts with a 2 s RPC timeout per FE). Compatibility: cancelFlightQuery has the same name and signature as on branch-4.1 and branch-4.2. An FE without it fails the call; the BE logs a warning for that FE and only the local cleanup of apache#69945 takes effect. A result BE without apache#69945 cancels the query but keeps its result buffer until its usual timeout. Differences from branch-4.x: the route is registered in master's `!isReturnResultFromLocal()` branch (apache#67900), before the `mustOutliveDispatch()` deferral, so deferred queries are routed as well; FlightProtocolAdapter needs no change because the route stays in FlightSqlChannel; the BE reader includes runtime/frontend_info.h itself (exec_env.h no longer includes it, apache#66400); the regression suite keeps its table at the end (AGENTS.md). ### Release note Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on every BE that runs it, not only on the BE that serves the stream. The FE and the BEs must both be upgraded. ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5, new), StmtExecutorTest and 15 other Flight session / coordinator test classes - BE UT: ArrowFlightCancelTest (9), ArrowFlightRemoteCancelTest (9, 6 new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4) - Regression: arrow_flight_sql_p0/test_flight_cancel_cleanup (new) and the rest of arrow_flight_sql_p0 on a local single-BE cluster; the cross-BE path is covered by the BE UT's in-process FE and remote BE - Behavior changed: Yes. A Flight result stream closed before EOF cancels its query on every BE of the query. - Does this need documentation: No Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s next request ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: The FE dropped a Flight query's cancellation route when the session's next request began and when the session was torn down (FlightSqlChannel.reset). The client reads the results from the BEs, so it can still hold a stream of the query open at that point, for example through a second statement on the same connection, and close that stream early afterwards. The BE's cancelFlightQuery then found no route, and the query's other BEs kept their pipelines and result buffers until the query timeout. branch-4.x (apache#68630) drops the route at the same points. A route now belongs to no session. It goes when its cancellation succeeds, when its query's execution timeout plus 5 s has passed, and, oldest first and as a whole, when the registry would hold more than 100,000 result ids, a few hundred bytes each. Without the session as a bound, that cap is what keeps the registry from growing with the query rate times the query timeout; a query whose route was pushed out ends at its timeout, as every Flight query did before this PR. The registry is an insertion-ordered map under a lock, with the RPCs sent outside it, instead of a Caffeine cache, whose size eviction may reject a new entry instead of evicting the oldest one. StmtExecutor registers a query through FlightSqlQueryCancellation.register(ConnectContext, Coordinator), and FlightSqlChannel is back to its master form. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5): a route survives the session's next request and its teardown and still cancels; the bound drops whole routes, oldest first; an expired route leaves at the next registration. With the 16 other Flight session and coordinator test classes. - Regression: test_flight_cancel_cleanup gains a mode that runs the session's next request before it aborts the stream. - Behavior changed: Yes. A stream aborted after its session moved on to another request, or was closed, still cancels its query on every BE. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rdown ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A Flight result stream closed before EOF asks the running FEs in turn to cancel its query (`cancelFlightQuery`), on the DoGet thread, from `Close()` and from the reader's destructor. The 2 s timeout it gave applied only to a connection it opened itself: `ClientCacheHelper::get_client` returns a cached transport with the timeout of whichever caller opened it, such as 60 s for profile and status reports, or the query's execution timeout (900 s by default) for the information_schema scanners. Acquiring a connection also backed off 1 + 2 + 4 s after failures, and each FE got two attempts. An FE that sorts before the one that planned the query is asked first on every BE, and a stopped FE stays in that list as long as it is not dropped (the master sends it with process UUID 0, which `get_running_frontends()` counts as running). So every early close waited about 14.6 s for such an FE when it refused connections, 32.6 s when it did not answer the connect, and up to minutes when a long-timeout transport to it sat in the cache. The other way round, this call's 2 s transports, and the 1 s ones it reopened after any thrift error (an older FE's unknown-method reply included), went back to the shared cache, where `fetchSplitBatch`, whose FE side holds the call for up to 1 s, or a status report could take them. The cancellation now keeps FE connections of its own (`ExecEnv::flight_cancel_frontend_client_cache`, at most two kept per FE, as each idle one holds an FE thrift thread), so every transport it uses was opened with its 2 s timeout and none reaches another caller. Acquiring a connection does not back off (`max_retries = 0`), a transport is reopened with the same 2 s, and an FE that cannot be connected to gets no second attempt, since neither reason for the retry (the FE's own deadline, a stale cached transport) applies to it. An FE ahead of the planning one now costs an early close about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect (`thrift_connect_timeout_seconds`), and 4 s when it accepts the connection and does not answer. ### Release note None ### Check List (For Author) - Test: Unit Test - `ArrowFlightRemoteCancelTest#CloseIgnoresLongTimeoutSharedTransport` leaves a 60 s transport to an FE whose first answer stalls 5 s in the shared cache and requires the close to finish within 4 s, through the retry; `#CloseMovesPastUnreachableFrontend` puts an FE that refuses connections first and requires the close to reach the planning FE within 2 s. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… registry bound ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A route of `FlightSqlQueryCancellation` keeps the BRPC addresses of all the BEs of its query, and the result ids of a query share one route, but the registry bound counted only the result ids. In parallel result mode, the default, a query has one result id, so a full registry of 100,000 routes held about 0.3 GB with 100 BEs per query and 2.9 GB with 1,000. Routes no longer end with their session, and a stream read to EOF does not drop its route, so a registry fills up whenever queries arrive faster than 100,000 per execution timeout (about 111 per second with the default 900 s). The registry now also bounds the addresses its routes hold, at 1,000,000 (about 30 MB), and drops whole routes, oldest first, while either bound is exceeded. A route's addresses leave with its last result id, also when a newer registration takes an id over, and are released only once when the route is pushed out while a cancellation through it is in flight. The registry now stays within about 55 MB whatever the cluster size. Large clusters give up retention instead: with 1,000 BEs per query it keeps the routes of the last 1,000 queries, and a query whose route was pushed out ends at its timeout, as before routes existed. A route must have a result id, which every Flight query that leaves its results on the BEs has. ### Release note None ### Check List (For Author) - Test: Unit Test - `FlightSqlQueryCancellationTest#boundCountsRetainedBackends` (addresses past the bound push out the oldest route as a whole) and `#reregisteredResultReleasesOldRouteBackends` (a route whose result id a newer registration took over releases its addresses once). The coordinator mock of `StmtExecutorTest#testCloudReplanDispatchProfileLifecycle` now adds an endpoint in `exec()`, as the coordinator does, since a route needs a result id. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… fails ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A Flight query whose results stay on the BEs registers its cancellation route in `StmtExecutor`, before GetFlightInfo fetches the Arrow schema from the result BE and builds the tickets. When the request fails after that, at the schema fetch or because a prepared statement's result schema no longer matches, no ticket names the query's results, so no BE can report an early close of them, yet the route stayed in `FlightSqlQueryCancellation` until its query's timeout plus 5 s, or until newer routes pushed it out. While routes belonged to the session, its next request dropped them; since routes outlive the session, nothing did. A GetFlightInfo that fails after it began now drops the routes of its endpoints, each as a whole, and so does the prepared-statement schema check before it resets the session. A request that fails before it begins, such as an empty query, finds the previous request's endpoints on the session and leaves their routes: a client may still be pulling those results. ### Release note None ### Check List (For Author) - Test: Unit Test - `DorisFlightSqlProducerTest#testGetFlightInfoDropsRouteWhenSchemaFetchFails` (a failed schema fetch drops the route its query registered), `#testFailedRequestKeepsPreviousRequestRoute` (an empty query leaves the previous request's route), and `FlightSqlQueryCancellationTest#unregisterDropsWholeRoutes` (a route goes as a whole, with its addresses). `DorisFlightSqlSchemaTest#executionSchemaDriftExpiresHandleAndCleansResults` now also requires the route of a prepared statement whose result schema changed to go, with no second cancellation. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… their routes ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: When a GetFlightInfo fails after its query ran, at the Arrow schema fetch for instance, no client can pull the query's results. A deferred query is then cancelled through its coordinator, which is still open. Any other query has closed its coordinator already, and nothing cancelled it: its fragments and result buffers ran on until the BE's result buffer timer ended them, execution_timeout plus 5 s later. The query's cancellation route holds what a cancellation needs. The failure path now sends the cancellation through it to every BE of each query that no deferred coordinator cancelled, and drops the routes as before. A deferred query whose coordinator fails to cancel it falls back to its route. The failure path does not wait for the BEs' answers: the routes go whatever they are, so no retry follows, the client is waiting for the request's error, and a deferred coordinator's cancellation does not wait for them either. ### Release note None ### Check List (For Author) - Test: Unit Test - `DorisFlightSqlProducerTest#testGetFlightInfoCancelsQueryAndDropsRouteWhenSchemaFetchFails` (a query that was not deferred is cancelled on both its BEs and its route goes), `#testGetFlightInfoLeavesDeferredQueryToItsCoordinator` (a deferred query is cancelled once, through its coordinator), and `FlightSqlQueryCancellationTest#failedRequestCancelsWithoutWaitingAndDropsRoutes` (BEs that never answer do not hold the request up). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A Flight result stream closed before EOF asks the running FEs, in address order, to cancel its query until one answers OK. Only the FE that planned the query holds the route, and the list includes the FEs the master reports with process UUID 0: those it cannot reach, which stay on a BE's list until they are dropped. Every such FE that sorted before the planning FE delayed every early close on every BE by its connection budget: about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect, and 4 s when it accepts connections and never answers. The planning FE is one the master reaches, unless it has lost the master's heartbeat since. The FEs with a non-zero process UUID are now asked first and the others last, so a stopped FE delays an early close only when no reachable FE holds the route: when the route has expired or been pushed out, or when the planning FE is itself the unreachable one. An FE that hangs while the master still reaches it costs up to 4 s as before, until the master's heartbeat to it fails. No overall deadline is added: with this order it would only cut those cases short, and in the last one the early close would lose its cancellation. ### Release note None ### Check List (For Author) - Test: Unit Test - `ArrowFlightRemoteCancelTest#CloseAsksReachableFrontendsFirst`: an FE that accepts connections and never answers, reported with process UUID 0 and sorting before the planning FE, no longer delays the close (14 ms; about 4 s with the FEs asked in address order). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t stream closes early ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: Port the FE route of branch-4.x apache#68630 to master, on top of apache#69945 (its BE-local half). With apache#69945 alone, a DoGet stream closed before EOF cancels its query only on the BE that holds the stream's results. A query whose results are spread over several BEs (parallel result sink, one endpoint per BE) kept running on the other BEs: master closes the coordinator when GetFlightInfo returns (apache#67504), so nothing sends them a cancel, and a single-fragment query has no exchange that would carry the cancellation between BEs. Their pipelines ran until the query finished or timed out. A remote reader (a DoGet served by a BE other than the result BE) could not cancel the result BE's query at all. The FE now keeps a route for each Flight query that leaves its results on the BEs: from each result id (the fragment instance id of an endpoint) to the query id and the BRPC addresses of all its BEs (FlightSqlQueryCancellation, registered by StmtExecutor before the coordinator is closed or deferred). The route belongs to the session's FlightSqlChannel and is dropped when the session's next request begins (FlightProtocolAdapter.beginRequest resets the channel), when the session is torn down (releaseSession closes the channel), when a cancellation succeeds, and at the latest execution_timeout + 5 s after registration. The new FrontendService RPC cancelFlightQuery(result_id) sends the existing cancel_plan_fragment, which older BEs understand and which does not use the Flight read pool, to every BE of the query and waits at most 1 s for all of them; when any BE fails, the route stays so that the caller can retry. A reader closed before EOF now asks the running FEs in turn: an FE without the route answers NOT_FOUND and the BE moves on to the next one, a failed attempt is retried once, and the transport is reopened after a thrift error. This runs on the DoGet thread, so with an unreachable FE one early close can take a few seconds (two attempts with a 2 s RPC timeout per FE). Compatibility: cancelFlightQuery has the same name and signature as on branch-4.1 and branch-4.2. An FE without it fails the call; the BE logs a warning for that FE and only the local cleanup of apache#69945 takes effect. A result BE without apache#69945 cancels the query but keeps its result buffer until its usual timeout. Differences from branch-4.x: the route is registered in master's `!isReturnResultFromLocal()` branch (apache#67900), before the `mustOutliveDispatch()` deferral, so deferred queries are routed as well; FlightProtocolAdapter needs no change because the route stays in FlightSqlChannel; the BE reader includes runtime/frontend_info.h itself (exec_env.h no longer includes it, apache#66400); the regression suite keeps its table at the end (AGENTS.md). ### Release note Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on every BE that runs it, not only on the BE that serves the stream. The FE and the BEs must both be upgraded. ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5, new), StmtExecutorTest and 15 other Flight session / coordinator test classes - BE UT: ArrowFlightCancelTest (9), ArrowFlightRemoteCancelTest (9, 6 new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4) - Regression: arrow_flight_sql_p0/test_flight_cancel_cleanup (new) and the rest of arrow_flight_sql_p0 on a local single-BE cluster; the cross-BE path is covered by the BE UT's in-process FE and remote BE - Behavior changed: Yes. A Flight result stream closed before EOF cancels its query on every BE of the query. - Does this need documentation: No Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s next request ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: The FE dropped a Flight query's cancellation route when the session's next request began and when the session was torn down (FlightSqlChannel.reset). The client reads the results from the BEs, so it can still hold a stream of the query open at that point, for example through a second statement on the same connection, and close that stream early afterwards. The BE's cancelFlightQuery then found no route, and the query's other BEs kept their pipelines and result buffers until the query timeout. branch-4.x (apache#68630) drops the route at the same points. A route now belongs to no session. It goes when its cancellation succeeds, when its query's execution timeout plus 5 s has passed, and, oldest first and as a whole, when the registry would hold more than 100,000 result ids, a few hundred bytes each. Without the session as a bound, that cap is what keeps the registry from growing with the query rate times the query timeout; a query whose route was pushed out ends at its timeout, as every Flight query did before this PR. The registry is an insertion-ordered map under a lock, with the RPCs sent outside it, instead of a Caffeine cache, whose size eviction may reject a new entry instead of evicting the oldest one. StmtExecutor registers a query through FlightSqlQueryCancellation.register(ConnectContext, Coordinator), and FlightSqlChannel is back to its master form. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5): a route survives the session's next request and its teardown and still cancels; the bound drops whole routes, oldest first; an expired route leaves at the next registration. With the 16 other Flight session and coordinator test classes. - Regression: test_flight_cancel_cleanup gains a mode that runs the session's next request before it aborts the stream. - Behavior changed: Yes. A stream aborted after its session moved on to another request, or was closed, still cancels its query on every BE. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rdown ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A Flight result stream closed before EOF asks the running FEs in turn to cancel its query (`cancelFlightQuery`), on the DoGet thread, from `Close()` and from the reader's destructor. The 2 s timeout it gave applied only to a connection it opened itself: `ClientCacheHelper::get_client` returns a cached transport with the timeout of whichever caller opened it, such as 60 s for profile and status reports, or the query's execution timeout (900 s by default) for the information_schema scanners. Acquiring a connection also backed off 1 + 2 + 4 s after failures, and each FE got two attempts. An FE that sorts before the one that planned the query is asked first on every BE, and a stopped FE stays in that list as long as it is not dropped (the master sends it with process UUID 0, which `get_running_frontends()` counts as running). So every early close waited about 14.6 s for such an FE when it refused connections, 32.6 s when it did not answer the connect, and up to minutes when a long-timeout transport to it sat in the cache. The other way round, this call's 2 s transports, and the 1 s ones it reopened after any thrift error (an older FE's unknown-method reply included), went back to the shared cache, where `fetchSplitBatch`, whose FE side holds the call for up to 1 s, or a status report could take them. The cancellation now keeps FE connections of its own (`ExecEnv::flight_cancel_frontend_client_cache`, at most two kept per FE, as each idle one holds an FE thrift thread), so every transport it uses was opened with its 2 s timeout and none reaches another caller. Acquiring a connection does not back off (`max_retries = 0`), a transport is reopened with the same 2 s, and an FE that cannot be connected to gets no second attempt, since neither reason for the retry (the FE's own deadline, a stale cached transport) applies to it. An FE ahead of the planning one now costs an early close about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect (`thrift_connect_timeout_seconds`), and 4 s when it accepts the connection and does not answer. ### Release note None ### Check List (For Author) - Test: Unit Test - `ArrowFlightRemoteCancelTest#CloseIgnoresLongTimeoutSharedTransport` leaves a 60 s transport to an FE whose first answer stalls 5 s in the shared cache and requires the close to finish within 4 s, through the retry; `#CloseMovesPastUnreachableFrontend` puts an FE that refuses connections first and requires the close to reach the planning FE within 2 s. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… registry bound ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A route of `FlightSqlQueryCancellation` keeps the BRPC addresses of all the BEs of its query, and the result ids of a query share one route, but the registry bound counted only the result ids. In parallel result mode, the default, a query has one result id, so a full registry of 100,000 routes held about 0.3 GB with 100 BEs per query and 2.9 GB with 1,000. Routes no longer end with their session, and a stream read to EOF does not drop its route, so a registry fills up whenever queries arrive faster than 100,000 per execution timeout (about 111 per second with the default 900 s). The registry now also bounds the addresses its routes hold, at 1,000,000 (about 30 MB), and drops whole routes, oldest first, while either bound is exceeded. A route's addresses leave with its last result id, also when a newer registration takes an id over, and are released only once when the route is pushed out while a cancellation through it is in flight. The registry now stays within about 55 MB whatever the cluster size. Large clusters give up retention instead: with 1,000 BEs per query it keeps the routes of the last 1,000 queries, and a query whose route was pushed out ends at its timeout, as before routes existed. A route must have a result id, which every Flight query that leaves its results on the BEs has. ### Release note None ### Check List (For Author) - Test: Unit Test - `FlightSqlQueryCancellationTest#boundCountsRetainedBackends` (addresses past the bound push out the oldest route as a whole) and `#reregisteredResultReleasesOldRouteBackends` (a route whose result id a newer registration took over releases its addresses once). The coordinator mock of `StmtExecutorTest#testCloudReplanDispatchProfileLifecycle` now adds an endpoint in `exec()`, as the coordinator does, since a route needs a result id. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… fails ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A Flight query whose results stay on the BEs registers its cancellation route in `StmtExecutor`, before GetFlightInfo fetches the Arrow schema from the result BE and builds the tickets. When the request fails after that, at the schema fetch or because a prepared statement's result schema no longer matches, no ticket names the query's results, so no BE can report an early close of them, yet the route stayed in `FlightSqlQueryCancellation` until its query's timeout plus 5 s, or until newer routes pushed it out. While routes belonged to the session, its next request dropped them; since routes outlive the session, nothing did. A GetFlightInfo that fails after it began now drops the routes of its endpoints, each as a whole, and so does the prepared-statement schema check before it resets the session. A request that fails before it begins, such as an empty query, finds the previous request's endpoints on the session and leaves their routes: a client may still be pulling those results. ### Release note None ### Check List (For Author) - Test: Unit Test - `DorisFlightSqlProducerTest#testGetFlightInfoDropsRouteWhenSchemaFetchFails` (a failed schema fetch drops the route its query registered), `#testFailedRequestKeepsPreviousRequestRoute` (an empty query leaves the previous request's route), and `FlightSqlQueryCancellationTest#unregisterDropsWholeRoutes` (a route goes as a whole, with its addresses). `DorisFlightSqlSchemaTest#executionSchemaDriftExpiresHandleAndCleansResults` now also requires the route of a prepared statement whose result schema changed to go, with no second cancellation. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… their routes ### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: When a GetFlightInfo fails after its query ran, at the Arrow schema fetch for instance, no client can pull the query's results. A deferred query is then cancelled through its coordinator, which is still open. Any other query has closed its coordinator already, and nothing cancelled it: its fragments and result buffers ran on until the BE's result buffer timer ended them, execution_timeout plus 5 s later. The query's cancellation route holds what a cancellation needs. The failure path now sends the cancellation through it to every BE of each query that no deferred coordinator cancelled, and drops the routes as before. A deferred query whose coordinator fails to cancel it falls back to its route. The failure path does not wait for the BEs' answers: the routes go whatever they are, so no retry follows, the client is waiting for the request's error, and a deferred coordinator's cancellation does not wait for them either. ### Release note None ### Check List (For Author) - Test: Unit Test - `DorisFlightSqlProducerTest#testGetFlightInfoCancelsQueryAndDropsRouteWhenSchemaFetchFails` (a query that was not deferred is cancelled on both its BEs and its route goes), `#testGetFlightInfoLeavesDeferredQueryToItsCoordinator` (a deferred query is cancelled once, through its coordinator), and `FlightSqlQueryCancellationTest#failedRequestCancelsWithoutWaitingAndDropsRoutes` (BEs that never answer do not hold the request up). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#69945 Problem Summary: A Flight result stream closed before EOF asks the running FEs, in address order, to cancel its query until one answers OK. Only the FE that planned the query holds the route, and the list includes the FEs the master reports with process UUID 0: those it cannot reach, which stay on a BE's list until they are dropped. Every such FE that sorted before the planning FE delayed every early close on every BE by its connection budget: about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect, and 4 s when it accepts connections and never answers. The planning FE is one the master reaches, unless it has lost the master's heartbeat since. The FEs with a non-zero process UUID are now asked first and the others last, so a stopped FE delays an early close only when no reachable FE holds the route: when the route has expired or been pushed out, or when the planning FE is itself the unreachable one. An FE that hangs while the master still reaches it costs up to 4 s as before, until the master's heartbeat to it fails. No overall deadline is added: with this order it would only cut those cases short, and in the last one the early close would lose its cancellation. ### Release note None ### Check List (For Author) - Test: Unit Test - `ArrowFlightRemoteCancelTest#CloseAsksReachableFrontendsFirst`: an FE that accepts connections and never answers, reported with process UUID 0 and sorting before the planning FE, no longer delays the close (14 ms; about 4 s with the FEs asked in address order). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

morningman added the backport/4.2-master label Nov 2, 2026

morningman mentioned this pull request Nov 2, 2026

[fix](be) Return missing-buffer errors for MySQL and Arrow result fetches #67644

Open

2 of 11 tasks

github-actions Bot reviewed Nov 2, 2026

View reviewed changes

morningman mentioned this pull request Nov 2, 2026

[fix](arrow-flight) Cancel a Flight query on all its BEs when a result stream closes early #69996

Open

github-actions Bot reviewed Nov 3, 2026

View reviewed changes

Gabriel39 approved these changes Nov 4, 2026

View reviewed changes

morningman merged commit 30ce25d into apache:master Nov 4, 2026
37 checks passed

github-actions Bot added the reviewed label Nov 4, 2026

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 4, 2026

Conversation

morningman commented Nov 2, 2026

What problem does this PR solve?

In short

Background

The problem, and what it cost

How this PR fixes it

Results

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

morningman commented Nov 2, 2026

Uh oh!

morningman commented Nov 2, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

morningman Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Nov 2, 2026

BE Regression && UT Coverage Report

Uh oh!

morningman commented Nov 2, 2026

Uh oh!

morningman commented Nov 2, 2026

Uh oh!

morningman commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

hello-stephen commented Nov 2, 2026

BE Regression && UT Coverage Report

Uh oh!

morningman commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

morningman commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

morningman commented Nov 3, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development