[fix](arrow-flight) Release a finished Flight query's coordinator instead of holding it until wait_timeout by raghav-reglobe · Pull Request #67504 · apache/doris

raghav-reglobe · 2026-09-03T15:21:19Z

What problem does this PR solve?

Issue Number: close #67503

Related PR: #64799 (kept the coordinator alive across GetFlightInfo → DoGet so the BE can fetch external-table splits; this PR narrows that to the queries that actually need it and bounds how long an abandoned session can hold the rest)

Problem Summary:

Since #64799 every Arrow Flight SQL query whose results are produced on the BE kept its coordinator alive after GetFlightInfo, until the session ran its next query or was closed. The coordinator holds the query's workload-group queue slot and its active_queries registration. Most Flight clients open a session per query and never send CloseSession (the ADBC/JDBC drivers don't do it on connection close, as the comment in DorisFlightSqlProducer.closeSession notes), so each abandoned session pinned one slot for up to wait_timeout (8h by default).

Observed on a 2-FE cluster with a max_concurrency=8 workload group: eight finished Flight queries sat in information_schema.active_queries as RUNNING for over an hour, and every later query in the group failed with query queue timeout, timeout: 60000 ms. Full write-up in #67503.

This PR fixes it in two steps:

Only defer the coordinator when the BE still needs it. The deferral added by [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799 is required only for an external-table scan in batch mode, where the BE lazily fetches splits during DoGet from the split source the coordinator holds. Every other query (internal tables, external tables in non-batch mode) now closes its coordinator at the end of GetFlightInfo again, releasing the queue slot and the active_queries entry right away. That is safe: the BE buffers its results independently of the coordinator, and finalizing the FE side does not cancel BE execution (QeProcessorImpl.reportExecStatus accepts reports for an unregistered query, and Coordinator.close() only releases the queue token and the split sources). New predicates ScanNode.hasBatchSplitSource() / Coordinator.hasBatchSplitSource() drive the gate in StmtExecutor.executeAndSendResult.
Bound the remaining deferred queries without killing the session. New mutable FE config arrow_flight_deferred_query_idle_timeout_second (default 3600). Once a Flight session has been sleeping for longer than this since its last query started, the connection timeout checker finalizes the session's deferred executors (releasing the slot and the registration) and leaves the session alive; wait_timeout still governs the session itself. The bound is floored at the execution timeout the deferred query actually ran with, captured when the executor is deferred (SET_VAR hint values are reverted at the end of execute(), so the session value cannot be read later). 0 disables the bound.

Why not kill the session (the first revision of this PR): a killed Flight session's bearer token stays in the token cache marked as already used, so the client's next call on it fails with UserSession expire after access and has to re-handshake. With a 1h bound, every pooled or BI-tool Flight connection that idles for an hour would fail once on its next use. Reaping only the deferred query releases the leaked resources with no client-visible change.

Release note

Arrow Flight SQL: a query no longer holds its workload-group queue slot and active_queries entry after GetFlightInfo unless it is an external-table scan in batch mode, the only case where the BE still fetches splits from the FE during DoGet. For that case a new FE config arrow_flight_deferred_query_idle_timeout_second (default 1h) releases the coordinator of an idle, never-closed session without killing the session.

Check List (For Author)

Test
- Regression test (arrow_flight_sql_p0/test_arrow_flight_query_release: a finished Flight query on an internal table no longer occupies a max_concurrency=1 group and is gone from active_queries; external_table_p0/iceberg/test_iceberg_arrow_flight_split_source: the batch-mode scan stays registered after DoGet and is released by the idle reaper while the session survives)
- Unit Test (FlightSqlDeferredQueryIdleTimeoutTest: the reaper through checkTimeout, the exec-timeout floor, 0 disables, nothing deferred, MySQL untouched; ArrowFlightDeferralGateTest: the batch-split-source predicates; StmtExecutorTest.testDeferForArrowFlightFreezesExecTimeoutInEffect)
- Manual test (the first revision was verified on a test cluster with an 8s bound, see the PR history; the current revision is covered by the regression tests above)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes. (1) A Flight query that is not an external-table batch-mode scan releases its coordinator, queue slot and active_queries entry at the end of GetFlightInfo, as it did before [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799. (2) The deferred coordinator of a batch-mode scan on an idle session is released after 1h by default instead of at wait_timeout; set arrow_flight_deferred_query_idle_timeout_second=0 for the previous behavior. Sessions are never killed by this change.
Does this need documentation?
- No.
- Yes. New FE config arrow_flight_deferred_query_idle_timeout_second — doris-website PR to follow once this is reviewed.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-09-03T15:21:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…instead of killing the session Follow-up to the first revision of apache#67504 after review: - Narrow the apache#64799 deferral gate: only a coordinator that still hands out splits to the BE (an external-table scan in batch mode, see the new ScanNode/Coordinator.hasBatchSplitSource) outlives GetFlightInfo. Every other Arrow Flight query closes its coordinator at the end of GetFlightInfo again, releasing the workload-group queue slot and the active_queries entry right away. Finalizing the FE side does not cancel BE execution, so DoGet is unaffected. - Replace arrow_flight_session_idle_timeout_second by arrow_flight_deferred_query_idle_timeout_second: the connection timeout checker now finalizes the deferred executors of a sleeping Flight session and leaves the session alive; wait_timeout still governs the session. A killed session would have made the client's next call fail with "UserSession expire after access". - Freeze the execution timeout when the executor is deferred, so a SET_VAR query_timeout hint (reverted at the end of execute()) still floors the bound. - Tests: FlightSqlDeferredQueryIdleTimeoutTest drives checkTimeout, ArrowFlightDeferralGateTest covers the predicates, StmtExecutorTest covers the frozen timeout; regression cases for the internal-table release (arrow_flight_sql_p0) and the idle reaper on a batch-mode Iceberg scan. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApxvNkMK8TiNycv44P8Ezk

morningman · 2026-09-05T07:33:09Z

Thanks for the precise write-up, the problem is real and #67503 nails the mechanism. I reviewed the branch and pushed a revision on top of your commit (the PR allows maintainer edits); the description is updated to match. What changed and why:

Reap the deferred query instead of killing the session. What leaks is the deferred coordinator, not the session. Killing the session has a client-visible cost: unregisterConnection leaves the bearer token in the cache marked as already used, so the client's next call fails with UserSession expire after access and has to re-handshake. With a 1h default, every pooled / BI-tool Flight connection that idles for an hour would fail once on its next use. The checker now finalizes the session's deferred executors when the bound passes and leaves the session under wait_timeout. The config is renamed to arrow_flight_deferred_query_idle_timeout_second accordingly.
Narrow the deferral gate to the queries that need it. [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799 deferred every BE-served Flight query, but only an external-table scan in batch mode needs the coordinator after GetFlightInfo (the BE fetches splits from its split source during DoGet). Finalizing the FE side does not cancel BE execution (reportExecStatus accepts reports for an unregistered query; Coordinator.close() only releases the queue token and the split sources), so every other query now closes its coordinator at the end of GetFlightInfo again, exactly as before [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799. This removes the leak for internal tables entirely, with no timeout involved; the idle bound only has to cover batch-mode external scans. New predicates: ScanNode.hasBatchSplitSource() / Coordinator.hasBatchSplitSource().
Capture the exec timeout when the executor is deferred. While the session sleeps executor is null, so getExecTimeoutS() returned the current session query_timeout; SET_VAR hint values are reverted at the end of execute(). A query run with /*+ SET_VAR(query_timeout=7200) */ would have been cut at 3600s, the case the floor is meant to protect. StmtExecutor.deferForArrowFlight() now freezes the value and the bound uses the max over the deferred executors.
Tests. FlightSqlSessionIdleTimeoutTest (formula only) is replaced by FlightSqlDeferredQueryIdleTimeoutTest, which drives checkTimeout and asserts both the release and that the session is not killed; ArrowFlightDeferralGateTest covers the predicates; StmtExecutorTest covers the frozen timeout. Two regression cases: arrow_flight_sql_p0/test_arrow_flight_query_release (internal table no longer occupies a max_concurrency=1 group) and an extension of test_iceberg_arrow_flight_split_source (batch-mode scan stays registered, then is released by the reaper while the session keeps working).
Smaller things from the review: the Config description now states the floor and that the session is not killed; masterOnly = false dropped; FeConstants.runningUnitTest is restored after the test.

Pick labels: #64799 is in 4.0.8 / 4.1.4, so this needs dev/4.0.x and dev/4.1.x.

morningman · 2026-09-05T07:33:12Z

run buildall

…rom wait_timeout A Flight SQL session idles under the same wait_timeout as a MySQL connection (28800s by default). Since the coordinator of a BE-served Flight query is kept alive across GetFlightInfo -> DoGet until the session's next query or its close, an abandoned session - a client that opens a session per query and never sends CloseSession - keeps that query's workload-group queue slot for the whole wait_timeout. Eight such sessions fill a max_concurrency=8 group and every later query in it fails with "query queue timeout". Add a mutable FE config, arrow_flight_session_idle_timeout_second (default 3600), applied by the existing connection timeout checker to ARROW_FLIGHT_SQL contexts only as min(wait_timeout, max(config, exec timeout)). The exec-timeout floor matters: a Flight session is COM_SLEEP while the client drains the result via DoGet and its idle clock runs from the query's start, so a bound below query_timeout would kill a long result stream before the query's own timeout could. 0 disables the bound. MySQL-protocol connections are unchanged. The kill log line now reports the effective idle timeout. Signed-off-by: Raghvendra Singh <raghav@cashify.in> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…instead of killing the session Follow-up to the first revision of apache#67504 after review: - Narrow the apache#64799 deferral gate: only a coordinator that still hands out splits to the BE (an external-table scan in batch mode, see the new ScanNode/Coordinator.hasBatchSplitSource) outlives GetFlightInfo. Every other Arrow Flight query closes its coordinator at the end of GetFlightInfo again, releasing the workload-group queue slot and the active_queries entry right away. Finalizing the FE side does not cancel BE execution, so DoGet is unaffected. - Replace arrow_flight_session_idle_timeout_second by arrow_flight_deferred_query_idle_timeout_second: the connection timeout checker now finalizes the deferred executors of a sleeping Flight session and leaves the session alive; wait_timeout still governs the session. A killed session would have made the client's next call fail with "UserSession expire after access". - Freeze the execution timeout when the executor is deferred, so a SET_VAR query_timeout hint (reverted at the end of execute()) still floors the bound. - Tests: FlightSqlDeferredQueryIdleTimeoutTest drives checkTimeout, ArrowFlightDeferralGateTest covers the predicates, StmtExecutorTest covers the frozen timeout; regression cases for the internal-table release (arrow_flight_sql_p0) and the idle reaper on a batch-mode Iceberg scan. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApxvNkMK8TiNycv44P8Ezk

morningman · 2026-09-05T07:44:22Z

Rebased onto current master (b58b2c53ff5) and force-pushed. The only conflict was the comment above the deferral gate in StmtExecutor.executeAndSendResult against #67487: kept the narrowed gate and adopted #67487's wording about short-circuit point queries no longer reaching it. Master also moved StmtExecutorTest to JUnit 5 assertions, so the new test in that file follows suit. No other changes; the affected FE unit tests and checkstyle pass locally on the rebased tree.

morningman · 2026-09-05T07:44:26Z

run buildall

hello-stephen · 2026-09-05T09:11:05Z

TPC-H: Total hot run time: 16940 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c662d85d1634bf1af5f37ca8cc265c1d42ab0777, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17593	3012	3010	3010
q2	2086	281	235	235
q3	10223	888	525	525
q4	4695	252	200	200
q5	7672	562	379	379
q6	141	116	98	98
q7	539	498	398	398
q8	9234	900	843	843
q9	3517	2445	2420	2420
q10	6521	873	712	712
q11	394	208	190	190
q12	611	258	201	201
q13	18109	1535	1177	1177
q14	162	147	144	144
q15	q16	440	405	367	367
q17	1430	896	796	796
q18	3058	2271	2257	2257
q19	1135	904	736	736
q20	372	280	205	205
q21	5298	1809	1950	1809
q22	325	266	238	238
Total cold run time: 93555 ms
Total hot run time: 16940 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3386	3328	3300	3300
q2	510	390	373	373
q3	2195	2388	2189	2189
q4	1191	1187	911	911
q5	2184	2152	2160	2152
q6	164	123	89	89
q7	1064	950	891	891
q8	1616	1406	1410	1406
q9	3156	3125	3112	3112
q10	1866	1836	1629	1629
q11	356	276	260	260
q12	464	430	352	352
q13	1502	1550	1160	1160
q14	185	184	161	161
q15	q16	403	404	355	355
q17	3608	3353	3308	3308
q18	4853	4492	4756	4492
q19	935	961	867	867
q20	1029	967	851	851
q21	3898	3245	3182	3182
q22	406	333	323	323
Total cold run time: 34971 ms
Total hot run time: 31363 ms

hello-stephen · 2026-09-05T09:17:37Z

TPC-DS: Total hot run time: 83156 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c662d85d1634bf1af5f37ca8cc265c1d42ab0777, data reload: false

query5	4264	432	357	357
query6	395	143	129	129
query7	4889	414	248	248
query8	293	129	120	120
query9	8698	3003	2999	2999
query10	398	228	191	191
query11	5393	1059	923	923
query12	127	73	72	72
query13	1203	455	326	326
query14	6093	2259	2149	2149
query14_1	2004	2027	2039	2027
query15	173	124	121	121
query16	922	406	385	385
query17	818	480	393	393
query18	2340	334	246	246
query19	164	156	114	114
query20	74	69	78	69
query21	202	106	93	93
query22	5405	5316	5393	5316
query23	6812	6311	6207	6207
query23_1	6166	6021	6272	6021
query24	7307	1103	793	793
query24_1	819	805	816	805
query25	438	315	263	263
query26	1226	237	140	140
query27	2784	416	261	261
query28	4667	1533	1514	1514
query29	956	455	376	376
query30	255	159	133	133
query31	824	408	331	331
query32	129	79	77	77
query33	463	230	187	187
query34	1000	840	487	487
query35	398	401	362	362
query36	584	586	542	542
query37	123	84	73	73
query38	1003	876	815	815
query39	502	516	492	492
query39_1	470	454	509	454
query40	205	94	80	80
query41	60	58	59	58
query42	76	80	74	74
query43	242	241	211	211
query44	1031	567	598	567
query45	114	107	100	100
query46	755	874	513	513
query47	769	757	720	720
query48	312	309	237	237
query49	537	237	206	206
query50	738	258	196	196
query51	8149	7981	8032	7981
query52	69	69	58	58
query53	195	214	209	209
query54	231	185	162	162
query55	72	60	65	60
query56	202	170	177	170
query57	719	662	640	640
query58	207	162	179	162
query59	1202	1231	1100	1100
query60	257	193	217	193
query61	134	123	139	123
query62	366	209	198	198
query63	167	135	156	135
query64	2759	675	651	651
query65	1607	1569	1604	1569
query66	1775	258	208	208
query67	9843	9720	9786	9720
query68	2744	1238	696	696
query69	334	227	213	213
query70	665	630	633	630
query71	257	184	164	164
query72	2474	1870	1585	1585
query73	643	601	351	351
query74	1553	1230	1160	1160
query75	1186	1116	980	980
query76	2284	735	565	565
query77	263	278	214	214
query78	3998	3692	3224	3224
query79	2993	859	563	563
query80	1602	346	297	297
query81	509	159	137	137
query82	626	128	98	98
query83	282	214	192	192
query84	306	114	92	92
query85	820	367	312	312
query86	401	184	179	179
query87	1015	994	902	902
query88	2905	2136	2103	2103
query89	298	204	180	180
query90	1972	149	136	136
query91	133	123	106	106
query92	81	73	72	72
query93	1674	1083	678	678
query94	628	269	238	238
query95	517	332	245	245
query96	781	578	263	263
query97	1052	1041	1002	1002
query98	157	134	141	134
query99	418	347	314	314
Total cold run time: 178604 ms
Total hot run time: 83156 ms

hello-stephen · 2026-09-05T09:22:00Z

ClickBench: Total hot run time: 14.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c662d85d1634bf1af5f37ca8cc265c1d42ab0777, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.03	0.03
query3	0.26	0.10	0.10
query4	1.61	0.11	0.10
query5	0.18	0.16	0.18
query6	1.24	0.72	0.68
query7	0.03	0.00	0.01
query8	0.05	0.03	0.04
query9	0.28	0.21	0.22
query10	0.36	0.35	0.36
query11	0.18	0.12	0.11
query12	0.15	0.12	0.12
query13	0.31	0.32	0.32
query14	0.46	0.45	0.46
query15	0.37	0.35	0.36
query16	0.22	0.22	0.22
query17	0.72	0.70	0.68
query18	0.16	0.17	0.17
query19	1.20	1.18	1.16
query20	0.01	0.01	0.01
query21	15.43	0.18	0.11
query22	5.07	0.04	0.04
query23	16.17	0.26	0.11
query24	2.97	0.32	0.25
query25	0.09	0.04	0.03
query26	0.83	0.18	0.12
query27	0.03	0.04	0.03
query28	3.63	0.59	0.27
query29	12.42	3.17	2.59
query30	0.26	0.11	0.12
query31	2.76	0.38	0.17
query32	3.51	0.32	0.24
query33	1.66	1.66	1.44
query34	15.47	2.23	1.82
query35	1.79	1.74	1.74
query36	0.47	0.30	0.28
query37	0.06	0.03	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.09	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.85 s
Total hot run time: 14.81 s

hello-stephen · 2026-09-05T09:30:15Z

FE UT Coverage Report

Increment line coverage 88.89% (32/36) 🎉
Increment coverage report
Complete coverage report

…elease path Follow-up to apache#67504 after review: the deferral gate this PR narrows moves exactly one case - an external-table scan that is NOT in batch mode - from "coordinator deferred past GetFlightInfo" to "coordinator closed at the end of GetFlightInfo". That case had no Arrow Flight coverage anywhere: the only external-catalog Flight suite forces batch mode on its session and asserts it, so all four of its data queries exercise the deferred side, and the new arrow_flight_sql_p0 case covers the eager side only for an internal table, where neither a split source nor a connector read session exists. Add the missing cell to test_iceberg_arrow_flight_split_source, reusing the catalog and Flight connection it already sets up: - A negative control mirroring the existing batch assertion: "(approximate)" is emitted only when isBatchMode(), so its absence proves the scan really is on the synchronous split path. Without it the block could silently run in batch mode and pass. enable_external_table_batch_mode=false is a reliable off switch here - IcebergScanPlanProvider.streamingSplitEstimate returns -1 when it is unset, and Iceberg does not override supportsBatchScan (SPI default false), so the partition-count flavor cannot route around it. - A full scan that must return every row: the FE has closed the coordinator by then, and the BE buffers the result independently of it. - A check that the release really was eager - the query is gone from active_queries right after the client has the rows. No polling is needed because finalizeQuery() runs inside GetFlightInfo. A distinct limit keeps the query text apart from the other scans and the LIKE pattern is built with CONCAT so the probe cannot match itself. Batch mode is restored afterwards, since the idle-reaper assertions below need a deferred coordinator to release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGqBt7EdtV7rc7tD4nmEqx

morningman · 2026-09-05T16:13:55Z

run buildall

hello-stephen · 2026-09-05T16:15:26Z

FE Regression Coverage Report

Increment line coverage 86.11% (31/36) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-05T17:36:39Z

TPC-H: Total hot run time: 16970 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 888313142ef112a5dd52d4e536259832b16e3852, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17597	3150	3112	3112
q2	2190	250	226	226
q3	10177	890	524	524
q4	4673	249	203	203
q5	7673	558	395	395
q6	133	112	93	93
q7	538	486	386	386
q8	9244	890	876	876
q9	3442	2393	2380	2380
q10	6497	834	728	728
q11	401	211	179	179
q12	611	271	205	205
q13	18112	1526	1171	1171
q14	159	145	142	142
q15	q16	444	397	370	370
q17	1404	938	811	811
q18	3130	2242	2255	2242
q19	1275	828	714	714
q20	398	284	206	206
q21	5568	1771	1794	1771
q22	331	264	236	236
Total cold run time: 93997 ms
Total hot run time: 16970 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3501	3449	3464	3449
q2	516	398	371	371
q3	2313	2641	2196	2196
q4	1200	1172	906	906
q5	2199	2126	2112	2112
q6	163	123	85	85
q7	1046	976	924	924
q8	1633	1451	1460	1451
q9	3199	3172	3145	3145
q10	1856	1795	1625	1625
q11	368	279	258	258
q12	457	436	357	357
q13	1469	1540	1160	1160
q14	169	170	153	153
q15	q16	390	395	385	385
q17	3616	3319	3265	3265
q18	4881	4453	4859	4453
q19	949	859	846	846
q20	976	950	820	820
q21	3894	3151	3246	3151
q22	402	347	305	305
Total cold run time: 35197 ms
Total hot run time: 31417 ms

hello-stephen · 2026-09-05T17:43:10Z

TPC-DS: Total hot run time: 82259 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 888313142ef112a5dd52d4e536259832b16e3852, data reload: false

query5	4282	420	345	345
query6	397	145	123	123
query7	4929	419	236	236
query8	298	125	132	125
query9	8694	2982	2980	2980
query10	381	223	183	183
query11	5372	1059	927	927
query12	118	73	72	72
query13	1189	448	298	298
query14	6157	2230	2104	2104
query14_1	2018	2006	1995	1995
query15	170	124	114	114
query16	922	369	315	315
query17	817	462	374	374
query18	2344	329	230	230
query19	165	133	106	106
query20	78	70	77	70
query21	192	100	85	85
query22	5548	5213	5244	5213
query23	6760	6332	6012	6012
query23_1	6184	6097	5931	5931
query24	7264	1103	761	761
query24_1	770	792	751	751
query25	414	269	245	245
query26	1228	251	128	128
query27	2768	405	247	247
query28	4738	1483	1491	1483
query29	915	422	332	332
query30	267	153	133	133
query31	816	399	324	324
query32	127	69	71	69
query33	469	217	179	179
query34	1013	871	496	496
query35	422	395	349	349
query36	578	563	520	520
query37	120	78	69	69
query38	1022	836	830	830
query39	508	478	489	478
query39_1	500	462	480	462
query40	207	97	80	80
query41	60	57	57	57
query42	74	73	77	73
query43	243	249	215	215
query44	1016	560	566	560
query45	112	108	101	101
query46	793	837	540	540
query47	778	790	709	709
query48	319	322	244	244
query49	554	256	194	194
query50	777	265	206	206
query51	8344	8520	8231	8231
query52	67	68	60	60
query53	198	200	159	159
query54	323	189	171	171
query55	69	62	57	57
query56	226	180	171	171
query57	701	639	634	634
query58	215	177	182	177
query59	1235	1317	1117	1117
query60	239	188	169	169
query61	136	119	136	119
query62	378	210	179	179
query63	171	143	134	134
query64	2837	702	611	611
query65	1609	1558	1585	1558
query66	2017	267	220	220
query67	10092	9826	9616	9616
query68	2783	1183	769	769
query69	344	231	196	196
query70	691	611	648	611
query71	248	172	193	172
query72	2331	1748	1626	1626
query73	653	562	337	337
query74	1564	1229	1135	1135
query75	1177	1095	962	962
query76	2291	743	551	551
query77	252	283	221	221
query78	3858	3599	3132	3132
query79	2578	827	595	595
query80	1568	319	257	257
query81	487	158	135	135
query82	639	127	98	98
query83	291	205	191	191
query84	297	115	90	90
query85	774	362	303	303
query86	392	173	176	173
query87	1037	974	928	928
query88	2796	2106	2102	2102
query89	293	199	173	173
query90	1985	132	130	130
query91	132	120	102	102
query92	82	71	67	67
query93	1478	1040	719	719
query94	641	271	226	226
query95	499	317	223	223
query96	821	565	268	268
query97	1076	1052	1033	1033
query98	166	139	136	136
query99	419	347	309	309
Total cold run time: 178798 ms
Total hot run time: 82259 ms

hello-stephen · 2026-09-05T17:47:32Z

ClickBench: Total hot run time: 14.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 888313142ef112a5dd52d4e536259832b16e3852, data reload: false

query1	0.01	0.00	0.00
query2	0.08	0.04	0.04
query3	0.25	0.10	0.11
query4	1.61	0.10	0.09
query5	0.18	0.16	0.16
query6	1.24	0.68	0.68
query7	0.03	0.00	0.01
query8	0.05	0.03	0.03
query9	0.30	0.22	0.22
query10	0.34	0.34	0.34
query11	0.18	0.12	0.12
query12	0.15	0.13	0.12
query13	0.32	0.31	0.31
query14	0.45	0.45	0.45
query15	0.38	0.38	0.35
query16	0.23	0.23	0.23
query17	0.74	0.72	0.69
query18	0.18	0.16	0.17
query19	1.24	1.13	1.16
query20	0.01	0.01	0.01
query21	15.47	0.17	0.12
query22	5.07	0.04	0.04
query23	16.14	0.24	0.11
query24	3.05	0.33	0.28
query25	0.11	0.05	0.04
query26	0.73	0.17	0.12
query27	0.04	0.04	0.02
query28	3.63	0.55	0.26
query29	12.50	3.26	2.56
query30	0.26	0.12	0.13
query31	2.76	0.39	0.17
query32	3.49	0.30	0.23
query33	1.41	1.39	1.38
query34	15.40	2.23	1.81
query35	1.78	1.74	1.76
query36	0.47	0.29	0.29
query37	0.06	0.04	0.04
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.12	0.08	0.08
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.02	0.02
Total cold run time: 90.69 s
Total hot run time: 14.75 s

morningman · 2026-09-05T22:19:26Z

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#67504
commit: 888313142ef112a5dd52d4e536259832b16e3852
base: b58b2c53ff56354c692c2dc7634d724d8780838f
reviewed_at: 2026-09-06T06:19+08:00
reviewer: morningman
model: claude-opus-5[1m]
effort: max
findings: {blocker: 0, major: 0, minor: 14, nit: 4}
rounds: 2
converged: true

Notes for maintainers

fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:1598 — the gate also un-defers finalizeQuery(), so an ACID Hive scan's metastore read lock is committed inside GetFlightInfo, before DoGet. Reviewed as Major on the previous head; rated Minor here because the blast radius is exactly Hive ACID (every other connector inherits the no-op releaseReadTransaction, per PluginDrivenScanNode.java:1471-1474) and that read path is slated for deprecation. Still worth fixing; suggested patch in the review document.
regression-test/suites/external_table_p0/iceberg/test_iceberg_arrow_flight_split_source.groovy:128-162 — the (external, non-batch, Arrow Flight) row that had no coverage anywhere is now covered by this head's test commit, with a negative control on the split path and an eager-release assertion. That closes the only other Major from the previous review.
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:1109-1117 — the mirror case is still open: a deferred batch-mode Flight query never closes its connector statement scope.
fe/fe-core/src/test/java/org/apache/doris/qe/ArrowFlightDeferralGateTest.java:57-68 — the unit test named for the deferral gate does not exercise the gate.
regression-test/suites/external_table_p0/iceberg/test_iceberg_arrow_flight_split_source.groovy:171-174 — the idle-reaper assertions give the reaper zero timing margin, which can reproduce [Bug] Arrow Flight SQL + Iceberg: SplitSource released before DoGet (fetchSplitBatch crash) #62259 as a false red in CI.

_{Reviewed locally with the doris-repo-review pipeline (a local port of .github/workflows/code-review-runner.yml). This is not a CI status check.}

…cuit on an Arrow Flight connection #67487 (#67557) Cherry-picked from #67487 ### Backport notes `LogicalResultSinkToShortCircuitPointQuery` and the `StmtExecutor` comment carry the upstream change; the rule keeps this branch's `private` signature for `scanMatchShortCircuitCondition`, since nothing on branch-4.1 calls it directly (upstream made it package-private for a different test, added by a PR that is not on this branch). `ShortCircuitPointQueryTest` does not carry the later master cases here, so it has no `rewrite(sql)` helper. The helper is added with the same body as upstream and the new case uses it; the existing case is untouched. The regression suite `arrow_flight_sql_p0/test_point_query_over_arrow_flight.groovy` applied cleanly. ### Merge order Touches the same `StmtExecutor` comment as #67558 (the branch-4.1 backport of #67504). Merging this one first keeps that one conflict-free; the other way round leaves a trivial comment conflict. There is no code dependency either way. ### Local verification `./build.sh --fe` on this branch: **BUILD SUCCESS**, no errors, checkstyle clean on every module (`fe-common` and `fe-core` included). Regression suites were not run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DzUYFcGHQH3bnLGCjpncVj

…tead of holding it until wait_timeout (apache#67504) Issue Number: close apache#67503 Related PR: apache#64799 (kept the coordinator alive across GetFlightInfo → DoGet so the BE can fetch external-table splits; this PR narrows that to the queries that actually need it and bounds how long an abandoned session can hold the rest) Problem Summary: Since apache#64799 every Arrow Flight SQL query whose results are produced on the BE kept its coordinator alive after `GetFlightInfo`, until the session ran its next query or was closed. The coordinator holds the query's workload-group queue slot and its `active_queries` registration. Most Flight clients open a session per query and never send `CloseSession` (the ADBC/JDBC drivers don't do it on connection close, as the comment in `DorisFlightSqlProducer.closeSession` notes), so each abandoned session pinned one slot for up to `wait_timeout` (8h by default). Observed on a 2-FE cluster with a `max_concurrency=8` workload group: eight finished Flight queries sat in `information_schema.active_queries` as RUNNING for over an hour, and every later query in the group failed with `query queue timeout, timeout: 60000 ms`. Full write-up in apache#67503. This PR fixes it in two steps: 1. **Only defer the coordinator when the BE still needs it.** The deferral added by apache#64799 is required only for an external-table scan in batch mode, where the BE lazily fetches splits during `DoGet` from the split source the coordinator holds. Every other query (internal tables, external tables in non-batch mode) now closes its coordinator at the end of `GetFlightInfo` again, releasing the queue slot and the `active_queries` entry right away. That is safe: the BE buffers its results independently of the coordinator, and finalizing the FE side does not cancel BE execution (`QeProcessorImpl.reportExecStatus` accepts reports for an unregistered query, and `Coordinator.close()` only releases the queue token and the split sources). New predicates `ScanNode.hasBatchSplitSource()` / `Coordinator.hasBatchSplitSource()` drive the gate in `StmtExecutor.executeAndSendResult`. 2. **Bound the remaining deferred queries without killing the session.** New mutable FE config `arrow_flight_deferred_query_idle_timeout_second` (default 3600). Once a Flight session has been sleeping for longer than this since its last query started, the connection timeout checker finalizes the session's deferred executors (releasing the slot and the registration) and leaves the session alive; `wait_timeout` still governs the session itself. The bound is floored at the execution timeout the deferred query actually ran with, captured when the executor is deferred (`SET_VAR` hint values are reverted at the end of `execute()`, so the session value cannot be read later). `0` disables the bound. Why not kill the session (the first revision of this PR): a killed Flight session's bearer token stays in the token cache marked as already used, so the client's next call on it fails with `UserSession expire after access` and has to re-handshake. With a 1h bound, every pooled or BI-tool Flight connection that idles for an hour would fail once on its next use. Reaping only the deferred query releases the leaked resources with no client-visible change. Arrow Flight SQL: a query no longer holds its workload-group queue slot and `active_queries` entry after `GetFlightInfo` unless it is an external-table scan in batch mode, the only case where the BE still fetches splits from the FE during `DoGet`. For that case a new FE config `arrow_flight_deferred_query_idle_timeout_second` (default 1h) releases the coordinator of an idle, never-closed session without killing the session. - Test  - [x] Regression test (`arrow_flight_sql_p0/test_arrow_flight_query_release`: a finished Flight query on an internal table no longer occupies a `max_concurrency=1` group and is gone from `active_queries`; `external_table_p0/iceberg/test_iceberg_arrow_flight_split_source`: the batch-mode scan stays registered after `DoGet` and is released by the idle reaper while the session survives) - [x] Unit Test (`FlightSqlDeferredQueryIdleTimeoutTest`: the reaper through `checkTimeout`, the exec-timeout floor, `0` disables, nothing deferred, MySQL untouched; `ArrowFlightDeferralGateTest`: the batch-split-source predicates; `StmtExecutorTest.testDeferForArrowFlightFreezesExecTimeoutInEffect`) - [x] Manual test (the first revision was verified on a test cluster with an 8s bound, see the PR history; the current revision is covered by the regression tests above) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes. (1) A Flight query that is not an external-table batch-mode scan releases its coordinator, queue slot and `active_queries` entry at the end of `GetFlightInfo`, as it did before apache#64799. (2) The deferred coordinator of a batch-mode scan on an idle session is released after 1h by default instead of at `wait_timeout`; set `arrow_flight_deferred_query_idle_timeout_second=0` for the previous behavior. Sessions are never killed by this change. - Does this need documentation? - [ ] No. - [x] Yes. New FE config `arrow_flight_deferred_query_idle_timeout_second` — doris-website PR to follow once this is reviewed. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Signed-off-by: Raghvendra Singh <raghav@cashify.in> Co-authored-by: Raghav <raghav@Raghavs-RMac.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: morningman <yunyou@selectdb.com> (cherry picked from commit 23b5ee5) Two adaptations for branch-4.1: - SplitAssignment is org.apache.doris.datasource.SplitAssignment here, not org.apache.doris.datasource.split.SplitAssignment, so ArrowFlightDeferralGateTest imports it from its package on this branch. - The StmtExecutor comment above the deferral gate is taken from upstream verbatim, so it already mentions apache#67368, whose branch-4.1 backport (apache#67557) has since landed. The rebase onto it resolved the one conflict in StmtExecutor, in that comment and the gate condition; the resolved block is identical to upstream master.

…rdinator instead of holding it until wait_timeout #67504 (#67558) Cherry-picked from #67504 ### Backport notes Two conflicts, both resolved without changing behaviour: - `Config`: branch-4.1 still has the bilingual `autobucket_partition_size_per_bucket_gb` description that master rewrote, so that entry is kept as it is here and the new `arrow_flight_deferred_query_idle_timeout_second` is inserted above it, verbatim. - `StmtExecutor`: the comment above the deferral gate is taken from upstream verbatim, so it already mentions #67368. That fix is being backported separately (see below); the gate itself does not depend on it. One adaptation: `SplitAssignment` is `org.apache.doris.datasource.SplitAssignment` on this branch, not `org.apache.doris.datasource.split.SplitAssignment`, so `ArrowFlightDeferralGateTest` imports it from its package here. Everything else -- `ScanNode.hasBatchSplitSource`, `Coordinator.hasBatchSplitSource`, the `ConnectContext` idle reaper, `FlightSqlConnectProcessor`, `StmtExecutorTest`, `FlightSqlDeferredQueryIdleTimeoutTest` and both regression suites -- applied cleanly. ### Merge order Touches the same `StmtExecutor` comment as #67557 (the branch-4.1 backport of #67487). Merging that one first keeps this one conflict-free. ### Local verification `./build.sh --fe` on this branch: **BUILD SUCCESS** (2:26 min), no errors, checkstyle clean on every module -- `fe-common` (the new `Config` entry) and `fe-core` (the five main-source changes plus `ArrowFlightDeferralGateTest`, `FlightSqlDeferredQueryIdleTimeoutTest` and the new `StmtExecutorTest` case) included. The regression suites were not run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DzUYFcGHQH3bnLGCjpncVj Co-authored-by: Raghvendra Singh <raghav@cashify.in>

…stream closes early (#69945) ### What problem does this PR solve? Issue Number: None Related PR: #68630, #67644 Problem Summary: #### In short This ports the BE half of branch-4.x's #68630 to master. When an Arrow Flight SQL client cancels or closes a result stream before its end, master keeps the query running: the BE thread serving the stream waits on the result buffer without ever checking whether the client is still there, and the unread result blocks stay in memory, holding back the query, until the query times out or the delayed buffer cleanup runs minutes later. With this PR the BE notices the cancelled stream within 20 ms, and when the stream is served by the BE that holds its result (the usual case), that BE cancels the query and frees all of the query's Flight result buffers there. The FE route that also reaches the query's other BEs (`cancelFlightQuery`, which changes `FrontendService.thrift`) comes in a follow-up PR. #### Background - A Flight SQL query leaves its result on the BEs. `GetFlightInfo` returns one endpoint per result buffer, and the client reads each endpoint with a `DoGet` stream. Since #67504 the FE closes the coordinator as soon as it has returned the `FlightInfo`, so it no longer cancels anything on behalf of an open stream. - `DoGet` is served by `ArrowFlightBatchLocalReader` when the result buffer is on the same BE, or by `ArrowFlightBatchRemoteReader`, which pulls the blocks from the result BE with the `fetch_arrow_data` BRPC. - The result buffer (`ArrowFlightResultBlockBuffer`) is keyed by the query id when parallel result sink is on (the default) and by the fragment instance id when it is off. When it is full, the result sink stops and the query waits for the reader. - `ResultBufferMgr` drops a buffer that nobody removed at `execution_timeout + 5 s`, or `result_buffer_cancelled_interval_time` (300 s by default) after the fragments writing it have finished, whichever comes first. Finishing with an error does not clear the unread blocks. `cancel_plan_fragment` is the BRPC the FE uses to cancel a query on a BE. #### The problem, and what it cost - **The local reader never looks at the client.** `get_arrow_batch` loops on 20 ms waits until a block arrives, the buffer is closed or it fails. It has no access to `ServerCallContext::is_cancelled()`, so a `DoGet` cancelled while the buffer is empty keeps a Flight server thread waiting until the query produces a block or ends. - **Closing a stream cancels nothing.** The readers do not override `RecordBatchReader::Close()`, and their destructor only logs. A client that stops reading (a `LIMIT` reached on the client side, a cancelled cursor, a dropped connection) leaves the query running and its unread blocks in the buffer. When the result does not fit in the buffer, the full buffer back-pressures the result sink and the fragments, with their memory, wait until the query times out; the unread blocks go only when the buffer is dropped. - **The remote reader cannot be interrupted.** It waits for `fetch_arrow_data` with `brpc::Join`, which returns only when the result BE answers or `arrow_flight_reader_brpc_controller_timeout_ms` (300 s by default) passes. - **An FE cancel leaves the results behind.** `cancel_plan_fragment` cancels the fragments but not the query's Flight buffers, so their unread blocks stay until the delayed cleanup 300 s later. - **A fetch for a removed buffer never completes.** When `fetch_arrow_data` cannot find the buffer, it logs and returns without running `done`, so the caller waits for its BRPC timeout instead of getting an error. #67644 fixes the same line. #### How this PR fixes it - **Bounded waits, cancellation checks in the reader.** `get_arrow_batch(result, eos)` waits at most 20 ms and then returns OK with no block and `eos = false`; `eos` now marks the real end. The local reader retries in a loop and checks `is_cancelled()` before each wait. `DoGetStatement` passes `context.is_cancelled()` to both readers. - **The reader closes itself.** `ArrowFlightBatchReaderBase` now implements `ReadNext` (which calls each reader's `ReadNextImpl`) and `Close()`. A stream counts as aborted when it is cancelled, closed before EOF, fails while reading, or is destroyed before EOF; a stream that reached EOF is never treated as aborted. - **Aborting a local stream cancels the query on this BE.** `ResultBufferMgr` keeps an index from query id to the query's Flight buffers, maintained under the buffer-map lock together with the map. `cancel_arrow_flight_buffers` removes every Flight buffer of the query from the map, then, outside the lock, fails each buffer (which wakes its waiting fetches and releases the back-pressure) and cancels the `QueryContext` it holds a weak reference to. Sibling buffers of the same query are cleared even after their `QueryContext` has expired. - **Remote waits can be interrupted.** The remote reader waits for its BRPC with a callback that checks `is_cancelled()` every 20 ms and calls `brpc::StartCancel` when the stream is gone. - **`cancel_plan_fragment` drops the Flight buffers of an aborted query.** It calls `cancel_arrow_flight_buffers` unless the reason is `LIMIT_REACH` or `FINISHED`, which can leave valid unread results that the client still has to read. - **`fetch_arrow_data` answers when the buffer is gone.** It completes the RPC with the lookup error. The line is identical to the one #67644 adds, so the two PRs merge in either order. Differences from branch-4.x: the two consistency checks between the index and the buffer map are `DORIS_CHECK` (an `if` and a `DCHECK` in 4.x), a null check on the reader's memory tracker that both constructors make unnecessary is not carried over, and the `ArrowFlightResultBlockBuffer` constructor moved to the .cpp so that `varrow_flight_result_writer.h`, which 13 TUs include, does not include `runtime/query_context.h`; only the .cpp needs its definition. The new lines also take a few clang-tidy fixes (the destructor calls `ArrowFlightBatchReaderBase::Close()` explicitly instead of a virtual call, `std::scoped_lock`), and the test starts its BRPC server inside `ScopedLeakCheckDisabler`, like the other BE tests that start one, because brpc registers process-lifetime built-in services on its first start. Not in this PR: the FE route. A remote reader's abort, and an abort on one result BE, still do not reach the query's other BEs; their fragments and buffers wait for the timeout as before. The follow-up adds `FrontendService.cancelFlightQuery` (same signature as branch-4.x), the BE broadcast to the FEs and the FE routing table, together with the FE unit test and the `test_flight_cancel_cleanup` regression test. | Situation | Before | After | |---|---|---| | Client cancels `DoGet` while the result buffer is empty (local reader) | Server thread waits until the query produces data or ends | Returns within 20 ms; the query is cancelled on this BE | | Client closes a local stream before EOF | Query keeps running, back-pressured until it times out if the result does not fit; unread blocks kept for minutes | Query cancelled on this BE; all of its Flight buffers there freed at once | | Client cancels while a remote fetch is pending | Waits for the result BE, up to 300 s | BRPC cancelled within 20 ms | | FE cancels the query (`cancel_plan_fragment`, abort reason) | Unread blocks kept until the delayed cleanup, 300 s later | Flight buffers freed at once; `LIMIT_REACH` / `FINISHED` still keep them | | `fetch_arrow_data` for a removed buffer | RPC never completes; caller waits for its timeout | Completes at once with an error | | Stream read to EOF, then closed | Nothing cancelled | Nothing cancelled | | New `DoGet` with the ticket of a stream aborted on its result BE | Reads the remaining blocks, or EOS | Fails: the buffer is gone | #### Results - `ArrowFlightCancelTest` (9 tests, new): early close and destruction free a back-pressured buffer and cancel the query; sibling buffers are freed after their `QueryContext` expired; EOF followed by close is not a cancellation; cancellation interrupts an empty-buffer wait; empty waits return for retry and do not end the stream; a conversion error cancels the query; a real Flight server sees a client-side `Cancel()` through `ServerCallContext` and cleans up. - `ArrowFlightRemoteCancelTest` (3 tests, new, against an in-process BRPC `PInternalService`): cancellation interrupts a pending `fetch_arrow_data`; an aborting `cancel_plan_fragment` frees the buffer and a following fetch completes with an error inside its 1 s timeout; `LIMIT_REACH` / `FINISHED` keep the unread results. - On a local single-BE cluster, branch-4.x's regression `test_flight_cancel_cleanup` (it aborts one endpoint after the first batch, by closing it and by an explicit cancel, and requires the query's pipelines to disappear within 20 s) fails on master with `Cancelled Flight query retained pipelines: [...: 1 pipeline fragment contexts in query ...]` and passes with this PR. The suite comes with the follow-up, because on a multi-BE cluster it also needs the FE route. The 20 `arrow_flight_sql_p0` suites and the 14 `external_table_p0/remote_doris` suites, which read results over Arrow Flight, pass before and after. - The 12 new tests plus `ResultBufferMgrTest` (6) and `ArrowResultBlockBufferTest` (4) pass 10 times in a row in one ASAN process. Ten single-point mutations of the change (dropping the buffer cleanup in `cancel_plan_fragment`, the `on_failure` call, the local cancel in `close()`, the cancellation check in the local read loop or in the BRPC wait, the sibling loop, the EOF flag, the `QueryContext` cancel, the cancel in the destructor, the `eos = false` on an empty wait) each turn their target tests red.

…t stream closes early ### What problem does this PR solve? Issue Number: None Related PR: apache#68630, apache#69945 Problem Summary: Port the FE route of branch-4.x apache#68630 to master, on top of apache#69945 (its BE-local half). With apache#69945 alone, a DoGet stream closed before EOF cancels its query only on the BE that holds the stream's results. A query whose results are spread over several BEs (parallel result sink, one endpoint per BE) kept running on the other BEs: master closes the coordinator when GetFlightInfo returns (apache#67504), so nothing sends them a cancel, and a single-fragment query has no exchange that would carry the cancellation between BEs. Their pipelines ran until the query finished or timed out. A remote reader (a DoGet served by a BE other than the result BE) could not cancel the result BE's query at all. The FE now keeps a route for each Flight query that leaves its results on the BEs: from each result id (the fragment instance id of an endpoint) to the query id and the BRPC addresses of all its BEs (FlightSqlQueryCancellation, registered by StmtExecutor before the coordinator is closed or deferred). The route belongs to the session's FlightSqlChannel and is dropped when the session's next request begins (FlightProtocolAdapter.beginRequest resets the channel), when the session is torn down (releaseSession closes the channel), when a cancellation succeeds, and at the latest execution_timeout + 5 s after registration. The new FrontendService RPC cancelFlightQuery(result_id) sends the existing cancel_plan_fragment, which older BEs understand and which does not use the Flight read pool, to every BE of the query and waits at most 1 s for all of them; when any BE fails, the route stays so that the caller can retry. A reader closed before EOF now asks the running FEs in turn: an FE without the route answers NOT_FOUND and the BE moves on to the next one, a failed attempt is retried once, and the transport is reopened after a thrift error. This runs on the DoGet thread, so with an unreachable FE one early close can take a few seconds (two attempts with a 2 s RPC timeout per FE). Compatibility: cancelFlightQuery has the same name and signature as on branch-4.1 and branch-4.2. An FE without it fails the call; the BE logs a warning for that FE and only the local cleanup of apache#69945 takes effect. A result BE without apache#69945 cancels the query but keeps its result buffer until its usual timeout. Differences from branch-4.x: the route is registered in master's `!isReturnResultFromLocal()` branch (apache#67900), before the `mustOutliveDispatch()` deferral, so deferred queries are routed as well; FlightProtocolAdapter needs no change because the route stays in FlightSqlChannel; the BE reader includes runtime/frontend_info.h itself (exec_env.h no longer includes it, apache#66400); the regression suite keeps its table at the end (AGENTS.md). ### Release note Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on every BE that runs it, not only on the BE that serves the stream. The FE and the BEs must both be upgraded. ### Check List (For Author) - Test: Unit Test / Regression test - FE UT: FlightSqlQueryCancellationTest (5, new), StmtExecutorTest and 15 other Flight session / coordinator test classes - BE UT: ArrowFlightCancelTest (9), ArrowFlightRemoteCancelTest (9, 6 new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4) - Regression: arrow_flight_sql_p0/test_flight_cancel_cleanup (new) and the rest of arrow_flight_sql_p0 on a local single-BE cluster; the cross-BE path is covered by the BE UT's in-process FE and remote BE - Behavior changed: Yes. A Flight result stream closed before EOF cancels its query on every BE of the query. - Does this need documentation: No Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

raghav-reglobe requested review from 924060929 and morrySnow as code owners September 3, 2026 15:21

morningman changed the title ~~[fix](arrow-flight) Bound idle Arrow Flight SQL sessions separately from wait_timeout~~ [fix](arrow-flight) Release a finished Flight query's coordinator instead of holding it until wait_timeout Sep 5, 2026

Raghav and others added 2 commits September 5, 2026 15:37

morningman added the dev/4.2.x label Sep 5, 2026

morningman force-pushed the fix-arrow-flight-session-idle-timeout branch from 212a026 to c662d85 Compare September 5, 2026 07:44

morningman approved these changes Sep 5, 2026

View reviewed changes

924060929 approved these changes Sep 7, 2026

View reviewed changes

morningman merged commit 23b5ee5 into apache:master Sep 7, 2026
29 of 31 checks passed

yiguolei added dev/4.1.5-merged and removed dev/4.2.x labels Sep 9, 2026

This was referenced Nov 2, 2026

[fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early #69945

Merged

[fix](arrow-flight) Cancel a Flight query on all its BEs when a result stream closes early #69996

Open

matanper mentioned this pull request Nov 3, 2026

[Bug] Arrow Flight SQL: audit query_time covers only FE dispatch #70145

Open

2 tasks done

Conversation

raghav-reglobe commented Sep 3, 2026 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

morningman commented Sep 5, 2026

Uh oh!

morningman commented Sep 5, 2026

Uh oh!

morningman commented Sep 5, 2026

Uh oh!

morningman commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

FE UT Coverage Report

Uh oh!

morningman commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

Uh oh!

morningman commented Sep 5, 2026

Local pipeline review — ✅ PASS

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

raghav-reglobe commented Sep 3, 2026 •

edited by morningman

Loading