[morningman]

What problem does this PR solve?

Issue Number: None

Related PR: #68630, #69945

Problem Summary:

Builds on #69945, now merged. This branch is rebased onto master 0eb5131db62 and holds seven commits: the port; its first review fix, which keeps a route past the session's next request; two fixes from the second review, which bound the BE's FE calls on stream teardown and count the BE addresses of the routes in the registry bound; and three from the third review, which drop the routes of a GetFlightInfo that fails, cancel through those routes its queries that no deferred coordinator cancels, and have the BE ask the FEs the master reaches first.

In short

This ports the FE half of branch-4.x's #68630 to master. With #69945, an Arrow Flight SQL stream that the client closes or cancels before its end cancels its query only on the BE that serves the stream, and only when that BE also holds the stream's results. A query whose results are spread over several BEs keeps running on all the other BEs until it finishes or times out. With this PR the FE remembers, for each Flight query that leaves its results on the BEs, which BEs run it; the BE that sees the aborted stream asks the FE, through the new FrontendService.cancelFlightQuery, to cancel the query on every one of them.

Background

• A Flight SQL query leaves its result on the BEs. With parallel result sink (the default), each BE that runs the result fragment holds one result buffer and GetFlightInfo returns one endpoint for it; the client reads each endpoint with its own DoGet stream.
• Since [fix](arrow-flight) Release a finished Flight query's coordinator instead of holding it until wait_timeout #67504 the FE closes the coordinator as soon as GetFlightInfo has returned (a query that still needs it, a batch-mode external-table scan or a remote Doris scan, is kept until the session's next request). After that, nothing on the FE cancels the query on behalf of an open stream.
• A DoGet is served by ArrowFlightBatchLocalReader when the result buffer is on the same BE, or by ArrowFlightBatchRemoteReader, which pulls the blocks from the result BE with the fetch_arrow_data BRPC.
• [fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early #69945 makes a reader that is closed, cancelled, fails or is destroyed before EOF cancel the query on its own BE when the buffer is local, and lets an aborting cancel_plan_fragment free the query's Flight buffers on the BE that receives it.

The problem, and what it cost

• An abort on one endpoint stays on that BE. A single-fragment query has no exchange between its BEs, so cancelling it on one BE does not reach the others, and the coordinator that could have sent them a cancel is already closed. When the client aborts one endpoint (a client-side LIMIT, a cancelled cursor, a dropped connection), the other BEs run their pipelines to the end and keep their unread results until the query's timeout or the delayed buffer cleanup; a result that does not fit in a buffer back-pressures the result sink until the query times out.
• A remote reader cannot cancel anything. [fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early #69945 interrupts its pending fetch, but the query on the result BE keeps running.
• A local reader whose query context has expired frees its own buffers but cannot tell the query's other BEs.

How this PR fixes it

• A route per Flight query on the FE. FlightSqlQueryCancellation maps each result id (the fragment instance id of an endpoint) to the query id and the BRPC addresses of all the query's BEs (Coordinator.getBackendBrpcAddresses(), from its per-BE tasks). StmtExecutor registers it when the query leaves its result on the BEs (!isReturnResultFromLocal()), before the coordinator is closed or deferred (FlightSqlQueryCancellation.register(ConnectContext, Coordinator)). The route belongs to no session. The client reads the results from the BEs, so it can still hold a stream of the query open when the session's next request begins or the session is torn down, and close it early afterwards. The BE also reports an abort from its DoGet thread, so even a client that closes a stream before it sends its next request can be overtaken by that request. A route goes when a cancellation through it succeeds, when the GetFlightInfo that registered it fails before answering with tickets, execution_timeout + 5 s after it was registered, or, oldest first and as a whole, when the registry would hold more than 100,000 result ids or 1,000,000 BE addresses. A route costs about 270 bytes plus about 30 bytes per BE address, and the result ids of a query share it, so the registry stays within about 55 MB however many BEs a query runs on. Without the session as a bound, these caps keep the registry from growing with the query rate times the timeout; a query whose route was pushed out ends at its timeout, as before this PR. The registry is an insertion-ordered map under a lock, and the cancellation RPCs are sent outside it.
• cancelFlightQuery(result_id), a new FrontendService RPC. It answers NOT_FOUND when the FE has no route for the id. Otherwise it sends the existing cancel_plan_fragment to every BE of the query, waits at most 1 s in total for their answers, and returns OK only when all of them succeeded; on any failure the route stays, so that the caller can retry. cancel_plan_fragment is understood by older BEs and runs on the light work pool, not on the Flight read pool, so it also gets through when the Flight pool is saturated or shut down.
• A GetFlightInfo that fails cancels its query. A GetFlightInfo can fail after its query ran, at the Arrow schema fetch or because a prepared statement's result schema no longer matches; no client can pull the results then. A deferred query is cancelled through its coordinator, which is still open, as before. Any other query has closed its coordinator, and nothing would cancel it before the BE's result buffer timer (execution_timeout + 5 s): its route now sends the cancellation to every BE of the query, without waiting for the answers, and the request's routes go. The prepared-statement check cancels its query through the executor and only drops the route. A request that fails before it begins, such as an empty query, finds the previous request's endpoints on the session and leaves their routes.
• The BE reports every abort. After the local cleanup of [fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early #69945, ArrowFlightBatchReaderBase::close() calls cancelFlightQuery on each running FE in turn, first the FEs the master reaches and then those it reports with process UUID 0. OK ends the loop; NOT_FOUND (an FE that did not plan the query) moves on to the next FE; any other failure is retried once, and the transport is reopened after a thrift error. These calls use FE connections of their own (ExecEnv::flight_cancel_frontend_client_cache, at most two kept per FE), all opened with a 2 s timeout, and acquire them without backing off; an FE that cannot be connected to is not retried. A stream read to EOF never calls the FE.
• The thrift method is appended to FrontendService with the same name, signature and comment as on branch-4.1 and branch-4.2.

Situation	Before (master with #69945)	After
Client aborts one endpoint of a query whose results are on several BEs	Only the BE holding that endpoint cancels the query; the other BEs run to the end and keep their unread results until the timeout or the delayed cleanup	The query is cancelled on every BE of the query
Client aborts a stream served by a remote reader	The pending fetch is interrupted; the query keeps running on the result BE	The query is cancelled on the result BE and on all other BEs
Client aborts a local stream after its query context expired	Only the local buffers are freed	The query's other BEs are cancelled too
FE that did not plan the query is asked	—	Answers NOT_FOUND; the BE asks the next FE
Stream read to EOF, then closed	Nothing cancelled	Nothing cancelled; the FE is not called

Upgrades and mixed versions. Both the FE that planned the query and the BE that serves the stream need this PR.

FE	Result BE	Effect of an abort
Without this PR	any	The BE gets an unknown-method error from cancelFlightQuery, logs a warning for that FE, and only the local cleanup of #69945 applies
With this PR	With #69945	The query is cancelled and its Flight buffers are freed on every BE
With this PR	Without #69945	The query's fragments are cancelled; that BE keeps its result buffer until its usual timeout

Cost. The FE calls run on the DoGet thread, including from the reader's destructor, after the local cleanup of #69945. A stopped FE stays on the BEs' list until it is dropped, reported with process UUID 0, and the FEs the master reaches are asked first, so a stopped FE delays an early close only when no reachable FE holds the route (the route expired or was pushed out, or the planning FE is itself unreachable): by about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect (thrift_connect_timeout_seconds), and 4 s when it accepts the connection and does not answer. An FE that hangs while the master still reaches it costs up to 4 s until the master's heartbeat to it fails. A GetFlightInfo that fails sends its cancellations without waiting for them. On the FE, cancelFlightQuery holds one thrift thread for at most 1 s, and each idle connection a BE keeps for these calls holds another. The route keeps only ids and addresses, not the coordinator, its scan state or its query queue slot, and the registry holds at most 100,000 result ids and 1,000,000 BE addresses, about 55 MB.

Differences from branch-4.x: the route is registered in master's !isReturnResultFromLocal() branch (#67900) instead of the ARROW_FLIGHT_SQL connect type branch, before the mustOutliveDispatch() deferral, so deferred queries are routed as well; a route does not belong to the session: branch-4.x drops it when the session's next request begins and when the session is torn down (FlightSqlChannel.reset()), so an abort reported after either finds no route and the other BEs run until the timeout, while here a route stays until its cancellation succeeds or it expires, and the registry is bounded instead; the BE reader includes runtime/frontend_info.h itself, because exec_env.h no longer includes it (#66400); the BE test reads and restores ExecEnv::_frontends through its unique_ptr; the FE test is JUnit 5 and tests that a route outlives the session's next request and its teardown, and the bound; the regression suite keeps its table at the end, as AGENTS.md asks; the two branches of the BE's FE loop got comments; the BE asks the FEs the master reaches first, where branch-4.x asks them in address order; and a GetFlightInfo that fails cancels through their routes its queries that were not deferred, where branch-4.x only finalizes the deferred executors and cancels nothing.

Results

• FE UT FlightSqlQueryCancellationTest (9 tests, new): a cancellation reaches every BE of the route after the coordinator and the selected endpoint's fragment are gone, and a successful one removes the route for all of its result ids; a BE that answers an error keeps the route, and the retry succeeds; routes expire execution_timeout + 5 s after registration; a route registered from a session survives the session's next request (beginRequest()) and its teardown (releaseProtocolSession()) and still cancels; the bound drops whole routes, oldest first, also when the BE addresses exceed it; a route whose result id a newer registration took over releases its addresses; an expired route leaves at the next registration; a request that fails before answering with tickets drops its routes as a whole, and cancels the queries no deferred coordinator cancelled on all their BEs without waiting for them.
• DorisFlightSqlProducerTest: a GetFlightInfo whose schema fetch fails cancels its query on both its BEs and drops its route; a deferred query is cancelled once, through its coordinator; an empty query leaves the previous request's route and cancels nothing. DorisFlightSqlSchemaTest#executionSchemaDriftExpiresHandleAndCleansResults now also requires the route of a prepared statement whose result schema changed to go.
• StmtExecutorTest#testCloudReplanDispatchProfileLifecycle now checks, for its two Flight outcomes (closed and deferred coordinator), that the dispatched query registered its route; its coordinator mock returns the query id and one BE address, and its exec() adds one endpoint, as the coordinator does.
• BE UT ArrowFlightRemoteCancelTest gets the FE half of branch-4.x's fixture back: an in-process thrift FE (FlightCancelCoordinator) that answers cancelFlightQuery by sending cancel_plan_fragment to an in-process BRPC result BE. The 9 new tests: a remote reader's close reaches the result BE; the cancellation also works when the Flight work pool rejects work; an FE error is retried once; an older result BE that only understands query cancellation gets it and keeps its buffer; a local reader whose query context has expired still reports the abort; a stream read to EOF does not call the FE; a close is not held up by a 60 s transport that another caller left in the shared FE cache to an FE whose first answer stalls, moves past an FE that refuses connections, and asks an FE that accepts connections and never answers, reported with process UUID 0 and sorting first, only after the planning FE (14 ms; 4.1 s in address order). CancellationInterruptsPendingBrpcFetch and FetchAfterCancellationStillCompletesRpc are back to branch-4.x's form, which needs the FE route ([fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early #69945 carried variants without it).
• 28 Flight session and coordinator FE test classes (297 tests) pass, with Checkstyle. The 4 BE suites (ArrowFlightCancelTest, ArrowFlightRemoteCancelTest, ResultBufferMgrTest, ArrowResultBlockBufferTest, 30 tests) pass 10 times in a row in one ASAN process.
• Mutations, each red in its target tests: on the FE, a failed cancellation drops the route; a successful one keeps it; routes never expire; StmtExecutor does not register; and, for the review fix, the session drops its routes again at its next request and teardown; the bound evicts the route just registered instead of the oldest one, or single result ids instead of whole routes; an expired route can still be cancelled; registration keeps expired routes. On the BE, the reader stops calling the FEs (the 7 tests that need the route turn red, the other 11 stay green); the reader does not retry. For the second review's fixes: on the BE, taking the connections from the shared cache and acquiring them with the default backoff (one round) turn the two new tests red, each its own (the close took 5.0 s and 7.4 s); with the reader as it was before these fixes they took 5.0 s and 14.7 s. On the FE, not releasing a route's addresses, releasing them per result id instead of with the last one, leaving the address bound out of the eviction, and keeping the addresses of the route an id leaves when a newer registration takes it over each turn their target test red. That an FE which cannot be connected to gets no second attempt has no test of its own: against an FE that refuses connections the second attempt costs about 0.1 s, and the difference only shows with an FE that does not answer the connect (3.1 s instead of 6.2 s). For the third review's fixes: on the FE, dropping the routes whether or not the request has begun, dropping single result ids instead of whole routes, cancelling deferred queries through their routes too, dropping the routes without cancelling, not handling the failure at all, waiting for the BEs' answers, and leaving the prepared-statement branch's route each turn their target tests red; on the BE, asking the FEs in address order turns the new test red (4.1 s).
• On a local single-BE cluster, arrow_flight_sql_p0/test_flight_cancel_cleanup (new; it aborts one endpoint after the first batch, by closing it, by an explicit cancel, and by closing it after the session ran its next request, and each time requires the query's pipelines to disappear from every BE within 20 s) passes, and the BE log shows the cancel_plan_fragment that the FE sent through the route for all three aborts. Before the review fix it showed it for one of the three: the third abort found no route, and so did the first, whose report came after the suite's next request (its first attempt met a stale connection after an FE restart; the retry reopened it and found no route). The other 20 arrow_flight_sql_p0 suites and the 14 external_table_p0/remote_doris suites pass. A single BE cannot show the cross-BE effect: there the BE serving the stream already cancels the query itself ([fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early #69945), so the suite passes without this PR too; on several BEs it needs this PR. Locally the cross-BE path is covered by the BE UT's in-process FE and remote result BE.
• Rebased onto master 0eb5131db62, after [refactor](coordinator) Unify query execution in Coordinator #69830 removed NereidsCoordinator and made its implementation the Coordinator: getBackendBrpcAddresses() is now a method of that class, with the body of the former NereidsCoordinator override (the legacy implementation went with the old class). git range-diff shows no other change to the first commit and none to the second. The FE and BE tests above ran on this base; master's doris_be_test does not build since [feature](be) Add session controls for Parquet page indexes #69971 (vparquet_writer_test.cpp), so the BE UT here ran with that test's one-line fix ([fix](test) Build vparquet_writer_test against the location-based Iceberg WriteInfo #70209) applied locally. The regression suites were not rerun on this base, including the arrow_flight_sql_p0 suites master added since 1adeb3a6801.

Release note

Arrow Flight SQL: cancelling or closing a result stream before its end now cancels the query on every BE that runs it, not only on the BE that holds the stream's results, and a GetFlightInfo that fails after its query ran cancels that query instead of leaving it to run until its timeout. This needs both the FE and the BEs to be upgraded.

Check List (For Author)

• Test: Unit Test / Regression test
  • FE UT: FlightSqlQueryCancellationTest (7, new), StmtExecutorTest (55) and 26 other Flight session / coordinator classes: 297 tests pass; fe-core Checkstyle 0 violations. Third review: FlightSqlQueryCancellationTest (9), DorisFlightSqlProducerTest (15), DorisFlightSqlSchemaTest (39) and 6 other Flight / coordinator classes, 149 tests pass on 35aece4ee7f, fe-core Checkstyle 0 violations
  • BE UT: ArrowFlightCancelTest (9), ArrowFlightRemoteCancelTest (11, 8 new), ResultBufferMgrTest (6), ArrowResultBlockBufferTest (4): 30 tests pass 10 times in a row (ASAN, macOS); 7 mutations (5 FE, 2 BE) each caught, 5 more on the FE for the first review fix, 6 more (2 BE, 4 FE) for the second, and 8 more (7 FE, 1 BE) for the third
  • build-support/check-build-hygiene.sh passes; clang-format 16 clean; clang-tidy reports nothing on the changed lines
  • Regression: arrow_flight_sql_p0 (21 suites, including the new test_flight_cancel_cleanup) and external_table_p0/remote_doris (14 suites) pass on a local single-BE cluster, also after the review fix; the cross-BE path is only covered by the UTs locally
  • After the rebase onto master 0eb5131db62: the FE and BE UT above; the regression suites were not rerun on this base
  • The third review's commits: the 9 FE classes above (149 tests) and the 21 Flight cancel BE tests (ArrowFlightCancelTest, ArrowFlightRemoteCancelTest, now 12 with 9 new) 10 times in a row, on 35aece4ee7f; clang-format, clang-tidy and the build hygiene check clean; the regression suites were not rerun
• Behavior changed: Yes. A Flight result stream closed, cancelled, failed or destroyed before EOF now cancels its query on every BE of the query, through the FE that planned it, also when the session has moved on to its next request or has been closed. A GetFlightInfo that fails after its query ran now also cancels that query on all its BEs.
• Does this need documentation: No

Original author: @Gabriel39

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morningman]

run buildall

[morningman]

/review

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.65% (2101/2741)
Line Coverage	65.98% (38776/58772)
Region Coverage	53.65% (36345/67743)
Branch Coverage	57.21% (11856/20722)

[github-actions]

Static exact-head review of PR 69996 completed after one review round. I traced the result-buffer query/index lifecycle, local and remote Arrow Flight reader cancellation, BRPC callback ownership, FE route registration and retries, Coordinator/Nereids endpoint identities, status filtering, fetch-RPC completion, and the appended thrift method across supported branch snapshots. One P1 finding is included: session reset or teardown can unregister a remote query's cancellation route while its DoGet is still active, so the later abort receives NOT_FOUND and leaves remote pipelines and result buffers running until timeout. The mixed-version older-BE behavior is documented by the PR and its test (fragment cancellation with normal buffer timeout), so it was not reported as a defect. No existing P0/P1 inline findings were present, and the supplied review-focus file contains no additional focus. This was a static review; no builds or tests were executed.

Review converged after 1 round(s).

[github-actions]

[P1] Keep the cancellation route alive while a DoGet is still active. Flight calls are not transport-serialized, so a client can start the next request or close the session while the previous stream is still reading; beginRequest()/teardown reaches this reset() and removes remoteResultIds. When that stream later closes, the BE's cancelFlightQuery gets NOT_FOUND and sends no cancelPipelineXPlanFragmentAsync to the remote result BEs, leaving their pipelines and result buffers running until the timeout. Track active DoGets or cancel before unregistering so reset cannot invalidate an in-flight stream's route.

[morningman]

Fixed in ba0c1d9926d. Agreed, and the local run shows the case is wider than overlapping streams: the BE reports an abort from its DoGet thread, so a client that closes a stream and goes straight to its next request can start that request before the report reaches the FE. In test_flight_cancel_cleanup, before this change, the report of the first abort came after the suite's health-check request and found no route.

A route now belongs to no session. FlightSqlChannel is back to its master form, and StmtExecutor registers through FlightSqlQueryCancellation.register(ConnectContext, Coordinator), which keeps nothing of the session. A route goes when its cancellation succeeds, when its query's execution timeout plus 5 s has passed, or, oldest first and as a whole, when the registry would hold more than 100,000 result ids. Without the session as a bound, that cap keeps the registry from growing with the query rate times the timeout. The registry is an insertion-ordered map instead of the Caffeine cache, whose size eviction can reject a new entry instead of evicting the oldest one.

Tests: routesOutliveTheSessionsNextRequestAndTeardown registers from a Flight session, runs beginRequest() and releaseProtocolSession(), and still cancels; boundDropsWholeRoutesOldestFirst covers the cap; test_flight_cancel_cleanup gains a mode that runs the session's next request before the abort. On a local cluster, the FE cancels all three aborts of that suite through the route; before the change it cancelled one of them.

[hello-stephen]

TPC-H: Total hot run time: 29312 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48324ff4c5a1d5e673746d456d5ee7b033855db5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17422	5381	5346	5346
q2	2121	331	288	288
q3	10034	1389	835	835
q4	4672	494	346	346
q5	7464	809	530	530
q6	205	201	168	168
q7	748	802	618	618
q8	9273	1335	1095	1095
q9	5513	4299	4260	4260
q10	6827	1305	1035	1035
q11	435	255	234	234
q12	635	405	294	294
q13	18088	3741	3066	3066
q14	284	277	250	250
q15	q16	729	717	658	658
q17	1193	777	593	593
q18	7018	6168	6085	6085
q19	1126	951	596	596
q20	399	337	230	230
q21	5932	2501	2508	2501
q22	403	356	284	284
Total cold run time: 100521 ms
Total hot run time: 29312 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6243	6256	6182	6182
q2	779	642	590	590
q3	5500	5799	5503	5503
q4	2220	2328	1720	1720
q5	5772	5709	5753	5709
q6	254	198	157	157
q7	2270	2075	1790	1790
q8	3110	2853	2548	2548
q9	7505	7441	7440	7440
q10	3765	3653	3176	3176
q11	607	443	417	417
q12	743	713	555	555
q13	3357	3631	3053	3053
q14	285	296	273	273
q15	q16	690	722	645	645
q17	7883	7198	7112	7112
q18	13006	12199	12803	12199
q19	939	824	842	824
q20	2171	2183	1937	1937
q21	6006	5038	5093	5038
q22	559	487	423	423
Total cold run time: 73664 ms
Total hot run time: 67291 ms

[hello-stephen]

TPC-DS: Total hot run time: 151636 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48324ff4c5a1d5e673746d456d5ee7b033855db5, data reload: false

query5	4321	612	466	466
query6	428	208	191	191
query7	4803	441	231	231
query8	320	160	146	146
query9	8782	3205	3175	3175
query10	504	309	260	260
query11	6049	3849	3587	3587
query12	147	92	88	88
query13	1258	442	310	310
query14	6113	3952	3682	3682
query14_1	3416	3439	3422	3422
query15	204	208	180	180
query16	964	468	441	441
query17	854	663	542	542
query18	2424	430	342	342
query19	207	168	138	138
query20	85	84	86	84
query21	211	125	108	108
query22	13058	12989	12833	12833
query23	13525	12820	12340	12340
query23_1	12559	12493	12634	12493
query24	7310	830	499	499
query24_1	469	494	486	486
query25	504	399	337	337
query26	1274	255	135	135
query27	2787	436	271	271
query28	4568	2028	2022	2022
query29	1539	573	442	442
query30	297	218	179	179
query31	1005	922	824	824
query32	141	86	90	86
query33	509	294	236	236
query34	886	840	510	510
query35	725	751	655	655
query36	814	809	705	705
query37	115	97	83	83
query38	1764	1731	1710	1710
query39	789	782	737	737
query39_1	741	728	747	728
query40	219	113	96	96
query41	67	64	63	63
query42	86	81	82	81
query43	340	362	307	307
query44	1287	685	694	685
query45	180	176	165	165
query46	834	970	561	561
query47	2707	2909	2868	2868
query48	315	301	215	215
query49	575	391	290	290
query50	697	274	213	213
query51	10459	10731	10641	10641
query52	77	76	69	69
query53	190	202	151	151
query54	233	193	180	180
query55	78	66	62	62
query56	236	213	205	205
query57	1647	1610	1581	1581
query58	287	262	264	262
query59	2137	2095	1997	1997
query60	303	248	214	214
query61	152	150	145	145
query62	382	328	286	286
query63	189	153	159	153
query64	2882	1067	887	887
query65	3328	3267	3259	3259
query66	1782	434	311	311
query67	20262	19980	20172	19980
query68	3059	1010	620	620
query69	401	301	261	261
query70	855	774	768	768
query71	295	235	215	215
query72	2831	2603	2319	2319
query73	508	518	299	299
query74	4571	4519	4324	4324
query75	2332	2276	1924	1924
query76	2254	1016	572	572
query77	339	372	266	266
query78	9910	9690	9173	9173
query79	1172	852	544	544
query80	1219	447	348	348
query81	582	316	263	263
query82	564	132	105	105
query83	303	196	182	182
query84	318	123	106	106
query85	863	424	376	376
query86	395	244	235	235
query87	1996	1968	1831	1831
query88	3806	2771	2769	2769
query89	323	279	245	245
query90	1851	166	162	162
query91	166	161	128	128
query92	95	89	88	88
query93	1102	983	586	586
query94	634	312	273	273
query95	554	329	296	296
query96	703	487	239	239
query97	2471	2405	2299	2299
query98	163	153	147	147
query99	781	762	661	661
Total cold run time: 235695 ms
Total hot run time: 151636 ms

[hello-stephen]

ClickBench: Total hot run time: 25.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48324ff4c5a1d5e673746d456d5ee7b033855db5, data reload: false

query1	0.01	0.00	0.01
query2	0.14	0.10	0.08
query3	0.33	0.11	0.12
query4	1.63	0.13	0.12
query5	0.26	0.25	0.25
query6	1.16	0.65	0.65
query7	0.04	0.01	0.01
query8	0.09	0.07	0.06
query9	0.44	0.36	0.37
query10	0.58	0.59	0.56
query11	0.32	0.19	0.18
query12	0.32	0.18	0.19
query13	0.49	0.52	0.51
query14	0.93	0.94	0.98
query15	0.66	0.58	0.57
query16	0.35	0.35	0.36
query17	0.96	1.01	1.02
query18	0.30	0.27	0.27
query19	1.95	1.91	1.84
query20	0.02	0.02	0.02
query21	15.39	0.34	0.31
query22	4.92	0.13	0.12
query23	15.87	0.48	0.29
query24	2.34	0.51	0.40
query25	0.15	0.11	0.10
query26	0.75	0.26	0.21
query27	0.11	0.10	0.10
query28	3.38	0.82	0.39
query29	12.60	4.34	3.39
query30	0.34	0.17	0.14
query31	2.75	0.59	0.32
query32	3.24	0.59	0.48
query33	2.89	3.01	3.03
query34	15.89	4.08	3.43
query35	3.36	3.38	3.37
query36	0.58	0.53	0.49
query37	0.11	0.09	0.09
query38	0.07	0.06	0.07
query39	0.06	0.06	0.05
query40	0.18	0.17	0.16
query41	0.10	0.05	0.06
query42	0.06	0.05	0.05
query43	0.07	0.05	0.05
Total cold run time: 96.19 s
Total hot run time: 25.13 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 88.89% (104/117) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	67.35% (32120/47692)
Line Coverage	52.16% (337759/647582)
Region Coverage	47.68% (274495/575710)
Branch Coverage	49.48% (129290/261315)

[morningman]

Addressed review round 1 in ba0c1d9926d, a new commit on top of the branch:

• 4443420978: cancellation routes no longer belong to the session, so an abort reported after the session's next request or its teardown still cancels the query on every BE. The registry is bounded at 100,000 result ids and drops the oldest routes first. Details are in the thread.

The description is updated to match: the route's lifecycle, the differences from branch-4.x, and the results.

[morningman]

run buildall

[morningman]

/review

[hello-stephen]

TPC-H: Total hot run time: 28707 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba0c1d9926d939fdfb0e8514a696706da72c9999, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17638	5129	5044	5044
q2	2107	317	288	288
q3	10214	1400	822	822
q4	4679	472	344	344
q5	7437	791	528	528
q6	180	182	150	150
q7	768	801	600	600
q8	9266	1367	1068	1068
q9	5467	4203	4179	4179
q10	6821	1285	1034	1034
q11	432	240	229	229
q12	644	400	279	279
q13	18118	3607	3059	3059
q14	283	277	248	248
q15	q16	714	710	645	645
q17	1179	762	571	571
q18	6970	6205	6094	6094
q19	1097	947	591	591
q20	379	331	234	234
q21	5193	2477	2415	2415
q22	379	331	285	285
Total cold run time: 99965 ms
Total hot run time: 28707 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5995	5915	6070	5915
q2	712	617	581	581
q3	5678	5801	5369	5369
q4	2028	2190	1559	1559
q5	5641	5734	5667	5667
q6	234	180	140	140
q7	2223	2074	1775	1775
q8	3011	2621	2596	2596
q9	7999	7788	7418	7418
q10	3637	3566	3144	3144
q11	574	407	384	384
q12	668	653	512	512
q13	3348	3626	3038	3038
q14	292	280	255	255
q15	q16	669	691	613	613
q17	7681	7014	6882	6882
q18	12951	12245	12890	12245
q19	896	778	772	772
q20	2186	2132	1906	1906
q21	5539	4564	4646	4564
q22	500	455	414	414
Total cold run time: 72462 ms
Total hot run time: 65749 ms

[github-actions]

Static exact-head review of PR 69996 completed after one review round at ba0c1d9. I reviewed all 18 changed paths, including the FE route registry and Coordinator/Nereids backend identity publication, BE local and remote Arrow Flight reader cancellation and BRPC callback lifetime, query-wide result-buffer bookkeeping and mixed-version cancel handling, the appended FrontendService thrift method across branch-3.1/4.0/4.1/4.2 snapshots, and the added unit and regression tests. The prior P1 inline finding 4443420978 was independently revalidated as fixed at this head: FlightSqlChannel no longer owns or unregisters the route, and registration is session-independent; it is not reposted and is omitted from existing_blocking_comment_ids. No additional actionable defect was found. The supplied review-focus file has no additional focus. This was a static review; no builds or tests were executed (tests were inspected only).

Review converged after 1 round(s).

Review converged after 1 round(s).

[hello-stephen]

TPC-DS: Total hot run time: 151724 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba0c1d9926d939fdfb0e8514a696706da72c9999, data reload: false

query5	4318	616	441	441
query6	428	228	195	195
query7	4817	461	228	228
query8	312	169	164	164
query9	8818	3217	3224	3217
query10	495	307	254	254
query11	5952	3812	3583	3583
query12	152	96	89	89
query13	1265	447	321	321
query14	6161	3922	3692	3692
query14_1	3466	3456	3421	3421
query15	207	203	191	191
query16	949	482	457	457
query17	852	672	539	539
query18	2407	420	317	317
query19	204	182	145	145
query20	89	86	83	83
query21	209	134	113	113
query22	12923	13131	12800	12800
query23	13685	13007	12508	12508
query23_1	12714	12612	12598	12598
query24	7382	828	481	481
query24_1	487	491	508	491
query25	542	427	373	373
query26	1246	264	146	146
query27	2771	437	275	275
query28	4624	2070	2072	2070
query29	1590	605	467	467
query30	298	228	183	183
query31	1014	926	818	818
query32	146	96	92	92
query33	530	308	246	246
query34	899	865	524	524
query35	739	759	644	644
query36	830	815	727	727
query37	130	105	100	100
query38	1757	1733	1671	1671
query39	809	796	753	753
query39_1	745	732	744	732
query40	226	120	99	99
query41	73	73	70	70
query42	86	83	83	83
query43	347	368	310	310
query44	1319	714	716	714
query45	199	193	170	170
query46	833	959	584	584
query47	2872	2979	2616	2616
query48	303	313	214	214
query49	574	391	287	287
query50	663	278	217	217
query51	10786	10499	10417	10417
query52	74	84	69	69
query53	186	212	153	153
query54	228	187	197	187
query55	75	66	64	64
query56	237	207	196	196
query57	1662	1621	1545	1545
query58	298	271	265	265
query59	2134	2210	1930	1930
query60	283	245	267	245
query61	150	144	150	144
query62	390	334	286	286
query63	190	169	156	156
query64	2748	919	768	768
query65	3336	3293	3292	3292
query66	1777	418	302	302
query67	20095	20139	20075	20075
query68	2942	995	620	620
query69	387	275	251	251
query70	861	796	825	796
query71	300	236	218	218
query72	2720	2589	2295	2295
query73	522	535	297	297
query74	4617	4501	4353	4353
query75	2357	2285	1913	1913
query76	2297	1025	609	609
query77	351	388	277	277
query78	9808	9831	9241	9241
query79	976	912	541	541
query80	1268	435	337	337
query81	574	333	280	280
query82	572	138	107	107
query83	281	200	188	188
query84	314	120	101	101
query85	1138	418	389	389
query86	361	242	236	236
query87	2018	2019	1781	1781
query88	3813	2805	2783	2783
query89	330	282	244	244
query90	1847	174	166	166
query91	163	155	125	125
query92	99	79	78	78
query93	1022	1002	592	592
query94	569	327	279	279
query95	581	390	317	317
query96	689	502	240	240
query97	2437	2437	2312	2312
query98	165	159	150	150
query99	770	762	647	647
Total cold run time: 236019 ms
Total hot run time: 151724 ms

[hello-stephen]

ClickBench: Total hot run time: 24.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba0c1d9926d939fdfb0e8514a696706da72c9999, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.06	0.05
query3	0.26	0.08	0.08
query4	1.61	0.11	0.11
query5	0.23	0.24	0.24
query6	1.17	0.67	0.67
query7	0.04	0.00	0.00
query8	0.05	0.04	0.04
query9	0.41	0.36	0.35
query10	0.57	0.60	0.61
query11	0.20	0.14	0.15
query12	0.19	0.14	0.15
query13	0.49	0.47	0.50
query14	1.03	1.02	1.01
query15	0.64	0.61	0.60
query16	0.35	0.36	0.34
query17	1.18	1.13	1.10
query18	0.20	0.19	0.19
query19	2.11	2.05	2.03
query20	0.02	0.01	0.02
query21	15.50	0.20	0.13
query22	4.92	0.06	0.05
query23	16.26	0.32	0.12
query24	2.70	0.59	0.38
query25	0.11	0.05	0.05
query26	1.59	0.22	0.16
query27	0.04	0.04	0.04
query28	3.58	0.81	0.34
query29	12.63	4.18	3.30
query30	0.28	0.13	0.10
query31	2.77	0.56	0.33
query32	3.22	0.61	0.50
query33	3.17	3.20	3.20
query34	15.50	4.02	3.39
query35	3.34	3.34	3.33
query36	0.70	0.52	0.48
query37	0.10	0.07	0.08
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 97.67 s
Total hot run time: 24.43 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 88.89% (104/117) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	67.35% (32120/47692)
Line Coverage	52.15% (337738/647582)
Region Coverage	47.65% (274353/575710)
Branch Coverage	49.46% (129237/261315)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 88.89% (104/117) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.67% (35886/46205)
Line Coverage	62.99% (403481/640515)
Region Coverage	59.58% (341391/573025)
Branch Coverage	60.73% (158129/260370)

[morningman]

run buildall

[morningman]

/review

[github-actions]

Static review of head 8c4c7d0: two new P2 findings concern synchronous FE transport waits during stream teardown and the cancellation registry's unbudgeted backend-address snapshots. No new P0/P1 finding was confirmed.

The existing P1 thread 4443420978 is fixed on this head: route ownership is independent of FlightSqlChannel/session cleanup, and the added test covers cancellation after both the next request and session teardown. Its ID is not retained as blocking, and that finding is not reposted. No additional user focus was supplied.

End-to-end flow inspected: GetFlightInfo dispatches BE fragments -> snapshots result IDs, query ID and all BE addresses -> closes or defers the coordinator -> publishes endpoint tickets -> an early DoGet close reports the result ID to the FE -> the FE sends the existing query-level cancel RPC to every captured BE -> the BE light work pool cancels query pipelines and Flight buffers. EOF suppresses the report.

Critical checkpoint conclusions:

• Goal and proof: query-wide cancellation is wired before endpoint publication, including remote readers and completed local fragments. Added FE/BE unit tests and regression modes cover the intended behavior; the two inline findings identify remaining resource-handling defects.
• Scope and clarity: changes are focused on cancellation routing, two coordinator address accessors, dispatch registration, the service method and associated tests; existing cancellation RPCs are reused.
• Concurrency and locks: registry map operations share one monitor; immutable snapshots and conditional removal protect concurrent cancellation/replacement. RPC dispatch/waits occur outside that monitor. Buffer locks are released before query cancellation, and the light work pool avoids a Flight-pool callback cycle. The synchronous transport wait is reported inline.
• Lifecycle and initialization: routes retain IDs/addresses rather than sessions, coordinators, scan state or queue slots; normal cancellation removes whole routes. No new production cross-translation-unit initialization dependency. Registry retention cost is reported inline.
• Configuration: no settings added; execution timeout is captured per route. Existing Thrift timeout/retry defaults interact with the new synchronous caller as described inline.
• Compatibility: no storage format or existing function signature changed. Old FEs can reject the additive method and retain baseline local cleanup; older BEs understand query cancellation but retain their historical buffer-cleanup behavior.
• Thrift/protobuf: HEAD and live branch-3.1, 4.0, 4.1 and 4.2 definitions inspected. The method is absent on 3.1/4.0 and has the same signature on 4.1/4.2; referenced TUniqueId/TStatus field IDs and types match. No enum values or required struct fields added.
• Parallel paths: old and Nereids coordinators capture all query BEs; CloudCoordinator inheritance, parallel/query-ID and serial/instance-ID tickets, deferred scans, local/remote readers and expired local contexts were traced. Flight disables the separate short-circuit point-query path.
• Conditional checks and errors: NOT_FOUND advances FE discovery; failed cancellation preserves the route for retry; EOF avoids cancelling successful streams. Reopening discards incomplete-response transports. No additional confirmed failure-propagation defect.
• Test coverage: FE tests cover fan-out, application errors, expiry, session independence and whole-route FIFO eviction. BE tests cover remote cleanup, rejected Flight work, retry, old BEs, expired contexts, pending fetch and EOF. Regression checks every live BE for each query in close/cancel/next-request modes. Tests omit inherited transport timeouts/unreachable FEs and multi-BE route-weight accounting, the scenarios raised inline.
• Test results: no expected-output artifacts changed; assertions and mocked responses inspected. No builds, tests, benchmarks, formatting checks or runtime validation were executed in this static-only review. Author-reported runs were not independently reproduced; a single-BE run cannot demonstrate cross-BE cleanup.
• Observability: failures carry query/result IDs, FE/backend addresses or status; existing backend cancellation logging supports tracing. No additional correctness issue in logging was confirmed.
• Transactions and persistence: no EditLog, visibility-version, MoW or durable metadata change. Routes are ephemeral; FE restart loses them and existing BE timeouts remain the fallback.
• Data writes and crash behavior: production changes cancel query execution and buffers, without changing data-write atomicity. Regression writes fixture data only. No new lock leak found; the retained-heap risk is reported inline.
• FE/BE values: the new service passes the ticket's existing result ID; the route translates it to the actual query ID used by the existing backend cancellation RPC in both sink modes.
• Performance and memory: registration snapshots result IDs and all participating BE addresses; eviction removes whole routes. Response waits share one deadline after dispatch. The two inline comments cover the verified timeout and heap-budget problems.
• Other issues and final recheck: multi-statement rejection, retry ID replacement, schema-fetch failure, EOF, mixed-version and cancellation callback paths were rechecked. FIFO expiry behind a live longer-timeout route does not permit expired cancellation because lookup rechecks its deadline; no separate finding was established.

Review converged in 2 rounds. All 11 changed files, assigned checkpoint scopes, existing threads, candidate dispositions and risk focuses were covered; all latest-round reviewers returned NO_NEW_VALUABLE_FINDINGS on the frozen payload. No unresolved static-review scope remains. Runtime validation and workload/heap measurements remain outside this review's authorized scope.

Review converged after 2 round(s).

[github-actions]

[P2] Bound the actual FE call on stream teardown

The 2000 ms argument is applied only when ClientCacheHelper::_create_client creates a transport; a cache hit keeps its previous socket timeout. The shared frontend_client_cache is also used by profile reporting with the default 60000 ms timeout (runtime_query_statistics_mgr.cpp), so an aborted DoGet can inherit a 60-second wait on an unresponsive FE before this loop reaches the FE that owns the route. ClientConnection also defaults to three acquisition attempts with 1+2+4-second backoffs inside each outer attempt, adding at least 14 seconds even on immediate connection refusal. These waits now run in Close() and the reader destructor, occupying the stream thread and delaying cancellation of the remote producers/buffers. Give this path an actual connection/RPC deadline, including reused transports and acquisition retries, or perform the reporting outside stream teardown; cover a long-timeout cached connection and an unreachable FE in the tests.

[morningman]

Fixed in 152c56c38cc. Agreed, and the cache worked against this call in both directions. A reused transport keeps whatever its opener set: profile and status reports open theirs with thrift_rpc_timeout_ms (60 s), and the information_schema scanners open theirs to the master FE with the query's execution timeout (900 s by default). The other way round, this call's 2 s transports, and the 1 s ones that reopen(1000) made after any thrift error (an older FE's unknown-method reply included), went back to the shared cache, where fetchSplitBatch, whose FE side holds the call for up to fetch_splits_max_wait_time (1 s by default), or a status report could take them. An FE that has stopped also stays on every BE's list: the master sends it with process UUID 0, which get_running_frontends() counts as running, so when it sorts before the FE that planned the query, every early close on every BE waits for it.

The cancellation now keeps FE connections of its own (ExecEnv::flight_cancel_frontend_client_cache, at most two kept per FE), as MasterServerClient does, so each transport it uses was opened with its 2 s timeout and none reaches another caller. Acquiring a connection does not back off (max_retries = 0), a transport is reopened with the same 2 s, and an FE that cannot be connected to gets no second attempt, since neither reason for the retry applies to it. An FE ahead of the planning one now costs an early close about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect (thrift_connect_timeout_seconds), and 4 s when it accepts the connection and does not answer. The local cleanup of #69945 still runs first. The Cost paragraph of the description understated this wait and is corrected.

Tests: CloseIgnoresLongTimeoutSharedTransport leaves a 60 s transport to an FE whose first answer stalls for 5 s in the shared cache and requires the early close to finish within 4 s, through the retry; CloseMovesPastUnreachableFrontend puts an FE that refuses connections first and requires the close to reach the planning FE within 2 s. Before this change they took 5.0 s and 14.7 s.

[github-actions]

[P2] Include backend address snapshots in the registry budget

results.size() counts distinct result IDs, but in the default parallel-result mode every endpoint of a query uses the same query ID, so registration consumes one capacity unit while the Route deep-copies every participating BE address. Normal EOF never unregisters the route. For example, on a 1000-BE cluster, about 111 completed distributed Flight queries/sec during the default 900-second timeout can leave roughly 100000 routes retaining 100 million address objects plus their list references. That can consume gigabytes of FE heap despite the claimed tens-of-MB bound and despite all streams having finished. A distributed aggregate with a single result endpoint has the same mismatch. Account for retained backend addresses in the eviction weight, or otherwise bound/share those snapshots, while preserving whole-route eviction and the session-independent lifetime.

[morningman]

Fixed in 1c9a89481ed. Agreed. The result ids of a query already share one route and its address list, but the bound counted only the ids, and in parallel result mode a query has one. The registry now also bounds the BE addresses that its routes hold, at 1,000,000 (about 30 MB), and drops whole routes, oldest first, while either bound is exceeded; a route's addresses are released when its last result id leaves, also when a newer registration takes the id over. Under the old bound, a route held about 3 KB with 100 BEs and 29 KB with 1,000, so a full registry held about 0.3 GB and 2.9 GB; now it stays within about 55 MB whatever the cluster size. Large clusters give up retention instead: with 1,000 BEs the registry keeps the routes of the last 1,000 queries, about 9 s of them at 110 queries/s, and a query whose route was pushed out ends at its timeout, as before this PR. Dropping a route when its streams reach EOF would need the BEs to report every EOF to the FE as well, one more FE call per endpoint and query; I left that out.

Tests: boundCountsRetainedBackends requires a registration whose addresses exceed the bound to drop the oldest route as a whole, and reregisteredResultReleasesOldRouteBackends requires a route whose result id another registration took over to release its addresses once.

[hello-stephen]

TPC-H: Total hot run time: 28824 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c4c7d032c3b5bf179b79b50fa9d177dcd92d96f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17604	5201	5050	5050
q2	2164	302	249	249
q3	10321	1386	832	832
q4	4727	467	337	337
q5	7766	796	537	537
q6	199	185	150	150
q7	783	793	588	588
q8	10210	1278	1140	1140
q9	5631	4221	4186	4186
q10	6885	1312	1050	1050
q11	433	242	233	233
q12	678	399	277	277
q13	18178	3805	3098	3098
q14	278	281	254	254
q15	q16	739	726	659	659
q17	3352	767	582	582
q18	7036	6177	6135	6135
q19	1687	965	596	596
q20	418	486	282	282
q21	5755	2313	2454	2313
q22	380	330	276	276
Total cold run time: 105224 ms
Total hot run time: 28824 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6024	6128	5976	5976
q2	712	597	568	568
q3	5624	5844	5355	5355
q4	2012	2148	1577	1577
q5	5889	5666	5584	5584
q6	242	189	143	143
q7	2327	1984	1781	1781
q8	3109	2685	2632	2632
q9	7883	7496	7473	7473
q10	3628	3567	3153	3153
q11	585	425	406	406
q12	666	653	492	492
q13	3325	3651	3065	3065
q14	277	298	273	273
q15	q16	659	664	620	620
q17	7784	7081	6929	6929
q18	13042	12226	12973	12226
q19	975	777	766	766
q20	2136	2132	1932	1932
q21	5650	4644	4659	4644
q22	493	449	419	419
Total cold run time: 73042 ms
Total hot run time: 66014 ms

[hello-stephen]

TPC-DS: Total hot run time: 151469 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c4c7d032c3b5bf179b79b50fa9d177dcd92d96f, data reload: false

query5	4320	582	443	443
query6	434	214	191	191
query7	4806	441	225	225
query8	318	173	158	158
query9	8767	3140	3220	3140
query10	488	319	256	256
query11	6072	3862	3564	3564
query12	140	91	86	86
query13	1255	451	343	343
query14	6085	3916	3648	3648
query14_1	3420	3453	3382	3382
query15	207	201	198	198
query16	959	434	397	397
query17	861	681	538	538
query18	2432	437	311	311
query19	195	159	148	148
query20	88	84	93	84
query21	216	125	109	109
query22	13023	13080	12828	12828
query23	13549	12871	12418	12418
query23_1	12628	12480	12566	12480
query24	7375	827	474	474
query24_1	480	486	495	486
query25	549	429	359	359
query26	1283	262	141	141
query27	2777	448	277	277
query28	4614	2041	2027	2027
query29	1560	602	459	459
query30	295	231	194	194
query31	1021	939	822	822
query32	148	100	90	90
query33	541	321	251	251
query34	904	856	510	510
query35	739	763	647	647
query36	820	825	726	726
query37	125	105	85	85
query38	1763	1728	1658	1658
query39	791	782	744	744
query39_1	748	721	745	721
query40	228	118	102	102
query41	72	72	68	68
query42	87	86	83	83
query43	347	369	312	312
query44	1342	700	712	700
query45	192	180	171	171
query46	891	989	564	564
query47	2795	3019	2762	2762
query48	297	304	231	231
query49	581	414	291	291
query50	670	281	216	216
query51	10689	10472	10556	10472
query52	77	76	69	69
query53	184	207	153	153
query54	230	188	188	188
query55	69	68	62	62
query56	231	219	206	206
query57	1667	1625	1555	1555
query58	290	270	258	258
query59	2155	2230	1998	1998
query60	271	240	224	224
query61	152	150	149	149
query62	392	339	282	282
query63	188	156	160	156
query64	2733	987	788	788
query65	3337	3283	3296	3283
query66	1756	426	323	323
query67	20351	20255	20162	20162
query68	3396	1024	617	617
query69	385	293	244	244
query70	839	772	790	772
query71	300	241	197	197
query72	3122	2630	2283	2283
query73	530	536	296	296
query74	4687	4529	4369	4369
query75	2373	2283	1914	1914
query76	2390	993	593	593
query77	359	377	275	275
query78	9746	9634	9168	9168
query79	974	856	531	531
query80	1052	421	340	340
query81	575	321	268	268
query82	582	136	111	111
query83	304	203	180	180
query84	317	129	98	98
query85	900	422	382	382
query86	353	237	233	233
query87	2018	1948	1798	1798
query88	3779	2784	2756	2756
query89	330	283	252	252
query90	1798	174	167	167
query91	168	158	128	128
query92	100	90	91	90
query93	1009	1020	582	582
query94	534	329	263	263
query95	580	338	381	338
query96	676	490	235	235
query97	2463	2424	2327	2327
query98	159	151	158	151
query99	778	765	664	664
Total cold run time: 236434 ms
Total hot run time: 151469 ms

[hello-stephen]

ClickBench: Total hot run time: 24.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c4c7d032c3b5bf179b79b50fa9d177dcd92d96f, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.05	0.05
query3	0.26	0.08	0.08
query4	1.60	0.10	0.10
query5	0.24	0.24	0.24
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.06	0.03	0.03
query9	0.41	0.33	0.34
query10	0.58	0.57	0.62
query11	0.20	0.14	0.14
query12	0.18	0.15	0.14
query13	0.48	0.48	0.50
query14	1.02	1.02	1.02
query15	0.64	0.61	0.61
query16	0.36	0.35	0.34
query17	1.13	1.19	1.16
query18	0.20	0.19	0.17
query19	2.10	2.04	1.93
query20	0.02	0.01	0.01
query21	15.53	0.18	0.13
query22	4.91	0.06	0.05
query23	16.28	0.32	0.12
query24	2.90	0.57	0.35
query25	0.12	0.06	0.05
query26	0.83	0.22	0.16
query27	0.06	0.05	0.05
query28	3.41	0.78	0.36
query29	12.50	4.13	3.20
query30	0.29	0.14	0.09
query31	2.78	0.54	0.33
query32	3.22	0.59	0.50
query33	3.19	3.20	3.28
query34	15.54	3.97	3.38
query35	3.34	3.34	3.33
query36	0.56	0.45	0.48
query37	0.13	0.08	0.08
query38	0.05	0.05	0.04
query39	0.04	0.04	0.04
query40	0.18	0.16	0.16
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.82 s
Total hot run time: 24.25 s

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 79.17% (19/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.79% (35970/46238)
Line Coverage	63.16% (404896/641019)
Region Coverage	59.79% (342980/573629)
Branch Coverage	60.87% (158718/260731)

[hello-stephen]

TPC-DS: Total hot run time: 151549 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35aece4ee7fc8214ebeb40623db399ca1e8605ca, data reload: false

query5	4306	587	449	449
query6	427	218	200	200
query7	4807	446	255	255
query8	318	165	151	151
query9	8775	3237	3246	3237
query10	481	309	257	257
query11	6012	3853	3584	3584
query12	153	93	88	88
query13	1253	450	333	333
query14	6069	4007	3712	3712
query14_1	3467	3412	3496	3412
query15	209	201	192	192
query16	964	473	439	439
query17	872	676	545	545
query18	2425	433	315	315
query19	198	171	139	139
query20	86	84	82	82
query21	208	133	111	111
query22	13119	13146	12925	12925
query23	13442	12835	12525	12525
query23_1	12659	12452	12574	12452
query24	7159	870	490	490
query24_1	483	496	520	496
query25	508	392	352	352
query26	1260	257	141	141
query27	2760	432	278	278
query28	4516	2018	2028	2018
query29	1527	577	446	446
query30	293	222	183	183
query31	1014	924	828	828
query32	146	91	94	91
query33	528	311	232	232
query34	873	864	509	509
query35	734	753	671	671
query36	815	816	727	727
query37	130	102	83	83
query38	1766	1728	1679	1679
query39	785	782	764	764
query39_1	731	734	732	732
query40	219	114	92	92
query41	68	63	68	63
query42	84	83	81	81
query43	347	359	312	312
query44	1316	701	708	701
query45	181	173	164	164
query46	839	969	578	578
query47	3053	2736	2463	2463
query48	308	304	223	223
query49	570	404	296	296
query50	688	308	225	225
query51	10624	10540	10422	10422
query52	82	85	70	70
query53	187	214	155	155
query54	233	207	191	191
query55	73	72	66	66
query56	216	224	221	221
query57	1650	1616	1560	1560
query58	294	259	256	256
query59	2138	2161	1975	1975
query60	323	239	224	224
query61	151	156	147	147
query62	390	331	282	282
query63	189	162	170	162
query64	2676	956	777	777
query65	3357	3289	3283	3283
query66	1761	413	298	298
query67	20145	20270	19903	19903
query68	3034	981	616	616
query69	390	284	257	257
query70	864	812	789	789
query71	313	230	211	211
query72	2739	2612	2352	2352
query73	511	528	303	303
query74	4636	4543	4349	4349
query75	2351	2306	1893	1893
query76	2132	1017	616	616
query77	348	382	278	278
query78	9792	9670	9148	9148
query79	905	844	538	538
query80	518	438	344	344
query81	530	318	278	278
query82	206	135	107	107
query83	195	200	190	190
query84	276	122	98	98
query85	768	424	365	365
query86	292	241	228	228
query87	2001	1990	1873	1873
query88	3735	2784	2802	2784
query89	323	277	242	242
query90	2026	177	173	173
query91	168	153	131	131
query92	105	92	90	90
query93	987	1008	597	597
query94	446	312	276	276
query95	591	397	326	326
query96	675	490	243	243
query97	2427	2406	2357	2357
query98	162	160	145	145
query99	766	767	640	640
Total cold run time: 233373 ms
Total hot run time: 151549 ms

[hello-stephen]

ClickBench: Total hot run time: 25.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35aece4ee7fc8214ebeb40623db399ca1e8605ca, data reload: false

query1	0.00	0.00	0.01
query2	0.14	0.08	0.08
query3	0.33	0.12	0.13
query4	1.60	0.13	0.13
query5	0.27	0.27	0.25
query6	1.16	0.65	0.65
query7	0.04	0.01	0.01
query8	0.10	0.07	0.07
query9	0.44	0.35	0.35
query10	0.59	0.62	0.61
query11	0.33	0.19	0.21
query12	0.32	0.18	0.18
query13	0.53	0.52	0.50
query14	0.94	0.92	0.93
query15	0.66	0.57	0.59
query16	0.34	0.37	0.37
query17	1.02	1.04	1.03
query18	0.30	0.28	0.28
query19	2.02	1.81	1.94
query20	0.02	0.02	0.01
query21	15.42	0.34	0.30
query22	4.89	0.13	0.12
query23	15.85	0.50	0.29
query24	1.97	0.56	0.38
query25	0.14	0.10	0.10
query26	0.76	0.27	0.21
query27	0.09	0.09	0.10
query28	3.52	0.84	0.39
query29	12.65	4.35	3.35
query30	0.34	0.18	0.14
query31	2.76	0.58	0.33
query32	3.23	0.60	0.49
query33	2.96	3.02	2.98
query34	15.91	4.03	3.42
query35	3.39	3.37	3.35
query36	0.56	0.49	0.48
query37	0.12	0.09	0.10
query38	0.07	0.08	0.07
query39	0.06	0.06	0.05
query40	0.20	0.17	0.17
query41	0.10	0.06	0.05
query42	0.06	0.06	0.05
query43	0.06	0.06	0.06
Total cold run time: 96.26 s
Total hot run time: 25.09 s

[github-actions]

Static review of apache/doris PR #69996 at head 35aece4 completed in one round and converged. No new inline finding is accepted. The existing threads were read first: the session-owned route bug, shared FE-client timeout, registry address bound, schema-failure route cleanup, unused-binding concern, and cumulative FE-probe concern are either fixed at this head or explicitly addressed in their replies, so none is reposted and no existing P0/P1 remains applicable.

The task goal is to cancel an Arrow Flight query across every participating BE when a result stream closes early, including remote readers, while preserving result cleanup for older nodes. The FE route is registered after remote execution has endpoint and backend information and before the coordinator is closed or deferred; the BE reader asks heartbeat-listed FEs in reachable-first order, uses a cancellation-only client cache with bounded connection attempts, and routes the result id to every captured BE. Local and remote reader paths, normal EOF, query-level cancellation, and old-BE handling were traced end to end.

Concurrency and lifecycle checkpoints are satisfied by the synchronized route registry and identity-aware whole-route removal, immutable route snapshots during asynchronous cancellation, the request/deferred-executor ownership rules, and the dedicated BE client-cache lifetime. Replacement, expiry, capacity eviction, partial backend dispatch, retry retention, session next-request, teardown, schema-fetch failure, and prepared-schema mismatch were checked. No lock-order, use-after-reset, or counter-drift issue was substantiated.

The change adds no configuration, persistence, transaction, storage-format, or user-authored network endpoint. The new FrontendService method is appended with an optional argument; the same signature is present on branch-4.1 and branch-4.2, while older branches do not reuse an id. Unknown-method responses from older FEs are caught by the BE path and it continues probing. The FE/BE cancellation status conversion and the existing cancel_plan_fragment compatibility path were inspected.

The changed unit and regression tests cover parallel and nonparallel result endpoints, local and remote readers, EOF versus abort, retry and stale transports, reachable and unknown FEs, old result BEs, route bounds and replacement, failed GetFlightInfo cleanup, prepared-schema drift, next-request and session teardown, and a subsequent healthy query. These tests were inspected but not executed because the review contract is static-only and prohibits builds and test runs. No user-provided review focus was supplied.

Review convergence: true. Coverage includes every changed path and all assigned subagent scopes; each subagent returned NO_NEW_VALUABLE_FINDINGS after its final recheck.

Review converged after 1 round(s).

[hello-stephen]

FE UT Coverage Report

Increment line coverage 84.50% (109/129) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	67.48% (32206/47728)
Line Coverage	52.33% (339181/648142)
Region Coverage	47.84% (275732/576393)
Branch Coverage	49.64% (129919/261709)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.65% (35905/46240)
Line Coverage	63.02% (403994/641074)
Region Coverage	59.63% (342117/573705)
Branch Coverage	60.72% (158341/260769)

[morningman]

run p0

[morningman]

run nonConcurrent

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.68% (35921/46240)
Line Coverage	63.06% (404273/641074)
Region Coverage	59.65% (342232/573705)
Branch Coverage	60.76% (158456/260769)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.68% (35921/46240)
Line Coverage	63.06% (404273/641074)
Region Coverage	59.65% (342232/573705)
Branch Coverage	60.76% (158456/260769)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.83% (35987/46240)
Line Coverage	63.20% (405188/641074)
Region Coverage	59.85% (343357/573705)
Branch Coverage	60.91% (158831/260769)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.83% (35987/46240)
Line Coverage	63.21% (405201/641074)
Region Coverage	59.85% (343375/573705)
Branch Coverage	60.91% (158841/260769)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 94.87% (37/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.83% (35988/46240)
Line Coverage	63.21% (405214/641074)
Region Coverage	59.82% (343216/573705)
Branch Coverage	60.91% (158828/260769)

dismiss

[morningman]

run buildall

[morningman]

/review

[github-actions]

Static review of all 17 changed files at 35aece4 found one new P2 issue: synchronous FE-to-BE client retirement can block cancellation dispatch beyond the new one-second deadline. The inline finding contains the concrete FQDN refresh and pending-fetch call chain. No existing P0/P1 finding was reconfirmed as applicable to this head.

The reviewed flow is GetFlightInfo -> coordinator dispatch -> detached cancellation route -> BE DoGet abort -> local cleanup -> FE cancelFlightQuery -> cancel_plan_fragment on every participating BE. Normal EOF bypasses cancellation; failed requests cancel through deferred coordinators or detached routes before dropping them.

Critical checkpoint conclusions:

1. Goal and evidence: detached routes implement query-wide cancellation after coordinator/session transitions; focused unit and regression tests cover it. The synchronous dispatch bound needs the inline fix.
2. Scope and clarity: changes stay focused on Flight cancellation plumbing, failed-request cleanup and tests; existing query-cancellation RPCs are reused.
3. Concurrency: registry map/accounting are monitor-protected; RPC work is outside that monitor. Command state is serialized and deferred executors use a separate take/finalize monitor. No new lock-order cycle was found.
4. Lifecycle and initialization: routes retain copied ids/addresses, not coordinators or sessions. Flight stops/joins before the dedicated cache is destroyed. No new ownership cycle or cross-TU static dependency was found.
5. Configuration: no configuration items were added.
6. Compatibility: old FEs are handled through caught unknown-method errors; older result BEs still understand query cancellation and retain their historical buffer cleanup policy. No storage-format change.
7. Thrift/protobuf: inspected the new method against live branch-3.1, 4.0, 4.1 and 4.2 definitions. It is absent on 3.1/4.0 and identical on 4.1/4.2; no field-id, enum-value or required-field change.
8. Parallel paths: checked local/remote readers, EOF/error/destructor, normal/deferred coordinators, statement/prepared execution, retry, cloud inheritance and failed GetFlightInfo cleanup. Multi-statement admission prevents endpoint remapping to a second result query.
9. Conditional branches: new comments explain reachable/unreachable FE ordering, bounded retry, current-request cleanup and deferred-query exclusions.
10. Test coverage: inspected fanout, failed replies/retry, expiry, session transitions, capacity/replacement, schema failures, old-BE behavior, pool exhaustion and EOF cases. Synchronous proxy replacement is the missing case identified inline.
11. Expected results: no golden result files changed. The new regression checks query-specific pipelines on every alive BE; cross-BE behavior requires multiple participating BEs. Test execution and author-reported pass results were not independently validated.
12. Observability: new cache metrics and failure logs include query/result/backend context; no additional observability defect found.
13. Persistence and failover: no EditLog or durable transaction metadata changes. Routes are deliberately ephemeral and capacity/time bounded; losing one leaves the existing query timeout fallback.
14. Data writes and correctness: no new data-write, visible-version or delete-bitmap behavior; no new atomicity issue was found in the modified cancellation paths.
15. FE/BE variables: no session variable propagation added; the RPC carries an existing result id and uses the existing query-id cancellation protocol.
16. Performance and memory: result-id and backend-address capacities bound retained routing metadata; heavy work stays outside locks. The new inline issue covers blocking client retirement. No new hub-header includes or libc API requirements were introduced.
17. Other issues and user focus: no additional user focus was supplied. R1-R3 and all candidate findings have explicit accepted, duplicate or evidence-backed dismissed dispositions; no unresolved static-review scope remains.

Prior threads: session-independent route lifetime, shared-cache timeouts, backend-address budgeting and failed-GetFlightInfo route cleanup are addressed at this head. The structured-binding P1 does not apply because info is used. Residual total BE-to-FE probing delay remains covered by its existing P2 thread and is not reposted; the new finding concerns a different FE-to-BE gRPC dispatch stage.

Review converged in round 2 after all three reviewers returned NO_NEW_VALUABLE_FINDINGS beyond the verified finding. Coverage limits: this was static inspection only; no builds, tests, benchmarks, formatting/static-analysis tools or runtime/ELF validation were executed.

Review converged after 2 round(s).

[github-actions]

[P2] Bound the synchronous phase of cancellation dispatch

sendCancellation is synchronous even though the proxy method returns a future. If an FQDN BE's cached IP changes while its old channel has an outstanding fetchDataAsync, BackendServiceProxy.getProxy() creates a replacement and then calls removedClient.shutdown() in finally before returning; BackendServiceClient.shutdown() can wait 5 seconds for existing calls, and up to 5 more after shutdownNow(). The one-second deadline here only bounds the later future.get(), so this blocks dispatch to that BE and the remaining BEs, times out the first 2-second BE report, and also blocks the supposedly non-waiting failed-GetFlightInfo cleanup. A retry may use the published replacement, but it does not bound the original FE handler. Move blocking retirement/dispatch off this path or enforce the budget across dispatch too; cover channel replacement with a pending fetch.

[morningman]

Not changing this here: the blocking step is master's shared FE-to-BE client, and nothing is lost while it blocks.

BackendServiceProxy.getProxy() serves every FE-to-BE RPC. When a cached client's IP changed (FQDN mode, after the 60 s DNS refresh) or its channel left CONNECTING/IDLE/READY, it publishes a new client and retires the old one on the caller's thread: shutdown() waits only while that channel has calls in flight, up to 5 s + 5 s, once per proxy instance (48) and BE, for whichever RPC comes first. Coordinator.cancel dispatches the same way, including the deferred-coordinator cancellation in the same failed-GetFlightInfo catch; "not awaited" means the answers, as there.

Meanwhile the handler still cancels on every BE once the channel is retired, answers INTERNAL_ERROR so the route stays, and the BE keeps its own bound (2 s per attempt, one retry, then the next FE). The retry goes through another proxy instance and normally removes the route; otherwise the query is cancelled up to 10 s late and the route expires.

Retiring replaced clients asynchronously in getProxy() fixes every caller; that belongs in its own PR, not in an executor for this one caller. The description's "at most 1 s" bounds the wait for the answers; the merge message will say so.

[morningman]

run buildall

[morningman]

/review

[github-actions]

Static review of all 17 changed paths at c5772bb found no additional distinct verified finding. The known P2 synchronous-dispatch issue in 4545638345 still applies: proxy replacement can wait for channel retirement before returning a cancellation future. The cumulative FE-probe delay in 4469160220 remains a known limitation, with reachable-first ordering now mitigating stale-FE delays. Neither issue is reposted.

Existing P1 4443420978 is fixed by session-independent route ownership. Existing P1 4469158953 is inapplicable at this head because both structured-binding elements are used. No independently confirmed existing P0/P1 remains. Earlier shared-cache timeout, backend-address budgeting, and failed-GetFlightInfo route-cleanup concerns are addressed.

The inspected execution chain is:

GetFlightInfo -> backend dispatch -> detached cancellation route -> ticket
DoGet abort -> local cleanup -> owning FE -> query cancellation on every BE
Schema admission failure -> deferred coordinator or route cancellation -> remove route

Critical checkpoint conclusions:

1. Goal: early stream termination now routes query-wide cancellation after coordinator/local-fragment completion; EOF preserves results. Added unit and regression cases exercise the intended paths in source.
2. Scope and reuse: the changes are focused on routing, failure cleanup, an isolated Thrift cache, and tests; they reuse the existing backend query-cancellation RPC.
3. Concurrency and locking: session commands serialize endpoint mutation; registry state/accounting share one monitor. RPCs and future waits occur outside it. Identity-conditioned removal protects replacement routes; frontend snapshots and transport checkout use existing locks. No new lock-order cycle found.
4. Lifecycle and static initialization: routes retain copied IDs/addresses rather than sessions/coordinators and have expiry and capacity bounds. Flight stops/joins before the new cache is destroyed. No new ownership cycle or cross-translation-unit initializer dependency found.
5. Configuration: no new configuration item; existing connection-open policy remains in force alongside the dedicated two-second RPC transports.
6. Compatibility: the additive FE method tolerates old FEs through caught unknown-method errors; cancellation uses the existing query-level RPC understood by older BEs. Older BEs retain their historical buffer cleanup behavior; the full improvement requires upgraded FE and BEs.
7. Thrift/protobuf: the method is absent on branch-3.1/4.0 and has the same signature on branch-4.1/4.2. No struct field IDs or enum values change, and no required field is introduced.
8. Parallel paths: local/proxied readers, statement/prepared requests, closed/deferred coordinators, request retries, schema rejection and normal EOF were traced. Flight disables incompatible SQL-cache replay and short-circuit execution. Non-final result statements are rejected before another result query runs.
9. Conditional checks: new retry, NOT_FOUND, reachability and failure-cleanup branches explain their concrete scenarios; no additional incorrect condition found.
10. Test coverage: inspected FE routing/budget/lifetime/error tests, producer/prepared-schema cleanup and executor registration assertions, BE cancellation/retry/legacy/EOF/transport cases, and the query-specific all-BE regression checks. Proxy mocks do not exercise the known synchronous retirement issue.
11. Test results: no generated .out result changes require validation.
12. Observability: failure logs identify result/query/backend addresses, and the dedicated cache registers client metrics. No additional critical observability gap found.
13. Transactions and persistence: no EditLog, metadata persistence or failover protocol changes.
14. Data modification: query-abort behavior changes; committed-data visibility, write atomicity, partition versions and delete bitmaps are untouched.
15. FE/BE variables: the new FE method passes an existing TUniqueId; backend cancellation retains the actual query ID and existing status fields. No new session-variable transport path.
16. Performance: registry retention is bounded by result and backend counts; fanout and FE probing scale with participant counts. Known dispatch/probing latency limitations are retained above.
17. Further recheck: no other substantiated new issue or unreviewed changed path remains.

No additional user focus was supplied. Validation was static inspection only: no builds, tests, benchmarks, formatters or runtime checks were executed. A single-BE regression run would not demonstrate the distributed effect; the cross-BE assertions require multiple BEs. All assigned reviewers returned NO_NEW_VALUABLE_FINDINGS, with no unresolved new candidates or static coverage gaps. Review converged after two rounds; the known P2 issues above remain applicable.

Review converged after 2 round(s).

[morningman]

run external