Repository navigation
[chore](ci) Publish pipeline policy and restore regression coverage - #70577
Open
hello-stephen wants to merge 1 commit into
Open
hello-stephen wants to merge 1 commit into
hello-stephen wants to merge 1 commit into
Conversation
### Background Community CI consumes source-bound Doris builds and regression case packages. Regression profiles already contain public configuration, but managed CI adapters also add build/test flags, SQL settings, fixture omissions and suite filters after reading those files. Some migration-era exclusions remain in the public profiles as well. ### What problem does this PR solve? Issue Number: None Related PR: apache#70457 (rootless CI validation context) PR Type: Build/CI/Chore Problem Summary: Developers cannot review the effective test policy when CI silently overrides the source configuration. This change publishes the retained policy with reasons, restores migration-era exclusions based on assertions, memory limits, timeouts or obsolete adapters, and provides source-owned consumers for the managed adapters. In particular, the regression adapter preserves appended source exclusions instead of replacing the final External exclusion list. Pipeline concurrency remains managed by CI, and Performance is unchanged. ### Details - **Change map:** 982 changed lines in policy/configuration and consumers (719 added, 263 removed), 168 test lines, and 143 documentation lines; no generated artifacts. Most configuration additions include license headers and inline rationale. Start with `regression-test/pipeline/CI_CONFIG.md`, then the profile configs and `common/source-config.py`. - **Affected:** Compile, FEUT, BEUT, CloudUT, P0, CloudP0, VaultP0, NonConcurrent and External. Source files now own fixed compiler/JVM/test policy, whole-build and test-stage timeout declarations, initialization SQL, FEUT's temporary mocked-FE port exclusion and External fixture omissions. Retained environment exclusions include restoration conditions. The BEUT policy uses `*`; the External consumer has no injected assertion/memory exclusion list. - **Behavior change:** Restored suites can expose existing product failures. The additional raw-JUnit verdict rejects assertion/error results even when CI has muted the test; it supplements existing completeness, sanitizer, exit-code and artifact checks. Existing baseline exclusions outside the audited migration blocks are not rewritten. Concurrency, Performance behavior, and Compile/BEUT whole-build timeout budgets stay unchanged. - **Developer and release impact:** Public settings travel with the exact tested source/cases revision. The new consumers reject missing/unknown configuration rather than restoring private legacy exclusions. The managed TeamCity adapter switch is a separate deployment: backport the public files to every enabled branch first, update all embedded modules/checksum pins together, and mirror/read back the public whole-build timeouts. This PR does not by itself deploy TeamCity changes. - **License and third-party:** Apache-licensed Python/Bash helpers using the standard library; no new dependency. CloudUT reuses the existing source-owned FDB entropy shim with the same runtime-image condition instead of embedding another C copy. - **Rollback:** Retain a matched source revision and CI configuration export. Do not remove the config marker to restore hidden exclusion behavior. See the rollout contract in `CI_CONFIG.md`. ### Testing - Locally ran `python3 -m unittest discover -s regression-test/pipeline/common/tests -p test_source_policy.py -v`: **9 tests passed**. They cover source exclusion append order, OSS source settings, missing/unknown marker rejection, restored suite names, branch-specific policy, unchanged concurrency inputs, public-only fixture omissions, and failure/error verdicts including muted metadata. - Python syntax checks, `bash -n` for new shell/policy files, and `git diff --cached --check` passed. - Rendered the companion managed-adapter candidate from a read-only TeamCity export and checked it offline: shell/embedded-Python syntax, module checksums, compressed-bundle hashes, final manifests including overlays, unchanged Performance, and unchanged build settings/triggers/dependencies. The candidate removes the private filters, provider/SQL overrides and component parameter in favor of these source consumers. - **Not run:** Doris compilation, complete unit/regression lanes, live adapter deployment, or end-to-end downstream dispatch. The offline checks do not establish runtime success; full source-bound validation is required after review and coordinated rollout. ### Notes for Reviewers Please review the retained prerequisite exclusions and the raw-failure verdict explicitly. Restoring test visibility may make previously green builds fail; those failures should be diagnosed rather than adding new product-assertion exclusions. Managed-adapter deployment must wait until the enabled source branches contain the new files. Performance rollback investigation is outside this change. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; the configuration ownership and rollout contract is included in `regression-test/pipeline/CI_CONFIG.md`. - Pre-submission status: Partially verified: offline contracts and candidate integrity passed; full CI and the managed adapter rollout remain unverified. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Policy/configuration changes, consumers, tests, documentation, companion offline adapter candidate and this description. - Human review: Pending maintainer review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Community CI consumes source-bound Doris builds and regression case packages. Regression profiles already contain public configuration, but managed CI adapters also add build/test flags, SQL settings, fixture omissions and suite filters after reading those files. Some migration-era exclusions remain in the public profiles as well.
What problem does this PR solve?
Issue Number: None
Related PR: #70457 (rootless CI validation context)
PR Type: Build/CI/Chore
Problem Summary:
Developers cannot review the effective test policy when CI silently overrides the source configuration. This change publishes the retained policy with reasons, restores migration-era exclusions based on assertions, memory limits, timeouts or obsolete adapters, and provides source-owned consumers for the managed adapters. In particular, the regression adapter preserves appended source exclusions instead of replacing the final External exclusion list. Pipeline concurrency remains managed by CI, and Performance is unchanged.
Details
regression-test/pipeline/CI_CONFIG.md, then the profile configs andcommon/source-config.py.*; the External consumer has no injected assertion/memory exclusion list.CI_CONFIG.md.Testing
python3 -m unittest discover -s regression-test/pipeline/common/tests -p test_source_policy.py -v: 9 tests passed. They cover source exclusion append order, OSS source settings, missing/unknown marker rejection, restored suite names, branch-specific policy, unchanged concurrency inputs, public-only fixture omissions, and failure/error verdicts including muted metadata.bash -nfor new shell/policy files, andgit diff --cached --checkpassed.Notes for Reviewers
Please review the retained prerequisite exclusions and the raw-failure verdict explicitly. Restoring test visibility may make previously green builds fail; those failures should be diagnosed rather than adding new product-assertion exclusions. Managed-adapter deployment must wait until the enabled source branches contain the new files. Performance rollback investigation is outside this change.
Release note
None
Check List (For Author)
regression-test/pipeline/CI_CONFIG.md.AI Disclosure