Skip to content

[fix](iceberg) Preserve rooted namespace identity - #70565

Draft
hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:fix/iceberg-root-namespace-master
Draft

hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:fix/iceberg-root-namespace-master

Conversation

@hubgeter

@hubgeter hubgeter commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Background

Iceberg catalogs can expose databases below the namespace configured by external_catalog.name. Ordinary discovery returns each direct child's relative name, while REST nested discovery returns complete names containing the root.

What problem does this PR solve?

Issue Number: None
Related PR: #56874 (introduced the reversed root order), #64304 (preserved it during the connector migration)
PR Type: Bugfix

Problem Summary:

A catalog rooted at root discovers root.db, but subsequent operations can resolve db.root or root.db.root. If the reversed namespace contains a same-named table, loading succeeds with the wrong object. Resolve database names as the inverse of discovery in the common catalog implementation, keeping the configured root in one level. Reject rooted nested names outside that root and child levels that cannot be represented unambiguously.

Details

  • Change map: Production +49/-14 lines; tests +379/-16 lines, including a SQL regression suite; no generated files or documentation changes.
  • Frequency: Affects catalogs using an external namespace root. Ordinary mode reverses the root/database order; nested mode appends an additional root.
  • Root cause: toNamespace always appended the external root after splitting the display name, while discovery started below that root and used two different display contracts.
  • Why existing tests missed it: Existing parity fixtures explicitly expected a trailing root, including the provider-construction fixture. They did not round-trip a discovered database against two different same-named tables.
  • Fix: Ordinary rooted names identify one direct child below the root. Nested rooted names already contain the root; remove the exact configured prefix before parsing child levels. Preserve literal dots in the configured root, direct-child dots and child whitespace. Leave unrooted dotted-name parsing unchanged. Use the same conversion for table/view metadata and DDL, including separate view facets.
  • Other paths and branches: Scan, write and procedure providers retain the common catalog resolver. branch-4.2 also has the defect at 7523405f6aa8e9df3d110eb432fed1ff6e1c3418; it needs a semantic backport in its legacy IcebergMetadataOps, rather than a mechanical cherry-pick of connector classes. That backport is not included here.
  • Behavior change: Operations address the namespace originally discovered. Rooted nested inputs must include the configured root; ambiguous dotted child levels fail explicitly. Existing remote objects are not migrated.

Testing

  • The original three correctness reproducers fail on unmodified master. All 19 new namespace cases now pass, covering reversed tables, nested descendants, root/child names containing dots, whitespace, target disappearance, and database/table DDL.
  • Full Iceberg connector suite through run-fe-ut.sh: 1,968 tests, zero failures/errors, 11 existing skips. Including upstream reactor dependencies: 3,151 tests, zero failures/errors, the same 11 skips. CUSTOM_MVN restricts the reactor to the Iceberg module and its dependencies and enables Checkstyle; the repository test entry point is unchanged.
  • Checkstyle passes for all 21 selected projects; the connector import gate and git diff --check pass.
  • New SQL regression checks table identity through unique column binding with LIMIT 0, expected unknown-column errors, cache refresh, and CREATE/RENAME routing. Its syntax compiles with the regression framework's Groovy 4.0.19. Live SQL/REST execution is pending.
  • Standard FE build passed: ./build.sh --fe -j2 --exclude-obs-dependencies with two Maven workers. The optional OBS dependency closure was excluded.

Release note

Fix Iceberg catalogs configured with external_catalog.name loading the wrong namespace or failing to find discovered databases and tables. Rooted nested names must include the configured root, and ambiguous dotted child levels are rejected explicitly.

Check List (For Author)

  • Test
    • Unit Test
    • Regression test execution (suite added; live cluster run pending)
    • Manual test
    • No need to test
  • Behavior changed: Yes (see Details)
  • Does this need documentation: No; no configuration is added, and the corrected name contract is described in the release note.
  • Pre-submission status: Partially verified: local component tests, full FE build, Checkstyle, import gate and regression syntax pass; live SQL/REST regression and remote CI remain pending.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

AI Disclosure

  • Tool: Codex
  • Generated: Code, tests and this description
  • Human review: Pending

### What problem does this PR solve?

Issue Number: None

Related PR: apache#56874, apache#64304

Problem Summary:
Catalogs configured with an external namespace root discover root.db but
subsequent operations append the root, selecting db.root or root.db.root.
A same-named table in the reversed namespace can load as the wrong object.
Resolve rooted database names as the inverse of discovery: direct children
use relative names and nested names already contain the configured root.
Keep the root as one level, preserve child whitespace, and reject ambiguous
nested child levels. Correct existing parity fixtures and cover table,
view, provider, DDL and SQL metadata-identity paths.

### Release note

Fix Iceberg catalogs with external_catalog.name loading the wrong namespace
or failing to find discovered databases and tables. Rooted nested names
must include the configured root, and ambiguous dotted child levels are
rejected explicitly.

### Check List (For Author)

- Test: Unit Test
    - run-fe-ut.sh: 3151 tests, 0 failures/errors, 11 existing skips; Iceberg has 1968 tests
    - All 19 new namespace cases pass, including the three original correctness failures
    - Checkstyle, connector import gate, Groovy 4.0.19 syntax and diff checks pass
    - Standard FE build is in progress; live SQL/REST regression has not run
- Behavior changed: Yes (namespace operations use the object originally discovered)
- Does this need documentation: No new configuration; the corrected contract is in the release note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant