Skip to content

Revisit cuda-bindings branching strategy #1199

Closed

Description

@leofang

Right now we have cuda-bindings 12.9.x maintained in the 12.9.x branch. This causes frictions for

We would like to lower the burden and the mental overhead on the team. Possible solutions include:

  • Eliminate the backport needs by hosting two directories cuda_bindings_12/ and cuda_bindings_13/ on the main branch
  • Generate the same binding code that have a bunch of macros that can be toggled at build time to target CUDA 12/13
  • ...

Activity

  1. added
    triageNeeds the team's attention
    cuda.bindingsEverything related to the cuda.bindings module
    on Oct 29, 2025
  2. removed this from the cuda.bindings backlog milestone on Apr 2, 2026
  3. rparolin commented on Jun 9, 2026

    @rparolin
    Collaborator

    TL;DR

    The pain comes from having multiple long-lived branches, not from how the source is laid out. Move to a single branch with per-major directories and most of it goes away. Build-time macros (proposal #2) is a nicer end-state but the wrong first step.

    What I found

    • The .pyx.in/.pxd.in files are not hand-written templates. They're generated, with a banner: "automatically generated… Do not modify it directly."
    • What differs between main (13.x) and 12.9.x is real CUDA 12-vs-13 API surface (new enums, structs, renumbered values) — emitted by the generator. ~600 lines in cydriver.pxd.in alone.
    • All generated output (56 .pyx/.pxd/.cpp + 18 .in) is git-tracked. The build already regenerates from the installed toolkit's headers, so the committed copies are effectively cached.

    So: every pain you listed (#965, #1058, #809, #1120) exists because there are multiple branches. Collapse to one branch and all four go away — independent of dirs-vs-macros.

    The two proposals

    #1 — per-major directories on main (recommended first step)

    • One branch. Each CUDA major is a generated directory: gen/v12/, gen/v13/. New major = copy + regenerate. EOL = delete the directory. No branch is ever cut or retired.
    • Hand-written code (__init__.py, _lib/, build_hooks.py, ci/, .github/, docs) lives once. A fix is one PR — no backport.
    • Build/release picks the major from the toolkit headers (already keys off CUDA_PATH) and maps gen/vNN/ onto the cuda.bindings package. Same package name, major-distinct versions, one installed at a time — no wheel collisions.

    #2 — single source + build-time macros (better end-state, later)

    The deltas are real API differences the generator owns. A single tree with {{if cuda_major==13}} means either hand-editing thousands of generated symbols, or teaching the upstream generator to emit version-gated code. That's a generator project, not a repo change. (CuPy does this — worth a look once/if the generator supports it.)

    Migration path

    1. Make the build's major-selection explicit so a gen/vNN/ dir can be picked deterministically.
    2. Move 13.x into gen/v13/; verify wheels are byte-equivalent to today.
    3. Re-home 12.9.x into gen/v12/ and cut a patch release from main — proves release works without a branch (closes the BUG: ./ci/tools/download-wheels does not exist in the backport branch #1120 class).
    4. Decide: commit gen/ vs. regenerate in CI. If regenerating, validate generator availability in CI.
    5. Retire 11.8.x / 12.2.x / 12.9.x as their lines move to directories or hit EOL.

    Bottom line: single-branch is the decision that matters. Start with per-major directories; treat build-time macros as a later optimization.

  4. added this to the cuda.core 1.3.0 milestone on Aug 19, 2026
  5. rwgk commented on Nov 13, 2026

    @rwgk
    Contributor

    — codex GPT-6 on behalf of Ralf —

    CI/release consolidation: before and after

    TL;DR: CUDA 12 and CUDA 13 now share CI/release infrastructure on main, replacing two separately maintained branch implementations. Counting .github/ and ci/ across both old branches, the combined maintenance surface decreased:

    Measure Before: main + 12.9.x After: shared main Reduction
    All tracked text lines 22,737 20,819 1,918 (8.4%)
    Code/configuration lines, including tests 15,488 14,211 1,277 (8.2%)

    Production/configuration code decreased 14.9% (13,196 → 11,229); CI-tool test code grew by 690 lines (2,292 → 2,982). These counts describe maintenance surface. The architectural benefit is removing routine cross-branch CI/release synchronization and its opportunities for divergence.

    Overview. The migration stack and cleanup in #3434 and #3456 put the CUDA-major choice into a shared registry and shared workflows. Builds, tests, releases and documentation can now be maintained together. Both bindings majors remain available; retiring routine 12.9.x maintenance preserves the branch's Git history.

    The tradeoff is explicit: the shared implementation is larger than old main alone—18,364 → 20,819 total text lines—while the combined two-branch footprint is smaller. The comparison also includes concurrent infrastructure changes; it does not isolate the migration's causal effect on every line.

    What changed, what remains, and the detailed breakdown

    Maintenance responsibilities reduced

    Remaining obligations

    • cuda_bindings_12/ and cuda_bindings_13/ remain separate package roots. Applicable handwritten fixes still require checking both; generated bindings still require each major's own toolkit inputs. These package sources are outside this LOC comparison. Maintenance guidance.
    • Current release workflows expect the registry layout. Releases of older, pre-registry layouts use compatible historical workflow refs, retaining access to old behavior without carrying its parser in the current workflows. Release policy.
    • Full CI, nightly, coverage and eight release rehearsals were completed for the cleanup. The validation summary records the scope and limits, including the exclusion of live release publication.

    Production, tests, documentation and assets

    Category All text lines before After Code/config lines before After
    Production/configuration 17,265 14,602 13,196 11,229
    CI-tool tests 3,071 3,939 2,292 2,982
    Documentation/diagrams 789 1,132 Excluded Excluded
    Certificate/data assets 1,612 1,146 Excluded Excluded
    Total 22,737 20,819 15,488 14,211

    The production/configuration reduction of 1,967 code lines and test increase of 690 yield the net reduction of 1,277. The physical-line total also includes the removal of one 466-line copy of certificate data; that asset contributes nothing to the code/configuration measure.

    By directory

    Directory All text lines before → after Code/config lines before → after
    .github/ 12,811 → 9,854 9,155 → 7,113
    ci/ 9,926 → 10,965 6,333 → 7,098

    Shared Python helpers and their tests grew as workflow handling was consolidated. Directory subtotals show this redistribution; the headline comparison includes both directories and both old branches.

    How different were the old trees?

    Matching files strictly by relative path:

    Relationship between old main and 12.9.x Paths
    Present in both, byte-identical 3
    Present in both, different contents 31
    Present only in main 92
    Present only in 12.9.x 7

    The two old trees contained 126 + 41 = 167 file copies; the shared tree contains 144. Different contents include intentional CUDA-major differences, and path-only comparisons do not identify renames or quantify harmful drift. These figures show the comparison/synchronization surface that routine branch maintenance previously involved.

    Full details: fixed snapshots, counting rules and reproducibility

    Snapshots

    The before boundary is immediately before #3314, the first PR of the implemented migration, merged on 2026-11-06 at 16:37:20 UTC. This includes its preparatory work in the measured change. It is not the earlier historical creation of the cuda_bindings_12/ directory.

    Snapshot Commit All text lines Code/config lines
    Before: main ceb91b50b91293231c13494ea414caa8d115a759 18,364 12,437
    Before: 12.9.x, at the same cutoff 06e182cee03624c6ebdf3b865eaaefa201197661 4,373 3,051
    After: reviewed #3456 head 8f01de551993ab0d9102002d97a9c113776bdd39 20,819 14,211

    The 12.9.x snapshot is the result of #3306; its next update (#3429) occurred November 10 and did not change either counted directory. The after snapshot is fixed to the reviewed #3456 revision, rather than a moving main reference.

    Counting rules

    Two independently implemented counts agreed on the headline numbers and directory/test subtotals. The standalone reproduction below was also run successfully against the fixed objects.

    Reproduce the counts from Git objects

    Run this Python script from a clone of NVIDIA/cuda-python containing the three commits, with Python 3 and cloc 1.98 installed. It reads fixed Git objects and extracts temporary data; it does not inspect or change the checkout, branches, or tags.

    # Run from a NVIDIA/cuda-python clone containing the three commit objects.
    # Requires Python 3 and cloc 1.98. Does not read or change the checkout.
    import json
    from pathlib import Path
    import subprocess
    import tempfile
    
    snapshots = {
        'main before': 'ceb91b50b91293231c13494ea414caa8d115a759',
        '12.9.x before': '06e182cee03624c6ebdf3b865eaaefa201197661',
        'after': '8f01de551993ab0d9102002d97a9c113776bdd39',
    }
    asset = '.github/actions/upgrade-ca-certificates/impacted-intermediates.json'
    excluded = {'.md', '.txt', '.mmd', '.pem'}
    results, manifests = {}, {}
    
    def git(*args):
        return subprocess.check_output(['git', *args])
    
    def cloc(files, output, force_language):
        command = ['cloc', '--config', '/dev/null', '--skip-uniqueness',
                   '--by-file', '--json', '--quiet', '--timeout=0',
                   '--force-lang=' + force_language, '--out=' + str(output)]
        subprocess.run(command + list(map(str, files)), check=True,
                       stdout=subprocess.DEVNULL)
        return {str(Path(p).resolve()): v for p, v in json.loads(output.read_text()).items()
                if p not in ('header', 'SUM')}
    
    with tempfile.TemporaryDirectory(prefix='issue1199-') as temporary:
        root = Path(temporary).resolve()
        for index, (label, sha) in enumerate(snapshots.items()):
            directory = root / str(index)
            physical = test_code = production_code = 0
            selected, manifests[label] = [], {}
            tree = git('ls-tree', '-rz', '--full-tree', sha, '--', '.github', 'ci')
            for entry in tree.split(b'\0'):
                if not entry:
                    continue
                metadata, raw_path = entry.split(b'\t', 1)
                mode, kind, oid = metadata.decode().split()
                assert kind == 'blob' and mode in ('100644', '100755')
                path = raw_path.decode()
                manifests[label][path] = oid
                data = git('cat-file', 'blob', oid)
                assert b'\0' not in data
                data.decode('utf-8')
                physical += data.count(b'\n') + int(bool(data) and not data.endswith(b'\n'))
                target = directory / path
                target.parent.mkdir(parents=True, exist_ok=True)
                target.write_bytes(data)
                if target.suffix.lower() not in excluded and path != asset:
                    selected.append(target)
            special = directory / '.github/BACKPORT_BRANCH'
            normal = [p for p in selected if p != special]
            counts = cloc(normal, root / 'counts.json', 'Bourne Shell,env')
            if special in selected:
                counts.update(cloc([special], root / 'special.json', 'Bourne Shell'))
            assert set(counts) == {str(p.resolve()) for p in selected}
            for filename, count in counts.items():
                path = str(Path(filename).relative_to(directory))
                if path.startswith('ci/tools/tests/') or path == 'ci/tools/smoke_test_lychee_precommit.py':
                    test_code += count['code']
                else:
                    production_code += count['code']
            results[label] = dict(physical=physical, code=production_code + test_code,
                                 production=production_code, tests=test_code)
    
    first, second = manifests['main before'], manifests['12.9.x before']
    common = first.keys() & second.keys()
    results['before file pairs'] = dict(
        identical=sum(first[p] == second[p] for p in common),
        different=sum(first[p] != second[p] for p in common),
        main_only=len(first.keys() - second.keys()),
        maintenance_only=len(second.keys() - first.keys()))
    print(json.dumps(results, indent=2))

    The two before results are added, and the reduction percentage is (before_total - after) / before_total * 100.

    Per-file counts (155 paths across the three snapshots)

    Each cell is physical text lines / code-config lines. A single - means the path is absent; P/- means the file contributes P physical lines and is excluded from code/configuration. Zero code lines remain 0.

    Relative path Old main P/C Old 12.9.x P/C After P/C
    .github/BACKPORT_BRANCH - 1/1 -
    .github/CODERABBIT.md 34/- - 34/-
    .github/ISSUE_TEMPLATE/bug_report.yml 115/103 116/103 115/103
    .github/ISSUE_TEMPLATE/config.yml 9/5 9/5 9/5
    .github/ISSUE_TEMPLATE/doc_request.yml 43/35 44/35 43/35
    .github/ISSUE_TEMPLATE/feature_request.yml 71/62 71/62 71/62
    .github/ISSUE_TEMPLATE/release_checklist.yml 34/28 37/31 36/30
    .github/PULL_REQUEST_TEMPLATE.md 13/- 14/- 13/-
    .github/RELEASE-bindings.md - - 119/-
    .github/RELEASE-core.md 317/- - 368/-
    .github/actionlint.yaml 49/36 - 49/36
    .github/actions/doc_preview/action.yml 50/37 69/55 50/37
    .github/actions/download-artifact-retry/action.yml 128/115 - 128/115
    .github/actions/fetch_ctk/action.yml 221/166 191/146 284/221
    .github/actions/get_pr_number/action.yml 26/19 58/49 26/19
    .github/actions/griffe-api-check/action.yml 44/36 - 44/36
    .github/actions/install-sccache/action.yml 75/43 - 75/43
    .github/actions/install_unix_deps/action.yml 47/39 49/41 47/39
    .github/actions/install_windows_deps/action.yml 92/79 - 92/79
    .github/actions/lychee-links/action.yml 146/119 - 146/119
    .github/actions/sccache-summary/action.yml 80/58 - 80/58
    .github/actions/setup-hardened-python/action.yml 168/133 150/117 168/133
    .github/actions/setup-pixi/action.yml 33/24 - 33/24
    .github/actions/upgrade-ca-certificates/action.yml 144/96 144/96 144/96
    .github/actions/upgrade-ca-certificates/emudhra_intermediates.ps1 124/79 50/30 124/79
    .github/actions/upgrade-ca-certificates/impacted-intermediates.json 466/- 466/- 466/-
    .github/copy-pr-bot.yaml 10/3 10/3 10/3
    .github/dependabot.yml 79/54 10/6 57/38
    .github/labeler.yml 29/19 - 29/19
    .github/matchers/compiler.json 43/43 - 43/43
    .github/matchers/compute-sanitizer.json 24/24 - 24/24
    .github/workflows/backport.yml 76/63 40/31 -
    .github/workflows/bandit.yml 50/37 23/17 50/37
    .github/workflows/build-docs.yml 394/317 261/212 460/376
    .github/workflows/build-wheel.yml 721/614 306/244 799/684
    .github/workflows/ci-nightly.yml 402/340 - 415/353
    .github/workflows/ci-pixi-lockfile-freshness-check.yml 43/32 - 43/32
    .github/workflows/ci-pixi-lockfile-refresh.yml 35/25 - 35/25
    .github/workflows/ci-pixi-source-test.yml 244/150 - 339/229
    .github/workflows/ci-workflow-health.yml 261/238 - 261/238
    .github/workflows/ci.yml 759/621 201/175 659/529
    .github/workflows/cleanup-pr-previews.yml 77/61 - 77/61
    .github/workflows/codeql.yml - 43/34 -
    .github/workflows/coverage.yml 614/500 - 643/523
    .github/workflows/guess_latest.sh - 42/21 -
    .github/workflows/install_gpu_driver.ps1 - 48/25 -
    .github/workflows/lychee.yml 104/78 - 104/78
    .github/workflows/pr-auto-label.yml 23/17 - 23/17
    .github/workflows/pr-metadata-check.yml 166/129 - 166/129
    .github/workflows/pre-commit.yml 189/125 - 189/125
    .github/workflows/release-upload.yml 103/82 89/64 54/45
    .github/workflows/release.yml 469/387 184/162 566/489
    .github/workflows/runner-trust-probe.yml 209/158 - 209/158
    .github/workflows/security-suite.yml 50/29 - 50/29
    .github/workflows/status-check.yml - 20/14 -
    .github/workflows/test-sdist-linux.yml 171/138 - 177/144
    .github/workflows/test-sdist-windows.yml 169/127 - 175/133
    .github/workflows/test-wheel-linux.yml 604/477 262/210 575/459
    .github/workflows/test-wheel-windows.yml 580/485 246/202 559/470
    .github/workflows/triage-author-reply.yml 42/32 - 42/32
    .github/workflows/triage-close-awaiting-response.yml 43/31 - 43/31
    .github/workflows/triagelabel.yml 27/20 26/19 27/20
    .github/workflows/woa-main-ci.yml 192/177 - 187/171
    ci/README.md 253/- - 418/-
    ci/__init__.py - - 2/0
    ci/ci-pipeline.mmd 99/- - 101/-
    ci/cleanup-pr-previews 343/257 - 343/257
    ci/pixi-version.env 4/1 - 4/1
    ci/pyproject.toml - - 30/24
    ci/test-matrix.json - 102/102 -
    ci/test-matrix.yml 164/99 - 164/99
    ci/tools/__init__.py - - 2/0
    ci/tools/bindings_config.py - - 308/247
    ci/tools/build_docs_for_link_check.sh 64/44 - 64/44
    ci/tools/check_mempool_hygiene.py 261/166 - 261/166
    ci/tools/check_pathfinder_ctk_libs.py - - 44/23
    ci/tools/check_pixi_cuda_version.py 91/73 - 79/62
    ci/tools/check_release_notes.py 274/195 - 201/134
    ci/tools/checkout_source_links.py 104/70 - 104/70
    ci/tools/ci_artifacts.py - - 65/41
    ci/tools/ci_env.py - - 134/115
    ci/tools/classify_pixi_lockfile_freshness.py 85/58 - 85/58
    ci/tools/compute_ci_plan.py 224/181 - 406/345
    ci/tools/configure_driver_mode.ps1 112/76 - 112/76
    ci/tools/cuda_core_bindings_floor.py 74/40 - 74/40
    ci/tools/download-wheels 112/69 75/45 180/131
    ci/tools/env-vars 145/102 77/61 -
    ci/tools/fetch_ctk_redistrib.py 270/195 308/234 271/196
    ci/tools/fold_pytest_sections.py 93/42 - 93/42
    ci/tools/guess_latest.sh 42/21 - -
    ci/tools/install_gpu_driver.ps1 92/50 - 92/50
    ci/tools/install_gpu_driver.sh 260/128 - 260/128
    ci/tools/list_pixi_workspaces.py 78/45 - 78/45
    ci/tools/lookup-run-id 322/249 104/71 308/243
    ci/tools/merge_cuda_core_wheels.py 252/162 - 252/162
    ci/tools/retry_lychee.py 234/191 - 234/191
    ci/tools/run-tests 276/174 108/84 297/197
    ci/tools/run_lychee.py 96/62 - 96/62
    ci/tools/run_lychee_precommit.py 104/83 - 104/83
    ci/tools/runner_trust_probe.ps1 163/128 - 163/128
    ci/tools/runner_trust_probe.sh 150/116 - 150/116
    ci/tools/runner_trust_probe_chains/README.txt 39/- - 39/-
    ci/tools/runner_trust_probe_chains/dv-g3b-aaa-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/dv-g3b-aaa-noct-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/dv-g3b-emudhra.we-have-a-problem.com.pem 56/- - 56/-
    ci/tools/runner_trust_probe_chains/dv-g3b-noct-emudhra.we-have-a-problem.com.pem 57/- - 57/-
    ci/tools/runner_trust_probe_chains/dv-g3c-aaa-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/dv-g3c-emudhra.we-have-a-problem.com.pem 56/- - 56/-
    ci/tools/runner_trust_probe_chains/ev-g3b-aaa-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/ev-g3b-emudhra.we-have-a-problem.com.pem 56/- - 56/-
    ci/tools/runner_trust_probe_chains/ev-g3c-aaa-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/ev-g3c-emudhra.we-have-a-problem.com.pem 56/- - 56/-
    ci/tools/runner_trust_probe_chains/ov-g3b-aaa-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/ov-g3b-emudhra.we-have-a-problem.com.pem 56/- - 56/-
    ci/tools/runner_trust_probe_chains/ov-g3c-aaa-emudhra.we-have-a-problem.com.pem 38/- - 38/-
    ci/tools/runner_trust_probe_chains/ov-g3c-emudhra.we-have-a-problem.com.pem 56/- - 56/-
    ci/tools/runner_trust_probe_issuers/README.txt 20/- - 20/-
    ci/tools/runner_trust_probe_issuers/emsign-root-cs-ca-g3-cross.pem 21/- - 21/-
    ci/tools/runner_trust_probe_node.js 67/53 - 67/53
    ci/tools/select-woa-batch 81/63 - 81/63
    ci/tools/select_pixi_packages.py - - 83/68
    ci/tools/select_test_matrix.py - - 126/107
    ci/tools/setup-sanitizer 30/13 22/13 30/13
    ci/tools/smoke_test_lychee_precommit.py 73/47 - 73/47
    ci/tools/tests/AGENTS.md - - 20/-
    ci/tools/tests/test_bindings_config.py - - 207/177
    ci/tools/tests/test_bindings_metadata.py - - 81/67
    ci/tools/tests/test_bindings_release_compat.py - - 115/98
    ci/tools/tests/test_build_docs_for_link_check.py 136/99 - 136/99
    ci/tools/tests/test_check_mempool_hygiene.py 219/142 - 219/142
    ci/tools/tests/test_check_release_notes.py 402/315 - 294/227
    ci/tools/tests/test_check_spdx.py 195/134 - 195/134
    ci/tools/tests/test_checkout_source_links.py 164/134 - 164/134
    ci/tools/tests/test_classify_pixi_lockfile_freshness.py 63/40 - 63/40
    ci/tools/tests/test_compute_ci_plan.py 184/155 - 217/178
    ci/tools/tests/test_cuda_core_bindings_floor.py 99/57 - 99/57
    ci/tools/tests/test_cuda_python_setup.py - - 107/96
    ci/tools/tests/test_download_wheels.py - - 131/89
    ci/tools/tests/test_fetch_ctk_redistrib.py 167/127 159/125 197/153
    ci/tools/tests/test_fold_pytest_sections.py 175/91 - 175/91
    ci/tools/tests/test_list_pixi_workspaces.py 118/93 - 118/93
    ci/tools/tests/test_merge_cuda_core_wheels.py - - 38/29
    ci/tools/tests/test_pixi_lockfile_freshness_workflow.py 120/88 - 109/80
    ci/tools/tests/test_release_publish_security.py - - 25/16
    ci/tools/tests/test_retry_lychee.py 449/364 - 449/364
    ci/tools/tests/test_run_lychee.py 132/109 - 132/109
    ci/tools/tests/test_run_lychee_precommit.py 178/146 - 178/146
    ci/tools/tests/test_select_pixi_packages.py - - 53/34
    ci/tools/tests/test_select_test_matrix.py - - 121/88
    ci/tools/tests/test_test_matrix_driver_hashes.py 38/26 - 38/26
    ci/tools/tests/test_validate_release_wheels.py - - 205/168
    ci/tools/validate-release-wheels 138/104 131/99 -
    ci/tools/validate_release_wheels.py - - 270/217
    ci/versions.json - 7/7 -
    ci/versions.yml 25/15 - 31/19
    Total 18,364/12,437 4,373/3,051 20,819/14,211
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Low priority - Nice to havecuda.bindingsEverything related to the cuda.bindings module

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions