Skip to content

Compile spilling KernelInterface kernels for SIMD16 - #773

Draft
maleadt wants to merge 2 commits into
mainfrom
tb/simd32
Draft

maleadt wants to merge 2 commits into
mainfrom
tb/simd32

Conversation

@maleadt

@maleadt maleadt commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Fixes #772.

KernelInterface kernels, and so every KernelAbstractions @kernel, are compiled for the sub-group width that KI.sub_group_size promises: 32 where the device supports it. Register-heavy kernels spill heavily at that width. For the register-blocked matmul from JuliaGPU/GPUArrays.jl#856, launched through KernelInterface at 1024³:

main (SIMD32) this PR (SIMD16)
Iris Xe, Float32 8.59 ms, 1856 B spill 5.16 ms, no spill
Iris Xe, ComplexF32 82.3 ms, 5568 B spill 15.7 ms, 1056 B spill
Arc A770, Float32 0.99 ms, 1152 B spill 0.63 ms, no spill
Arc A770, Int32 1.29 ms, 1152 B spill 0.82 ms, no spill
Arc A770, ComplexF32 6.76 ms, 4768 B spill 1.34 ms, 576 B spill

Promising 16 instead doesn't help in general: kernels that don't spill are faster at 32. On Iris Xe, GPUArrays' tiled matmul runs 20% slower at 16, and AcceleratedKernels' reductions and scans 10–20% slower. Leaving the width to IGC isn't better either. Without a required size, IGC compiles the blocked matmul for SIMD8 on Xe-LP and Xe-HPG (7.4 ms vs 4.9 ms at SIMD16 on Iris Xe). On Xe-HPC it keeps a spilling SIMD32. And kernels that use sub-groups need the promised width: AcceleratedKernels sizes local memory from it, and returns wrong sums without it.

So the promise stays, but only for kernels that can observe it. When a KernelInterface kernel spills at the promised width, its SPIR-V is checked for anything that could make the width observable:

  • a capability other than the ones kernels use without sub-groups (all sub-group operations need their own);
  • a sub-group built-in variable, sub-group execution mode or kernel query;
  • a barrier, fence or atomic whose scope is the sub-group or isn't a constant.

If there is none, the kernel is compiled again for SIMD16. That version is used unless the driver limits it to smaller work-groups, or doesn't report its limit. The check runs on the emitted SPIR-V rather than LLVM IR, so that it fails closed: anything it doesn't recognize keeps the promised width. Kernels that don't spill compile exactly as before, and so do kernels launched with @oneapi. A spilling kernel is compiled twice the first time it is used: on Iris Xe, the blocked matmul takes 0.5–0.6 s instead of 0.2–0.3 s.

The tests check:

  • that every KernelInterface sub-group function, and sub-group-scope fences and atomics, are detected, and that work-group barriers and device-scope atomics are not;
  • that a spilling kernel gets SIMD16 while a variant that reads its width keeps the promised one and observes it on the device, and that a kernel that doesn't spill keeps it too;
  • that malformed SPIR-V counts as observing.

Validated on Iris Xe and Arc A770, including GPUArrays' sum, cumsum, findmax/findmin, sum(dims), permutedims and matmul, which go through AcceleratedKernels' sub-group kernels.

Follow-ups, not in this PR:

  • A way for kernels that depend on their width to request one, so AcceleratedKernels can pick 16 where its sub-group kernels are faster (e.g. on DG2: sum 1.83 → 1.17 ms) and 32 where they aren't (Xe-LP). That is a KernelInterface change; see KernelInterface 1.0: a better API for writing kernels KernelAbstractions.jl#810, item 4.
  • OpenCL.jl pins the width of every kernel, including plain @opencl ones, and has the same problem on Intel GPUs.

So that a launch can link more than one job for the same kernel, and look at
the cached results of each.
KernelInterface kernels were all compiled for the sub-group width that
`KI.sub_group_size` promises, 32 where the device supports it. Register-heavy
kernels spill heavily at that width: a register-blocked matmul runs 1.6x
(Float32) to 5x (ComplexF32) slower than at SIMD16 on Xe-LP and Xe-HPG.
Forcing 16 for every kernel isn't better: kernels that don't spill are up to
~20% faster at 32, and IGC's own choice, when no width is required, drops such
kernels to SIMD8.

Only kernels that can observe the width need the promised one. When a
KernelInterface kernel spills at that width, check its SPIR-V for anything
that could make the width observable: a capability beyond the ones kernels use
without sub-groups, a sub-group built-in variable or kernel query, or a
barrier, fence or atomic whose scope is the sub-group or not a constant. If
there is none, compile the kernel again for SIMD16, and use that unless the
driver limits it to smaller work-groups or doesn't report its limit. Kernels
launched with `@oneapi` keep the width they ask for.

Fixes #772.
@maleadt maleadt added performance Runtime performance. kernels Writing and compiling kernels: device intrinsics, codegen, KernelAbstractions back-end. labels Nov 19, 2026
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.87%. Comparing base (7ed9854) to head (99b9797).

Files with missing lines Patch % Lines
src/compiler/compilation.jl 91.66% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #773      +/-   ##
==========================================
- Coverage   87.34%   86.87%   -0.48%     
==========================================
  Files          56       55       -1     
  Lines        5263     5180      -83     
==========================================
- Hits         4597     4500      -97     
- Misses        666      680      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@maleadt
maleadt marked this pull request as draft November 20, 2026 08:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernels Writing and compiling kernels: device intrinsics, codegen, KernelAbstractions back-end. performance Runtime performance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KernelInterface kernels are always compiled for SIMD32, which makes register-heavy kernels spill

1 participant