[lizthegrey]

Description

Speeds up bitmapContainerManyIterator.nextMany and nextMany64 on arm64. All other targets are unchanged.

Type of Change

• Performance improvement

Changes Made

What was changed?

• On arm64 only, decode set bits with bits.TrailingZeros64 and clear them with x &= x - 1, and drain a whole word in a tight loop when the output buffer has room for 64 values.
• The new code sits behind a compile-time constant, decodeWithTrailingZeros, which is true only in decodewords_arm64.go.
• Added TestBitmapContainerNextMany64BufferSizes and BenchmarkBitmapContainerNextMany64.

Why was it changed?

• arm64 has no scalar popcount, so bits.OnesCount64(t-1) compiles to FMOV/VCNT/UADDLV/FMOV for every set bit, where TrailingZeros64 is RBIT/CLZ.

Relationship to #553

The arm64 loop is the one proposed in #553, and it measures the same as #553 on all three Graviton generations below. The difference is scope: #553 changed every architecture, and I could not justify that for amd64 (see below). #553 has since been closed in favour of this PR. Its author re-ran it on Zen 3 and concluded that a separate amd64 path was not worth it.

Testing

go test ./... passes with GOAMD64=v1 and GOAMD64=v3. The iterator tests also pass natively on Neoverse N1, V1 and V2 (-test.run 'NextMany|ManyIterator|Iterator').

Fuzzing

Not run. The template's go test -run='^$' -fuzz=FuzzSmat reports "no fuzz tests to fuzz", as no FuzzSmat target is defined in the repository. The change does not touch deserialization. Coverage comes from the buffer-size tests, which cross the 64-slot drain boundary with buffers of 63, 64 and 65 entries and compare against a reference decode.

Performance Impact

sec/op change against master, go1.27.3, taskset-pinned, 10 rounds interleaved with master, benchstat (all p=0.000):

Benchmark	Neoverse N1 (c6g)	Neoverse V1 (c7g)	Neoverse V2 (c8g)
BitmapContainerNextMany/card=4096/buf=4096	−21.5%	−26.1%	−43.5%
BitmapContainerNextMany/card=16384/buf=4096	−26.9%	−49.6%	−37.1%
BitmapContainerNextMany/card=32768/buf=4096	−22.7%	−43.9%	−29.9%
BitmapContainerNextMany/card=65536/buf=4096	−24.8%	−40.2%	−30.5%
BitmapContainerNextMany/card=32768/buf=512	−21.3%	−41.2%	−30.9%
BitmapContainerNextMany/card=32768/buf=64	−5.8%	−20.6%	−23.7%
BitmapContainerNextMany64 (card=32768, buf=4096)	−19.7%	−45.2%	−25.8%

Why amd64 is left alone

The machine code of nextMany and nextMany64 is byte-identical to master at GOAMD64=v1 and v3 (compared with go tool objdump), so amd64 performance does not change.

I tried to extend this to amd64 and the results did not support it. Measured on Cascade Lake, Ice Lake, Zen 3 (EPYC 7R13 and Ryzen 5600X) and Zen 4:

• Optimize bitmap container batch iteration #553 at the default GOAMD64=v1 was 10–30% slower on most nextMany cases on Zen 3, but up to 50% faster on Cascade Lake and on most cases on Ice Lake and Zen 4.
• Functionally equivalent versions of the loop gave very different results. At v1, a version using the original popcount sequence through an inlined helper, with no TrailingZeros64 at all, was still 12–20% slower than master on most nextMany cases on Zen 3. At v3 on Zen 3, two equivalent forms gave −12% and −40%.

So the amd64 effect appears to depend on register allocation and layout as much as on the instruction chosen. I don't have a cause I can defend, and someone with a better handle on amd64 may want to revisit it. Other non-arm64 architectures were not measured and are also unchanged.

🤖 Generated with Claude Code

[lemire]

We will merge this.