Skip to content

Test with adversarial fixtures and check the structure and types of sparse results - #991

Merged
ViralBShah merged 3 commits into
mainfrom
vs/test-adversarial-fixtures
Oct 4, 2026
Merged

ViralBShah merged 3 commits into
mainfrom
vs/test-adversarial-fixtures

Conversation

@ViralBShah

Copy link
Copy Markdown
Member

Line coverage of src/ is about 98%, but a mutation run showed the tests check less than that suggests: 198 small realistic faults were injected into src/ one at a time (an off-by-one in a column range, a dropped conj, a stored zero read as structural, a swapped dimension check, a wrong index type), and 80 of the 193 that change behaviour passed every test in both modes. The survivors share a few causes: inputs are mostly square, real and random, so a swapped dimension or a missing conj changes nothing; and results are compared with == against dense, which cannot see a wrong element or index type or an invalid structure.

This adds two things to SparseTestHelpers and uses them across the suites:

  • mismatch(S, D) returns nothing, or a message saying how the sparse result S is not a well-formed sparse array equal to the dense reference D: not sparse, wrong size, unsorted, repeated or out-of-range indices, buffers shorter than the column pointers claim, wrong element type, wrong index type (with Ti), or different values. Used as @test mismatch(S, D) === nothing, so a failure prints the reason.
  • fixture(T, m, n) and fixturevec(T, n) are small fixed inputs with an empty row, an empty column and a stored zero, used over a tall, a wide and a square shape, and for a complex T no entry equals its conjugate.

Existing tests switch to these where they pushed a square real random matrix through an operation and compared values; a new assertion is added only where nothing tested the function. The fixtures are of the types standard mode already compiles.

Before / After, each surviving fault re-injected against the changed tests:

Before After
Faults caught by standard mode 89 of 193 most of the 80 survivors in addition (78 re-injections now fail a standard test)
Standard mode, serial, cold 250 s, 6144 methods compiled 253 s, 6303
Comprehensive mode 998 s, 18,570 1024 s, 18,989

The new inputs exposed four behaviours that differ from dense on the unmodified source. They are not fixed here:

  • UpperTriangular(Adjoint(Adjoint(S))) \ b and * b give wrong values for complex S (only reachable through the explicit constructors); recorded as @test_broken.
  • findmax/findmin along a dimension of a matrix that stores nothing return CartesianIndex(1, 1) for every slice; recorded as @test_broken.
  • findmax of an array with a stored -0.0 returns an index whose entry is not the value returned.
  • circshift! reads the buffer length instead of nnz when the source's buffers have spare capacity.

test/AGENTS.md describes the helper and the fixtures. No change to src/.

Written with Claude Code.

🤖 Generated with Claude Code

ViralBShah and others added 3 commits October 4, 2026 04:38
…d helpers

mismatch(S, D) reports what == against a dense reference cannot see: a dense
result, unsorted or repeated indices, a wrong element or index type. fixture and
fixturevec are small fixed inputs with an empty row and column, a stored zero,
non-square shapes and complex values that differ from their conjugates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…suites

Tests that pushed a square real random matrix through an operation and compared
the values with dense now take non-square, complex, stored-zero inputs and check
the result's structure, element type and index type as well. Faults that survived
a mutation run of src/ (a missing conj, a swapped dimension check, a wrong index
type, a stored zero read as structural) now fail a test.

Four behaviours that differ from dense on the unmodified source are marked broken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.57%. Comparing base (91b5d1e) to head (990f7cf).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #991      +/-   ##
==========================================
+ Coverage   95.52%   95.57%   +0.05%     
==========================================
  Files          16       16              
  Lines        9180     9182       +2     
==========================================
+ Hits         8769     8776       +7     
+ Misses        411      406       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ViralBShah
ViralBShah merged commit d352f18 into main Oct 4, 2026
10 checks passed
@ViralBShah
ViralBShah deleted the vs/test-adversarial-fixtures branch October 4, 2026 12:26
ViralBShah added a commit that referenced this pull request Oct 4, 2026
…992)

Stacked on #991; the diff here is the last commit only, and this should
merge after it.

Comprehensive mode, which the coverage CI job runs, compiled about three
times what standard mode does. A per-testset analysis (each of its 590
testsets removed in turn, with coverage and a compile trace) showed 258
of them reach no source line the rest of comprehensive mode does not,
and that the heaviest ones push one kernel through products of element
type, index type, wrapper, transform and shape. Five testsets in
`matmul.jl` compiled 950 methods for 37 lines of their own.

This cuts those guarded grids to `eachvalue` subsets or short explicit
lists, and drops types that only repeat another type's code path.
Standard mode is unchanged.

**Before / After**, whole suite in one serial process, object cache off,
the three runs side by side (before is #991):

| Comprehensive mode | Time | Methods compiled | Passing tests | `src/`
line coverage |
|---|---|---|---|---|
| Before | 1036 s | 19,008 | 27,970 | 98.16% |
| After | 656 s | 13,056 | 19,176 | 98.18% |

| Standard mode | Time | Methods compiled | Passing tests |
|---|---|---|---|
| Before and after | 249 s | 6,298 | 7,847 |

No source line that comprehensive mode covered before is uncovered after
(compared line by line from LCOV output of both runs). The cut was also
checked against fault detection: the faults from the mutation run
described in #991 that only a guarded test catches were re-injected
after the cut, and each still fails a test.

Per file, comprehensive mode (time, methods compiled; each file is
charged only for what earlier files had not compiled):

| File | Before | After |
|---|---|---|
| `higherorderfns` | 134 s, 2742 | 79 s, 1852 |
| `sparsematrix` | 110 s, 2362 | 76 s, 1594 |
| `reductions` | 92 s, 2387 | 54 s, 1579 |
| `matmul` | 92 s, 1532 | 57 s, 996 |
| `triangular` | 93 s, 1201 | 51 s, 734 |
| `concatenation` | 66 s, 1474 | 39 s, 973 |
| `linalg` | 64 s, 1225 | 38 s, 781 |
| `sparsevector` | 49 s, 1316 | 27 s, 824 |
| `indexing` | 48 s, 895 | 29 s, 576 |
| `solvers/umfpack` | 61 s, 724 | 26 s, 443 |
| `solvers/cholmod` | 60 s, 1074 | 46 s, 1012 |
| `solvers/spqr` | 39 s, 585 | 20 s, 354 |
| `fixed`, `constructors`, `solvers/solvers` | 42 s, 837 | 31 s, 653 |

Issue-regression testsets stay; where one looped over types, the loop
keeps the type the issue was about. `matmul.jl` has four edits to
unguarded conditions (`ElType === Float64` to `ElType <: Real` and
similar) so that `BigFloat` no longer takes every transform pair in
comprehensive mode; the standard-mode pass count of every suite is the
same before and after.

Both modes pass under `Pkg.test`, under the serial runner and with
`--check-bounds=yes`. No change to `src/`.

Written with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah added a commit that referenced this pull request Oct 4, 2026
… a sparse matrix (#996)

`circshift!` of a sparse matrix took the length of the value buffer for
the number of stored entries, so a source whose buffers are longer than
that gives a wrong result, silently. Found during the mutation run
described in #991.

```julia
A = sparse([1 0 2; 0 3 0; 4 0 5.0]); B = copy(A); push!(rowvals(B), 1); push!(nonzeros(B), 99.0)
O = circshift!(similar(A), B, (1, 1))
```

**Before**
```
Matrix(O) = [5.0 4.0 0.0; 99.0 1.0 0.0; 0.0 0.0 3.0]      nnz(O) = 6
expected    [5.0 4.0 0.0;  2.0 1.0 0.0; 0.0 0.0 3.0]
```

**After**
```
Matrix(O) = [5.0 4.0 0.0; 2.0 1.0 0.0; 0.0 0.0 3.0]       nnz(O) = 5
```

The column-shift branch sized the destination from `length(nonzeros(X))`
and rotated the whole buffers, which moved the unused tail into the
stored range and gave the last column an extra entry. It now uses
`nnz(X)` and rotates only the stored prefix. The local variable was also
named `nnz`, shadowing the function.

Scope: such a source is rejected elsewhere in the package
(`_goodbuffers` requires the buffers to be exactly as long as the stored
entries, so `similar(B)`, `Matrix(B)` and `copy!` assert on it), and a
shift that moves no column still goes through `copy!` and throws. This
only stops the one kernel that accepted the input from returning a wrong
answer. Whether spare capacity is a supported state is a separate
question.

One guarded test in the `circshift` testset.

Written with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah added a commit that referenced this pull request Oct 4, 2026
…negative zero (#995)

`findmax` and `argmax` of a sparse matrix with a stored `-0.0` return an
index whose entry is not the value returned, and differ from dense.
Found while trying the adversarial test fixtures of #991 with negated
values.

```julia
Z = sparse([1], [1], [-0.0], 2, 1)
```

**Before**
```
findmax(Z)         = (0.0, CartesianIndex(1, 1))      # Z[1, 1] is -0.0
findmax(Array(Z))  = (0.0, CartesianIndex(2, 1))
findmax(Z, dims=2) = ([0.0; 0.0;;], …)                # dense: [-0.0; 0.0;;]
```

**After**
```
findmax(Z)         = (0.0, CartesianIndex(2, 1))
findmax(Z, dims=2) = ([-0.0; 0.0;;], CartesianIndex{2}[CartesianIndex(1, 1); CartesianIndex(2, 1);;])
```

`_findz` looked for the first zero and counted a stored `-0.0` as one,
and `_findr` then reported `zero(Tv)` at that position. Dense compares
with `isless`, for which `-0.0 < 0.0`. `_findz` now returns the first
entry that is not stored, and a stored zero is compared like any other
stored value; each of the three scans gets a tie rule so that the
earliest of equal entries still wins, as in dense. The cost is
unchanged. `findmin` was already right, and sparse vectors use separate
code that already agrees with dense.

One guarded test compares `findmax` and `findmin` with dense, by
`isequal`, on five small matrices with stored signed zeros and a `NaN`,
for the whole array and along each dimension; 15 of its 40 comparisons
fail on `main`. An exhaustive sweep of every stored and unstored pattern
over `0.0`, `-0.0`, `NaN`, `1.0` and `-1.0` up to six cells shows no
remaining difference involving stored zeros, signed zeros or `NaN`.

Left out: along rows (`dims=2`) `_findr` does a binary search per row
per column, so that path is not O(nnz); that predates this change. This
PR and the one for a matrix that stores nothing both edit `_findr`;
whichever merges second needs a trivial rebase.

Written with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah added a commit that referenced this pull request Oct 4, 2026
Stacked on #997; the diff here is the last commit only, and this should
merge after it.Most test inputs came from `sprand`, `sprandn`, `rand`
and `randn`, usually unseeded. Which source branches a run reached
therefore varied (line coverage moved by about 0.1 point between
identical runs), a failure might not reproduce, and a random square real
matrix rarely has the structure that exposes a bug.

This replaces random inputs with fixed ones wherever the test does not
need randomness: the fixtures from #991, literals, or a matrix built so
that the property the test relies on (positive definite, nonsingular,
full rank, a nonzero diagonal, a given density) is guaranteed, not
likely.

**Before / After**

| | Before | After |
|---|---|---|
| Random constructor calls in `test/` | about 580 | 56, each seeded |
| Lines covered, two standard runs of the same commit | differ |
identical |
| Standard mode, serial, cold | 249 s, 6310 methods | 253 s, 6412 |
| Comprehensive mode | 693 s, 13,874 | 694 s, 14,020 |
| `src/` line coverage, standard / comprehensive | 97.98% / 98.18% |
98.08% / 98.23% |

Randomness stays, seeded, where it is the point: the tests of
`sprand`/`sprandn` themselves, the sort and permutation tests, a seeded
indexing stress loop, a few issue regressions, and the condition-number
estimate compared with a tolerance. `matmul.jl`, `higherorderfns.jl`,
`triangular.jl`, `concatenation.jl`, `solvers/cholmod.jl` and
`solvers/spqr.jl` have none left.

`SparseTestHelpers` gains `fixturedense(T, dims...)`, a dense operand
with values of either sign, none zero and none real for a complex type,
and `fixturestrided(T, m, n, s)`, a sparse matrix of density `1/s` for
tests that need a size or density the small fixture does not suit.

Coverage was checked file by file in both modes against a baseline taken
before the change: no line that a suite covered and no other suite
covers is lost. Where a baseline line had been reached only by a lucky
draw, a fixed input now reaches it. Loops that repeated random draws
became loops over shapes, so some pass counts change; no assertion was
weakened or removed.

Both modes pass under `Pkg.test`, under the serial runner and with
`--check-bounds=yes`. No change to `src/`.

Written with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah added a commit that referenced this pull request Oct 4, 2026
…g a dimension of a sparse matrix that stores nothing (#994)

`findmax`, `findmin`, `argmax` and `argmin` along a dimension of a
sparse matrix that stores nothing return `CartesianIndex(1, 1)` for
every slice. Found by the adversarial test fixtures added in #991, where
it is recorded as `@test_broken`.

**Before**
```
julia> findmax(spzeros(3, 2); dims=2)[2]
3×1 Matrix{CartesianIndex{2}}:
 CartesianIndex(1, 1)
 CartesianIndex(1, 1)
 CartesianIndex(1, 1)

julia> findmin(spzeros(3, 2); dims=1)[2]
1×2 Matrix{CartesianIndex{2}}:
 CartesianIndex(1, 1)  CartesianIndex(1, 1)
```

**After**, as for `zeros(3, 2)`
```
julia> findmax(spzeros(3, 2); dims=2)[2]
3×1 Matrix{CartesianIndex{2}}:
 CartesianIndex(1, 1)
 CartesianIndex(2, 1)
 CartesianIndex(3, 1)

julia> findmin(spzeros(3, 2); dims=1)[2]
1×2 Matrix{CartesianIndex{2}}:
 CartesianIndex(1, 1)  CartesianIndex(1, 2)
```

The early returns of `_findr` for a matrix without stored entries filled
the index array with the first index of the matrix. They now return the
first index of each slice, in time proportional to the number of slices.
The `@test_broken` in `test/reductions.jl` becomes a `@test` over
`findmax` and `findmin` for both dimensions.

Left out: a complex matrix that stores nothing returns a result where
dense throws (no `isless`), and `dims=3` or `(2, 1)` throw where dense
accepts them. This PR and the one for a stored `-0.0` both edit
`_findr`; whichever merges second needs a trivial rebase.

Written with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah added a commit that referenced this pull request Oct 4, 2026
…se parent (#993)

A triangular solve or product whose parent is a wrapper applied twice by
its constructor gives wrong values for a complex sparse matrix.
`adjoint(adjoint(S))` unwraps, so only the explicit constructors reach
this. Found by the adversarial test fixtures added in #991, where it is
recorded as `@test_broken`.

```julia
S = sparse(ComplexF64[2+im 1+2im; 3-im 2+im]); b = [1.0, 1.0]; Y = [1.0 1.0]
X = UpperTriangular(Adjoint(Adjoint(S)))   # the matrix [2+im 1+2im; 0 2+im]
```

**Before**
```
X \ b = ComplexF64[0.4 + 0.2im, 0.4 + 0.2im]      Matrix(X) \ b = ComplexF64[-0.04 - 0.28im, 0.4 - 0.2im]
X * b = ComplexF64[2.0 - 1.0im, 2.0 - 1.0im]      Matrix(X) * b = ComplexF64[3.0 + 3.0im, 2.0 + 1.0im]
Y * X = ComplexF64[5.0 + 0.0im 2.0 - 1.0im]       Y * Matrix(X) = ComplexF64[2.0 + 1.0im 3.0 + 3.0im]
```

**After**, all three equal the dense results; so do `ldiv!`, `lmul!` and
`rmul!`.

LinearAlgebra calls `generic_trimatdiv!`, `generic_trimatmul!` and
`generic_mattrimul!` with the outer wrapper as `tfun` and the inner
wrapper still on the matrix. For a mixed pair (`Transpose(Adjoint(S))`)
the matrix is `conj(S)` and the triangle is passed unflipped, which is
what the hooks assumed. For the same wrapper twice the matrix is `S`
itself and LinearAlgebra passes the flipped triangle; the hooks ignored
`tfun`, so they conjugated and used the wrong triangle. They now use
`identity` and the opposite triangle in that case. The kernels are
unchanged.

Left out, because the wrong values come from LinearAlgebra
(JuliaLang/LinearAlgebra.jl#1736):
- **Real eltypes are still wrong** for the same parents. LinearAlgebra
passes `transpose` for either outer wrapper of a real matrix, so the
hook receives identical arguments for `Adjoint(Adjoint(R))` and
`Transpose(Adjoint(R))` apart from a triangle character that is wrong
for one of them. A real pair that is right today stays right.
- `Y / X`, for which SparseArrays has no hook, and a triangle times a
sparse matrix (`X * S`, `S * X`), which gets a sparse destination and so
runs LinearAlgebra's generic kernel instead of these hooks. `X \ S`, `X
* sparsevec` and `mul!` into a dense destination are correct.
- Dense parents and triple wrappers.

The two `@test_broken` lines in `test/triangular.jl` become `@test`, and
the same loop gains the right multiplication. A sweep of 13,440
sparse-versus-dense checks over 20 operations, the single, mixed and
double wrappers, both eltypes and all four triangle types shows nothing
that matched dense before differing after, and the remaining differences
are the ones listed above. Touches dispatch only in that the three hooks
now name their `tfun` argument.

Written with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah added a commit that referenced this pull request Oct 9, 2026
…ember (#1017)

Documentation only, drawn from the pull requests merged since early
September and their review comments.

- **PR text.** The rule against reporting routine checks is its own
bullet and names the forms that still got through: test modes passing,
the ambiguity check, the Julia build, pass counts. Titles fit one line
of `git log --oneline`.
- **Added.** Julia's CI flags, `--depwarn=error` and the trim test
(#971, #972); dense results where every entry would be stored (#1004,
#1005, #1007); opaque elements (#982, #977, #1006); fixed test inputs
and `mismatch` (#991, #998).
- **Removed.** Rules that Aqua, `.gitignore` or #990 made unnecessary,
and test detail that `test/AGENTS.md` already holds.
- **`src/solvers/`.** Caller-owned solve workspaces (#914), copying an
aliased right-hand side (#942), validating before a refactorization
(#899, #901). Corrects the lock field names and the index-type testing
rule.
- **`test/`.** Fixed inputs and top-level helper loading (#971); drops
the notes on worker groups, which `runtests.jl` no longer has.
- **`gen/`.** Header order, and running `make` before a change (#1003).

Written by Claude Code (Claude Opus 5.5).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Viral B. Shah <ViralBShah@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant