Repository navigation
Test with adversarial fixtures and check the structure and types of sparse results - #991
Merged
Merged
Conversation
…d helpers mismatch(S, D) reports what == against a dense reference cannot see: a dense result, unsorted or repeated indices, a wrong element or index type. fixture and fixturevec are small fixed inputs with an empty row and column, a stored zero, non-square shapes and complex values that differ from their conjugates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…suites Tests that pushed a square real random matrix through an operation and compared the values with dense now take non-square, complex, stored-zero inputs and check the result's structure, element type and index type as well. Faults that survived a mutation run of src/ (a missing conj, a swapped dimension check, a wrong index type, a stored zero read as structural) now fail a test. Four behaviours that differ from dense on the unmodified source are marked broken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #991 +/- ##
==========================================
+ Coverage 95.52% 95.57% +0.05%
==========================================
Files 16 16
Lines 9180 9182 +2
==========================================
+ Hits 8769 8776 +7
+ Misses 411 406 -5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ViralBShah
added a commit
that referenced
this pull request
Oct 4, 2026
…992) Stacked on #991; the diff here is the last commit only, and this should merge after it. Comprehensive mode, which the coverage CI job runs, compiled about three times what standard mode does. A per-testset analysis (each of its 590 testsets removed in turn, with coverage and a compile trace) showed 258 of them reach no source line the rest of comprehensive mode does not, and that the heaviest ones push one kernel through products of element type, index type, wrapper, transform and shape. Five testsets in `matmul.jl` compiled 950 methods for 37 lines of their own. This cuts those guarded grids to `eachvalue` subsets or short explicit lists, and drops types that only repeat another type's code path. Standard mode is unchanged. **Before / After**, whole suite in one serial process, object cache off, the three runs side by side (before is #991): | Comprehensive mode | Time | Methods compiled | Passing tests | `src/` line coverage | |---|---|---|---|---| | Before | 1036 s | 19,008 | 27,970 | 98.16% | | After | 656 s | 13,056 | 19,176 | 98.18% | | Standard mode | Time | Methods compiled | Passing tests | |---|---|---|---| | Before and after | 249 s | 6,298 | 7,847 | No source line that comprehensive mode covered before is uncovered after (compared line by line from LCOV output of both runs). The cut was also checked against fault detection: the faults from the mutation run described in #991 that only a guarded test catches were re-injected after the cut, and each still fails a test. Per file, comprehensive mode (time, methods compiled; each file is charged only for what earlier files had not compiled): | File | Before | After | |---|---|---| | `higherorderfns` | 134 s, 2742 | 79 s, 1852 | | `sparsematrix` | 110 s, 2362 | 76 s, 1594 | | `reductions` | 92 s, 2387 | 54 s, 1579 | | `matmul` | 92 s, 1532 | 57 s, 996 | | `triangular` | 93 s, 1201 | 51 s, 734 | | `concatenation` | 66 s, 1474 | 39 s, 973 | | `linalg` | 64 s, 1225 | 38 s, 781 | | `sparsevector` | 49 s, 1316 | 27 s, 824 | | `indexing` | 48 s, 895 | 29 s, 576 | | `solvers/umfpack` | 61 s, 724 | 26 s, 443 | | `solvers/cholmod` | 60 s, 1074 | 46 s, 1012 | | `solvers/spqr` | 39 s, 585 | 20 s, 354 | | `fixed`, `constructors`, `solvers/solvers` | 42 s, 837 | 31 s, 653 | Issue-regression testsets stay; where one looped over types, the loop keeps the type the issue was about. `matmul.jl` has four edits to unguarded conditions (`ElType === Float64` to `ElType <: Real` and similar) so that `BigFloat` no longer takes every transform pair in comprehensive mode; the standard-mode pass count of every suite is the same before and after. Both modes pass under `Pkg.test`, under the serial runner and with `--check-bounds=yes`. No change to `src/`. Written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
ViralBShah
added a commit
that referenced
this pull request
Oct 4, 2026
… a sparse matrix (#996) `circshift!` of a sparse matrix took the length of the value buffer for the number of stored entries, so a source whose buffers are longer than that gives a wrong result, silently. Found during the mutation run described in #991. ```julia A = sparse([1 0 2; 0 3 0; 4 0 5.0]); B = copy(A); push!(rowvals(B), 1); push!(nonzeros(B), 99.0) O = circshift!(similar(A), B, (1, 1)) ``` **Before** ``` Matrix(O) = [5.0 4.0 0.0; 99.0 1.0 0.0; 0.0 0.0 3.0] nnz(O) = 6 expected [5.0 4.0 0.0; 2.0 1.0 0.0; 0.0 0.0 3.0] ``` **After** ``` Matrix(O) = [5.0 4.0 0.0; 2.0 1.0 0.0; 0.0 0.0 3.0] nnz(O) = 5 ``` The column-shift branch sized the destination from `length(nonzeros(X))` and rotated the whole buffers, which moved the unused tail into the stored range and gave the last column an extra entry. It now uses `nnz(X)` and rotates only the stored prefix. The local variable was also named `nnz`, shadowing the function. Scope: such a source is rejected elsewhere in the package (`_goodbuffers` requires the buffers to be exactly as long as the stored entries, so `similar(B)`, `Matrix(B)` and `copy!` assert on it), and a shift that moves no column still goes through `copy!` and throws. This only stops the one kernel that accepted the input from returning a wrong answer. Whether spare capacity is a supported state is a separate question. One guarded test in the `circshift` testset. Written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah
added a commit
that referenced
this pull request
Oct 4, 2026
…negative zero (#995) `findmax` and `argmax` of a sparse matrix with a stored `-0.0` return an index whose entry is not the value returned, and differ from dense. Found while trying the adversarial test fixtures of #991 with negated values. ```julia Z = sparse([1], [1], [-0.0], 2, 1) ``` **Before** ``` findmax(Z) = (0.0, CartesianIndex(1, 1)) # Z[1, 1] is -0.0 findmax(Array(Z)) = (0.0, CartesianIndex(2, 1)) findmax(Z, dims=2) = ([0.0; 0.0;;], …) # dense: [-0.0; 0.0;;] ``` **After** ``` findmax(Z) = (0.0, CartesianIndex(2, 1)) findmax(Z, dims=2) = ([-0.0; 0.0;;], CartesianIndex{2}[CartesianIndex(1, 1); CartesianIndex(2, 1);;]) ``` `_findz` looked for the first zero and counted a stored `-0.0` as one, and `_findr` then reported `zero(Tv)` at that position. Dense compares with `isless`, for which `-0.0 < 0.0`. `_findz` now returns the first entry that is not stored, and a stored zero is compared like any other stored value; each of the three scans gets a tie rule so that the earliest of equal entries still wins, as in dense. The cost is unchanged. `findmin` was already right, and sparse vectors use separate code that already agrees with dense. One guarded test compares `findmax` and `findmin` with dense, by `isequal`, on five small matrices with stored signed zeros and a `NaN`, for the whole array and along each dimension; 15 of its 40 comparisons fail on `main`. An exhaustive sweep of every stored and unstored pattern over `0.0`, `-0.0`, `NaN`, `1.0` and `-1.0` up to six cells shows no remaining difference involving stored zeros, signed zeros or `NaN`. Left out: along rows (`dims=2`) `_findr` does a binary search per row per column, so that path is not O(nnz); that predates this change. This PR and the one for a matrix that stores nothing both edit `_findr`; whichever merges second needs a trivial rebase. Written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah
added a commit
that referenced
this pull request
Oct 4, 2026
Stacked on #997; the diff here is the last commit only, and this should merge after it.Most test inputs came from `sprand`, `sprandn`, `rand` and `randn`, usually unseeded. Which source branches a run reached therefore varied (line coverage moved by about 0.1 point between identical runs), a failure might not reproduce, and a random square real matrix rarely has the structure that exposes a bug. This replaces random inputs with fixed ones wherever the test does not need randomness: the fixtures from #991, literals, or a matrix built so that the property the test relies on (positive definite, nonsingular, full rank, a nonzero diagonal, a given density) is guaranteed, not likely. **Before / After** | | Before | After | |---|---|---| | Random constructor calls in `test/` | about 580 | 56, each seeded | | Lines covered, two standard runs of the same commit | differ | identical | | Standard mode, serial, cold | 249 s, 6310 methods | 253 s, 6412 | | Comprehensive mode | 693 s, 13,874 | 694 s, 14,020 | | `src/` line coverage, standard / comprehensive | 97.98% / 98.18% | 98.08% / 98.23% | Randomness stays, seeded, where it is the point: the tests of `sprand`/`sprandn` themselves, the sort and permutation tests, a seeded indexing stress loop, a few issue regressions, and the condition-number estimate compared with a tolerance. `matmul.jl`, `higherorderfns.jl`, `triangular.jl`, `concatenation.jl`, `solvers/cholmod.jl` and `solvers/spqr.jl` have none left. `SparseTestHelpers` gains `fixturedense(T, dims...)`, a dense operand with values of either sign, none zero and none real for a complex type, and `fixturestrided(T, m, n, s)`, a sparse matrix of density `1/s` for tests that need a size or density the small fixture does not suit. Coverage was checked file by file in both modes against a baseline taken before the change: no line that a suite covered and no other suite covers is lost. Where a baseline line had been reached only by a lucky draw, a fixed input now reaches it. Loops that repeated random draws became loops over shapes, so some pass counts change; no assertion was weakened or removed. Both modes pass under `Pkg.test`, under the serial runner and with `--check-bounds=yes`. No change to `src/`. Written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah
added a commit
that referenced
this pull request
Oct 4, 2026
…g a dimension of a sparse matrix that stores nothing (#994) `findmax`, `findmin`, `argmax` and `argmin` along a dimension of a sparse matrix that stores nothing return `CartesianIndex(1, 1)` for every slice. Found by the adversarial test fixtures added in #991, where it is recorded as `@test_broken`. **Before** ``` julia> findmax(spzeros(3, 2); dims=2)[2] 3×1 Matrix{CartesianIndex{2}}: CartesianIndex(1, 1) CartesianIndex(1, 1) CartesianIndex(1, 1) julia> findmin(spzeros(3, 2); dims=1)[2] 1×2 Matrix{CartesianIndex{2}}: CartesianIndex(1, 1) CartesianIndex(1, 1) ``` **After**, as for `zeros(3, 2)` ``` julia> findmax(spzeros(3, 2); dims=2)[2] 3×1 Matrix{CartesianIndex{2}}: CartesianIndex(1, 1) CartesianIndex(2, 1) CartesianIndex(3, 1) julia> findmin(spzeros(3, 2); dims=1)[2] 1×2 Matrix{CartesianIndex{2}}: CartesianIndex(1, 1) CartesianIndex(1, 2) ``` The early returns of `_findr` for a matrix without stored entries filled the index array with the first index of the matrix. They now return the first index of each slice, in time proportional to the number of slices. The `@test_broken` in `test/reductions.jl` becomes a `@test` over `findmax` and `findmin` for both dimensions. Left out: a complex matrix that stores nothing returns a result where dense throws (no `isless`), and `dims=3` or `(2, 1)` throw where dense accepts them. This PR and the one for a stored `-0.0` both edit `_findr`; whichever merges second needs a trivial rebase. Written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah
added a commit
that referenced
this pull request
Oct 4, 2026
…se parent (#993) A triangular solve or product whose parent is a wrapper applied twice by its constructor gives wrong values for a complex sparse matrix. `adjoint(adjoint(S))` unwraps, so only the explicit constructors reach this. Found by the adversarial test fixtures added in #991, where it is recorded as `@test_broken`. ```julia S = sparse(ComplexF64[2+im 1+2im; 3-im 2+im]); b = [1.0, 1.0]; Y = [1.0 1.0] X = UpperTriangular(Adjoint(Adjoint(S))) # the matrix [2+im 1+2im; 0 2+im] ``` **Before** ``` X \ b = ComplexF64[0.4 + 0.2im, 0.4 + 0.2im] Matrix(X) \ b = ComplexF64[-0.04 - 0.28im, 0.4 - 0.2im] X * b = ComplexF64[2.0 - 1.0im, 2.0 - 1.0im] Matrix(X) * b = ComplexF64[3.0 + 3.0im, 2.0 + 1.0im] Y * X = ComplexF64[5.0 + 0.0im 2.0 - 1.0im] Y * Matrix(X) = ComplexF64[2.0 + 1.0im 3.0 + 3.0im] ``` **After**, all three equal the dense results; so do `ldiv!`, `lmul!` and `rmul!`. LinearAlgebra calls `generic_trimatdiv!`, `generic_trimatmul!` and `generic_mattrimul!` with the outer wrapper as `tfun` and the inner wrapper still on the matrix. For a mixed pair (`Transpose(Adjoint(S))`) the matrix is `conj(S)` and the triangle is passed unflipped, which is what the hooks assumed. For the same wrapper twice the matrix is `S` itself and LinearAlgebra passes the flipped triangle; the hooks ignored `tfun`, so they conjugated and used the wrong triangle. They now use `identity` and the opposite triangle in that case. The kernels are unchanged. Left out, because the wrong values come from LinearAlgebra (JuliaLang/LinearAlgebra.jl#1736): - **Real eltypes are still wrong** for the same parents. LinearAlgebra passes `transpose` for either outer wrapper of a real matrix, so the hook receives identical arguments for `Adjoint(Adjoint(R))` and `Transpose(Adjoint(R))` apart from a triangle character that is wrong for one of them. A real pair that is right today stays right. - `Y / X`, for which SparseArrays has no hook, and a triangle times a sparse matrix (`X * S`, `S * X`), which gets a sparse destination and so runs LinearAlgebra's generic kernel instead of these hooks. `X \ S`, `X * sparsevec` and `mul!` into a dense destination are correct. - Dense parents and triple wrappers. The two `@test_broken` lines in `test/triangular.jl` become `@test`, and the same loop gains the right multiplication. A sweep of 13,440 sparse-versus-dense checks over 20 operations, the single, mixed and double wrappers, both eltypes and all four triangle types shows nothing that matched dense before differing after, and the remaining differences are the ones listed above. Touches dispatch only in that the three hooks now name their `tfun` argument. Written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ViralBShah
added a commit
that referenced
this pull request
Oct 9, 2026
…ember (#1017) Documentation only, drawn from the pull requests merged since early September and their review comments. - **PR text.** The rule against reporting routine checks is its own bullet and names the forms that still got through: test modes passing, the ambiguity check, the Julia build, pass counts. Titles fit one line of `git log --oneline`. - **Added.** Julia's CI flags, `--depwarn=error` and the trim test (#971, #972); dense results where every entry would be stored (#1004, #1005, #1007); opaque elements (#982, #977, #1006); fixed test inputs and `mismatch` (#991, #998). - **Removed.** Rules that Aqua, `.gitignore` or #990 made unnecessary, and test detail that `test/AGENTS.md` already holds. - **`src/solvers/`.** Caller-owned solve workspaces (#914), copying an aliased right-hand side (#942), validating before a refactorization (#899, #901). Corrects the lock field names and the index-type testing rule. - **`test/`.** Fixed inputs and top-level helper loading (#971); drops the notes on worker groups, which `runtests.jl` no longer has. - **`gen/`.** Header order, and running `make` before a change (#1003). Written by Claude Code (Claude Opus 5.5). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Viral B. Shah <ViralBShah@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Line coverage of
src/is about 98%, but a mutation run showed the tests check less than that suggests: 198 small realistic faults were injected intosrc/one at a time (an off-by-one in a column range, a droppedconj, a stored zero read as structural, a swapped dimension check, a wrong index type), and 80 of the 193 that change behaviour passed every test in both modes. The survivors share a few causes: inputs are mostly square, real and random, so a swapped dimension or a missingconjchanges nothing; and results are compared with==against dense, which cannot see a wrong element or index type or an invalid structure.This adds two things to
SparseTestHelpersand uses them across the suites:mismatch(S, D)returnsnothing, or a message saying how the sparse resultSis not a well-formed sparse array equal to the dense referenceD: not sparse, wrong size, unsorted, repeated or out-of-range indices, buffers shorter than the column pointers claim, wrong element type, wrong index type (withTi), or different values. Used as@test mismatch(S, D) === nothing, so a failure prints the reason.fixture(T, m, n)andfixturevec(T, n)are small fixed inputs with an empty row, an empty column and a stored zero, used over a tall, a wide and a square shape, and for a complexTno entry equals its conjugate.Existing tests switch to these where they pushed a square real random matrix through an operation and compared values; a new assertion is added only where nothing tested the function. The fixtures are of the types standard mode already compiles.
Before / After, each surviving fault re-injected against the changed tests:
The new inputs exposed four behaviours that differ from dense on the unmodified source. They are not fixed here:
UpperTriangular(Adjoint(Adjoint(S))) \ band* bgive wrong values for complexS(only reachable through the explicit constructors); recorded as@test_broken.findmax/findminalong a dimension of a matrix that stores nothing returnCartesianIndex(1, 1)for every slice; recorded as@test_broken.findmaxof an array with a stored-0.0returns an index whose entry is not the value returned.circshift!reads the buffer length instead ofnnzwhen the source's buffers have spare capacity.test/AGENTS.mddescribes the helper and the fixtures. No change tosrc/.Written with Claude Code.
🤖 Generated with Claude Code