Skip to content

perf(api): load the relations of prefetched duplicate findings once per page - #16789

Merged
Maffooch merged 4 commits into
devfrom
perf/prefetch-duplicate-finding-batch
Nov 11, 2026
Merged

Maffooch merged 4 commits into
devfrom
perf/prefetch-duplicate-finding-batch

Conversation

@Maffooch

@Maffooch Maffooch commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

GET /api/v2/findings/?prefetch=duplicate_finding cost about 25 queries for every distinct original the page's duplicates point at. On a 1,000-row page with 20 distinct originals that came to 549 queries, against 52 for the same page once each original's relations are loaded together.

For a single related object the prefetcher serializes the entry's own instance with the full FindingSerializer. Nothing loaded that instance's relations, so each original read its tags, notes (with authors, note types and history), endpoints or locations, finding groups and their JIRA issues, CWEs, vulnerability ids, risk acceptances, files, request/response pairs, metadata, reviewers, found_by, its test's type and, for a non-superuser, the product its accepted_risks permission check walks to, one query at a time.

This PR:

  • Registers Finding in _PREFETCH_PREFETCH_RELATED with the relations FindingSerializer reads. The entry is a callable, built at each call, because endpoints reads locations under V3 and the notes and vulnerability id lookups are Prefetch objects (notes_prefetch(), vulnerability_id_prefetch()). _prefetch_related_for() resolves an entry for both the single and the many path.
  • Has _prefetch_single reuse one serializer for every object, as a list serializer reuses its child. Building a FindingSerializer's fields was most of the time left once the queries were batched (about 0.3 s of 0.5 s for 20 originals under a profiler), and none of it depends on the object.
  • Has RiskAcceptanceSerializer.get_path skip the engagement lookup when the acceptance has no proof, and read the acceptance's engagement_set instead of Engagement.objects.filter(...).first(), so the prefetch serves it. It names the same engagement as before: Engagement's manager (BaseManager) orders by id, prefetched or not, so both pick the lowest id. (Its Meta.ordering of -target_start is overridden by the manager; the test pins the lowest-id behaviour.)
  • Loads what accepted_risks reads (each acceptance's notes, engagements and accepted findings) only onto the findings it lists acceptances for. FindingSerializer.shows_accepted_risks() is now the check get_accepted_risks makes (the requester may edit the finding), and _PREFETCH_PREFETCH_RELATED_WHEN_SHOWN applies the loader only to the findings it accepts, on both paths. Without that gate, a requester who may only view would load every acceptance, accepted finding included, and get [].
  • Reads those accepted findings as (acceptance, finding id) pairs, through Finding.objects so the relation's own filters and id order apply, and gives each acceptance its ids as the prefetched relation (accepted_findings renders ids only). Prefetched as whole rows, an acceptance holding 100,001 findings meant 100,001 model instances built to print their ids. Over several acceptances at once, that query also merge-joined along dojo_finding's primary key: 612 ms of SQL, against 225 ms when one acceptance is read on its own.

Because the many path now also applies the Finding entry to the authorized queryset, a list of findings prefetched through a to-many relation (for example ?prefetch=findings on endpoints) is batched the same way.

The response body is unchanged, byte for byte.

Test results

New unittests/test_apiv2_prefetch_duplicate_finding.py. Each original carries two tags, a public note with history and a private note, an endpoint (V2) or a location (V3), a vulnerability id, a CWE, a finding group with a JIRA issue, its own JIRA issue, a risk acceptance with a proof, a reviewer, a found_by, a file, a request/response pair and a metadata row. JIRA is enabled with a project and instance.

  • ?prefetch=duplicate_finding with 2 originals, then 6, costs the same, for a superuser and for a product member. On dev it failed: 58 against 170 queries (superuser), 59 against 171 (member).
  • The payload with the change is byte-identical (JSONRenderer) to the payload with the old _prefetch_single, no Finding entry and the old get_path, for both users.
  • get_path: no proof needs no query (1 on dev); a proof with no engagement still says "No proof has been supplied"; with two engagements it names the lowest id, matching the old query; a prefetched engagement_set serves it with no query (1 on dev).
  • A requester who may view but not edit (simulated by denying edit in the serializer's permission check), with acceptances holding three more findings each: the single path reads no risk acceptance table, as on dev (it read 3 tables before the gate). The many path (?prefetch=accepted_findings on the acceptances) reads them only for the field itself, as on dev (4 reads against 1 before the gate). Bodies are identical to the old code. An editor on the many path still gets every acceptance with its four findings.
  • The batch reads accepted findings as id pairs, never whole finding rows, and each acceptance's ids equal what its own relation returns, in the same order.

Run with DD_V3_FEATURE_LOCATIONS True and False:

  • the new module: 11 passed in both modes. 4 of the first 7 fail on dev for the reasons above, and the two requester-without-edit tests fail on this PR's first commit.
  • test_api_product_prefetch, test_apiv2_list_query_scaling, test_apiv2_prefetch_*, test_bulk_risk_acceptance_api, test_risk_acceptance*, test_finding_model and test_rest_framework (which requests ?prefetch= on every viewset): 1087 tests, OK in both modes.
  • ruff check (0.16.10) on the changed files: clean.

Measured on a 10M-finding database (one in fifty findings past a hot range is a duplicate; the 20 originals were given the relations above), GET /api/v2/findings/?prefetch=duplicate_finding at offset 500000, old and new code in one process, alternating with the order flipped each run, 7 runs, medians:

Rows (distinct originals) User Queries Prefetch block wall ms Prefetch block SQL ms Request wall ms
100 (2) superuser 81 -> 52 66.5 -> 46.5 16.0 -> 10.3 729 -> 724
1000 (20) superuser 549 -> 52 663 -> 114 166 -> 13 6381 -> 5830
100 (2) product type member 87 -> 56 73.6 -> 48.9 19.7 -> 12.0 1449 -> 1428
1000 (20) product type member 573 -> 56 1056 -> 119 173 -> 15 7124 -> 6125

A 25-row page there has no duplicate and does not change. The rest of a 1,000-row request is the list itself and the per-row FindingSerializer cost of the main results, which this PR does not touch. (These rows were measured before the last three commits, with every original's acceptance holding one finding and no Reader.)

The same database with one original's acceptance holding 100,001 findings, every other acceptance with a proof, and a Reader and an Owner of every product type. Measured on the final code, 7 runs, medians, highest 1-minute load 11.5, every old and new response body byte-identical:

Request User Queries Prefetch block SQL ms Request wall ms
findings/?prefetch=duplicate_finding&offset=500000&limit=100 superuser 81 -> 52 354 -> 167 11,902 -> 927
same Owner 87 -> 56 320 -> 134 11,614 -> 1,624
same Reader 82 -> 55 15.3 -> 9.5 1,431 -> 1,423
endpoints/?host=<that original's endpoint>&prefetch=findings&limit=50 superuser 34 -> 31 291 -> 108 9,559 -> 214
same Owner 39 -> 35 300 -> 113 9,506 -> 241
same Reader 38 -> 34 12.6 -> 12.3 72 -> 76

The Reader's endpoint request is within its spread (old 70.9 to 83.4 ms, new 70.9 to 96.9 ms).

Documentation

No user-visible change; nothing to document.

Checklist

  • Make sure to rebase your PR against the very latest dev.
  • Submit all PRs, features and bug fixes alike, against the dev branch.
  • Give a meaningful name to your PR, as it may end up being used in the release notes.
  • Your code is Ruff compliant (see ruff.toml).
  • Your code is python 3.13 compliant.
  • If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
  • Model changes must include the necessary migrations in the dojo/db_migrations folder.
  • Add applicable tests to the unit tests.
  • Add the proper label to categorize your PR.

…etch=duplicate_finding

?prefetch=duplicate_finding serializes each distinct original a page's duplicates
name with the full FindingSerializer, from the original's own instance. Nothing
loaded that instance's relations, so each original read its tags, notes and their
authors, types and history, endpoints or locations, finding groups and their JIRA
issues, CWEs, vulnerability ids, risk acceptances and the rest one query at a time:
about 25 queries per distinct original.

Register the relations FindingSerializer reads for Finding in the prefetcher's batch
map (built per call, since endpoints reads locations under V3), so the single-object
path loads them onto all the page's originals at once. The single-object path also
reuses one serializer for every object, as a list serializer reuses its child:
building a FindingSerializer's fields was most of the remaining time.
RiskAcceptanceSerializer.path now reads the acceptance's engagement_set, which a
prefetch serves, and skips the lookup when there is no proof. It names the same
engagement as before: Engagement's manager orders by id whether or not the relation
is prefetched.

The payload is unchanged, byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Maffooch Maffooch added this to the 3.5.200 milestone Nov 11, 2026
@Maffooch Maffooch added performance performance improvement or bug report apiv2 unittests labels Nov 11, 2026
Maffooch and others added 3 commits November 11, 2026 10:57
…ester who sees them

accepted_risks is empty for a requester who may not edit the finding, but the
Finding batch loaded every prefetched finding's acceptances, their notes,
engagements and accepted findings for everyone. An acceptance can hold a very
large number of findings: for a Reader, a page whose original carried a
100,001-finding acceptance read them all and threw them away.

FindingSerializer.shows_accepted_risks() is now the check get_accepted_risks
makes, and the prefetcher loads the risk_acceptance_set lookups only onto the
findings it accepts, on the single and the many path. Everyone else's
prefetched findings read no acceptances, as before the batch. The payload is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batched over several acceptances, the accepted findings query filtered
risk_acceptance_id = ANY(...) and ordered by dojo_finding.id, so Postgres
merge-joined along dojo_finding's primary key instead of walking the through
table's (risk_acceptance_id, finding_id) index as the per-acceptance query did:
612 ms against 225 ms for one 100,001-finding acceptance.

Read them with no ORDER BY and put each acceptance's findings in id order in
Python, the order the relation gives them (Finding's manager orders by id). The
payload is unchanged; a test makes the database return them backwards to show the
order is restored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An acceptance's accepted_findings renders ids, and an acceptance can hold a very
large number of findings. Prefetched whole, they were one model instance per
accepted finding, built only to print its id. Over several acceptances at once
the query also merge-joined along dojo_finding's primary key: 612 ms of SQL
against 225 ms for one acceptance read on its own, with or without its ORDER BY.

The batch now reads (acceptance, finding id) pairs through Finding.objects, so
the relation's own filters and id order apply, and gives each acceptance its ids
as the prefetched relation. This replaces the previous commit's no-ORDER-BY
approach, which did not change the plan. The payload is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Maffooch
Maffooch added this pull request to the merge queue Nov 11, 2026
Merged via the queue into dev with commit 61bc31d Nov 11, 2026
49 checks passed
@Maffooch
Maffooch deleted the perf/prefetch-duplicate-finding-batch branch November 11, 2026 22:52
@defectdojo-release-tracker defectdojo-release-tracker Bot modified the milestones: 3.5.200, 3.5.102 Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apiv2 performance performance improvement or bug report unittests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants