Repository navigation
perf(api): load the relations of prefetched duplicate findings once per page - #16789
Merged
Merged
Conversation
…etch=duplicate_finding ?prefetch=duplicate_finding serializes each distinct original a page's duplicates name with the full FindingSerializer, from the original's own instance. Nothing loaded that instance's relations, so each original read its tags, notes and their authors, types and history, endpoints or locations, finding groups and their JIRA issues, CWEs, vulnerability ids, risk acceptances and the rest one query at a time: about 25 queries per distinct original. Register the relations FindingSerializer reads for Finding in the prefetcher's batch map (built per call, since endpoints reads locations under V3), so the single-object path loads them onto all the page's originals at once. The single-object path also reuses one serializer for every object, as a list serializer reuses its child: building a FindingSerializer's fields was most of the remaining time. RiskAcceptanceSerializer.path now reads the acceptance's engagement_set, which a prefetch serves, and skips the lookup when there is no proof. It names the same engagement as before: Engagement's manager orders by id whether or not the relation is prefetched. The payload is unchanged, byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ester who sees them accepted_risks is empty for a requester who may not edit the finding, but the Finding batch loaded every prefetched finding's acceptances, their notes, engagements and accepted findings for everyone. An acceptance can hold a very large number of findings: for a Reader, a page whose original carried a 100,001-finding acceptance read them all and threw them away. FindingSerializer.shows_accepted_risks() is now the check get_accepted_risks makes, and the prefetcher loads the risk_acceptance_set lookups only onto the findings it accepts, on the single and the many path. Everyone else's prefetched findings read no acceptances, as before the batch. The payload is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batched over several acceptances, the accepted findings query filtered risk_acceptance_id = ANY(...) and ordered by dojo_finding.id, so Postgres merge-joined along dojo_finding's primary key instead of walking the through table's (risk_acceptance_id, finding_id) index as the per-acceptance query did: 612 ms against 225 ms for one 100,001-finding acceptance. Read them with no ORDER BY and put each acceptance's findings in id order in Python, the order the relation gives them (Finding's manager orders by id). The payload is unchanged; a test makes the database return them backwards to show the order is restored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An acceptance's accepted_findings renders ids, and an acceptance can hold a very large number of findings. Prefetched whole, they were one model instance per accepted finding, built only to print its id. Over several acceptances at once the query also merge-joined along dojo_finding's primary key: 612 ms of SQL against 225 ms for one acceptance read on its own, with or without its ORDER BY. The batch now reads (acceptance, finding id) pairs through Finding.objects, so the relation's own filters and id order apply, and gives each acceptance its ids as the prefetched relation. This replaces the previous commit's no-ORDER-BY approach, which did not change the plan. The payload is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blakeaowens
approved these changes
Nov 11, 2026
devGregA
approved these changes
Nov 11, 2026
6 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GET /api/v2/findings/?prefetch=duplicate_findingcost about 25 queries for every distinct original the page's duplicates point at. On a 1,000-row page with 20 distinct originals that came to 549 queries, against 52 for the same page once each original's relations are loaded together.For a single related object the prefetcher serializes the entry's own instance with the full
FindingSerializer. Nothing loaded that instance's relations, so each original read its tags, notes (with authors, note types and history), endpoints or locations, finding groups and their JIRA issues, CWEs, vulnerability ids, risk acceptances, files, request/response pairs, metadata, reviewers,found_by, its test's type and, for a non-superuser, the product itsaccepted_riskspermission check walks to, one query at a time.This PR:
Findingin_PREFETCH_PREFETCH_RELATEDwith the relationsFindingSerializerreads. The entry is a callable, built at each call, becauseendpointsreadslocationsunder V3 and the notes and vulnerability id lookups arePrefetchobjects (notes_prefetch(),vulnerability_id_prefetch())._prefetch_related_for()resolves an entry for both the single and the many path._prefetch_singlereuse one serializer for every object, as a list serializer reuses its child. Building aFindingSerializer's fields was most of the time left once the queries were batched (about 0.3 s of 0.5 s for 20 originals under a profiler), and none of it depends on the object.RiskAcceptanceSerializer.get_pathskip the engagement lookup when the acceptance has no proof, and read the acceptance'sengagement_setinstead ofEngagement.objects.filter(...).first(), so the prefetch serves it. It names the same engagement as before:Engagement's manager (BaseManager) orders byid, prefetched or not, so both pick the lowest id. (ItsMeta.orderingof-target_startis overridden by the manager; the test pins the lowest-id behaviour.)accepted_risksreads (each acceptance's notes, engagements and accepted findings) only onto the findings it lists acceptances for.FindingSerializer.shows_accepted_risks()is now the checkget_accepted_risksmakes (the requester may edit the finding), and_PREFETCH_PREFETCH_RELATED_WHEN_SHOWNapplies the loader only to the findings it accepts, on both paths. Without that gate, a requester who may only view would load every acceptance, accepted finding included, and get[].(acceptance, finding id)pairs, throughFinding.objectsso the relation's own filters and id order apply, and gives each acceptance its ids as the prefetched relation (accepted_findingsrenders ids only). Prefetched as whole rows, an acceptance holding 100,001 findings meant 100,001 model instances built to print their ids. Over several acceptances at once, that query also merge-joined alongdojo_finding's primary key: 612 ms of SQL, against 225 ms when one acceptance is read on its own.Because the many path now also applies the
Findingentry to the authorized queryset, a list of findings prefetched through a to-many relation (for example?prefetch=findingson endpoints) is batched the same way.The response body is unchanged, byte for byte.
Test results
New
unittests/test_apiv2_prefetch_duplicate_finding.py. Each original carries two tags, a public note with history and a private note, an endpoint (V2) or a location (V3), a vulnerability id, a CWE, a finding group with a JIRA issue, its own JIRA issue, a risk acceptance with a proof, a reviewer, afound_by, a file, a request/response pair and a metadata row. JIRA is enabled with a project and instance.?prefetch=duplicate_findingwith 2 originals, then 6, costs the same, for a superuser and for a product member. Ondevit failed: 58 against 170 queries (superuser), 59 against 171 (member).JSONRenderer) to the payload with the old_prefetch_single, noFindingentry and the oldget_path, for both users.get_path: no proof needs no query (1 ondev); a proof with no engagement still says "No proof has been supplied"; with two engagements it names the lowest id, matching the old query; a prefetchedengagement_setserves it with no query (1 ondev).editin the serializer's permission check), with acceptances holding three more findings each: the single path reads no risk acceptance table, as ondev(it read 3 tables before the gate). The many path (?prefetch=accepted_findingson the acceptances) reads them only for the field itself, as ondev(4 reads against 1 before the gate). Bodies are identical to the old code. An editor on the many path still gets every acceptance with its four findings.Run with
DD_V3_FEATURE_LOCATIONSTrue and False:devfor the reasons above, and the two requester-without-edit tests fail on this PR's first commit.test_api_product_prefetch,test_apiv2_list_query_scaling,test_apiv2_prefetch_*,test_bulk_risk_acceptance_api,test_risk_acceptance*,test_finding_modelandtest_rest_framework(which requests?prefetch=on every viewset): 1087 tests, OK in both modes.ruff check(0.16.10) on the changed files: clean.Measured on a 10M-finding database (one in fifty findings past a hot range is a duplicate; the 20 originals were given the relations above),
GET /api/v2/findings/?prefetch=duplicate_findingat offset 500000, old and new code in one process, alternating with the order flipped each run, 7 runs, medians:A 25-row page there has no duplicate and does not change. The rest of a 1,000-row request is the list itself and the per-row
FindingSerializercost of the main results, which this PR does not touch. (These rows were measured before the last three commits, with every original's acceptance holding one finding and no Reader.)The same database with one original's acceptance holding 100,001 findings, every other acceptance with a proof, and a Reader and an Owner of every product type. Measured on the final code, 7 runs, medians, highest 1-minute load 11.5, every old and new response body byte-identical:
findings/?prefetch=duplicate_finding&offset=500000&limit=100endpoints/?host=<that original's endpoint>&prefetch=findings&limit=50The Reader's endpoint request is within its spread (old 70.9 to 83.4 ms, new 70.9 to 96.9 ms).
Documentation
No user-visible change; nothing to document.
Checklist
dev.devbranch.