Skip to content

fix(dedupe): stop a single finding delete from deadlocking the duplicate cleanup - #16568

Merged
Maffooch merged 1 commit into
devfrom
fix/dupe-delete-deadlock
Oct 27, 2026
Merged

Maffooch merged 1 commit into
devfrom
fix/dupe-delete-deadlock

Conversation

@Maffooch

Copy link
Copy Markdown
Contributor

Description

The excess-duplicate cleanup task (dojo.tasks.async_dupe_delete) keeps failing in production with a Postgres deadlock:

psycopg.errors.DeadlockDetected: deadlock detected
DETAIL:  Process A waits for ShareLock on transaction T1; blocked by process B.
         Process B waits for ShareLock on transaction T2; blocked by process A.
CONTEXT:  while deleting tuple (...) in relation "dojo_test_import_finding_action"

The other side of the cycle is a single-finding delete (DELETE /api/v2/findings/{id}/ or the UI delete) of a duplicate the cleanup is removing at the same moment. The two take the same rows in opposite order:

  • The cleanup (_delete_excess_duplicates → _bulk_delete_findings_internal) locks the duplicate's dojo_finding row FOR UPDATE first, then deletes its child rows, dojo_test_import_finding_action among them.
  • Finding.delete hands off to Django's collector, which deletes the child rows first and gets to the dojo_finding row last.

So the single delete holds the Test_Import_Finding_Action rows and waits on the finding, while the cleanup holds the finding and waits on those rows. Postgres aborts one of them. If it picks the API request, the user gets a 500. If it picks the cleanup, the run aborts.

The per-chunk conflict retry in _bulk_delete_findings_internal can't recover from this. The chunk runs as a savepoint inside _delete_excess_duplicates's transaction, and that outer transaction keeps holding the finding lock across retries, so each retry runs straight back into the same cycle. The new test confirms it: with the real retry budget, the chunk deadlocks on all four attempts and the task still fails. The singleton guard from #16337 doesn't apply either, since the counterparty here is a user request, not a second cleanup run.

Fix

Finding.delete now takes the finding's row lock with lock_findings_for_delete (the same helper the bulk delete uses) before it removes tags and runs the cascade, all in one transaction. A single delete and a bulk delete now acquire locks in the same order: finding row first, then children. Whichever starts second waits for the other to commit. If the cleanup already removed the finding, the single delete finds no row to lock and returns without doing anything.

Moving the tag removal into the same transaction also means the tag count decrements and the row delete commit or roll back together.

No migration, no settings change.

Test results

New class TestDupeDeleteRacesSingleFindingDelete in unittests/test_dedupe_delete_commit_race.py. It uses two real Postgres connections. The cleanup pauses after it locks its chunk, and the single delete runs on a second connection until it blocks. Conflict retries are switched off on both sides so a deadlock shows up as an error.

  • test_deleting_a_duplicate_the_dupe_delete_is_removing_does_not_deadlock: the default ordering. Before the fix, the single delete was aborted with deadlock detected ... CONTEXT: while deleting tuple (0,7) in relation "dojo_finding".
  • test_dupe_delete_is_not_the_deadlock_victim_of_a_single_delete: holds the cleanup back past deadlock_timeout, which makes the cleanup the victim. Before the fix this reproduced the production error exactly: deadlock detected ... CONTEXT: while deleting tuple (0,14) in relation "dojo_test_import_finding_action".
  • test_deleting_an_unrelated_finding_during_the_dupe_delete_succeeds: control. Passes before and after.

All three pass with the fix. I also ran these modules locally against Postgres 18, and all pass:

  • test_dedupe_delete_commit_race, test_async_dupe_delete_singleton, test_duplication_loops, test_finding_group_cleanup, test_finding_delete_conflict_retry, test_finding_patch_concurrent_delete, test_importers_deleted_finding_child_rows, test_importers_closeold, test_prepare_duplicates_for_delete, test_tag_utils_bulk, test_importers_test_tags_deadlock, test_deduplication_logic: 273 tests, OK (2 skipped).
  • test_apiv2_delete_preview, test_apiv2_restricted_delete_message, test_async_delete, test_bulk_delete_duplicate_references, test_bulk_delete_findings_deadlock_retry, test_bulk_delete_findings_m2m, test_cascade_delete, test_classic_delete_warning_message, test_delete_with_endpoints_v3, test_finding_model, test_importers_deleted_target, test_rest_framework: 1055 tests, OK (409 skipped).

Ruff 0.16.10 passes on both changed files.

Documentation

None needed. Nothing changes from the user's side except that the deadlock is gone.

Checklist

  • Make sure to rebase your PR against the very latest dev.
  • Submit all PRs, features and bug fixes alike, against the dev branch.
  • Give a meaningful name to your PR, as it may end up being used in the release notes.
  • Your code is Ruff compliant (see ruff.toml).
  • Your code is python 3.13 compliant.
  • If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
  • Model changes must include the necessary migrations in the dojo/db_migrations folder.
  • Add applicable tests to the unit tests.
  • Add the proper label to categorize your PR.

🤖 Generated with Claude Code

…nnot deadlock the duplicate cleanup

A single-finding delete (DELETE /api/v2/findings/{id}/, the UI delete) ran
Django's collector, which deletes the finding's child rows
(Test_Import_Finding_Action and the rest) first and reaches the dojo_finding
row last. The excess-duplicate cleanup (async_dupe_delete) does the opposite:
it locks the duplicate's dojo_finding row FOR UPDATE and then deletes its
children. When a user deleted a duplicate the cleanup was removing, each
side held a row the other needed and Postgres aborted one with 40P01,
"while deleting tuple ... in relation dojo_test_import_finding_action".

The cleanup's per-chunk retry could not absorb this: the chunk runs as a
savepoint inside _delete_excess_duplicates's transaction, which keeps the
finding lock across retries, so every retry re-entered the same cycle.

Finding.delete now takes the finding's row lock (lock_findings_for_delete)
before the tag removal and the cascade, in one transaction, matching the
bulk delete's order. A delete that loses the race waits for the cleanup to
commit and then finds nothing left to delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Maffooch
Maffooch requested a review from blakeaowens as a code owner October 27, 2026 16:00
@Maffooch Maffooch added this to the 3.5.0 milestone Oct 27, 2026
@Maffooch

Copy link
Copy Markdown
Contributor Author

Review verdict: approve

Tier: full (backend locking change in Finding.delete), run on Opus. This was an independent session that did not write the change.

CI: all 48 checks pass on f4f75776e. That covers ruff, the Rest Framework unit tests (both variants), the UI tests, the k8s deploy tests and the performance tests.

Regression test: I ran it in an isolated unit-test compose stack (Postgres, real commits, two connections).

  • PR tree: all 3 new tests in TestDupeDeleteRacesSingleFindingDelete pass.
  • dojo/finding/models.py reverted to the parent commit: test_deleting_a_duplicate_..._does_not_deadlock fails with deadlock detected ... while deleting tuple (0,54) in relation "dojo_finding", and test_dupe_delete_is_not_the_deadlock_victim_... fails with the production signature, ... in relation "dojo_test_import_finding_action". The unrelated-finding control passes on both trees, as it should. The tests fail without the fix and pass with it.

What I checked by reading: Finding.delete now takes lock_findings_for_delete([self.pk]) (SELECT ... FOR UPDATE ORDER BY id) inside one transaction.atomic(), before the tag removal and the collector cascade. That is the same order _delete_excess_duplicates and _bulk_delete_findings_internal use. The early return on an empty lock result skips product grading, which is correct because the other transaction already deleted the row. delete_finding_with_conflict_retry still wraps each attempt, and each attempt is a fresh transaction. The cost is one extra SELECT ... FOR UPDATE per single delete, which also applies once per row in the classic UI bulk-delete loop (dojo/finding/ui/views.py:2458). That is small next to the cascade, and the performance tests are green.

Break attempts

Each attempt was a real two-connection test reusing the PR's harness (_single_delete_during_dupe_delete), with conflict retries off.

  1. Delete the original (default reconfigure path) while the cleanup removes its duplicate. Passed. Both findings are gone and neither side errored.
  2. Same race with DUPLICATE_CLUSTER_CASCADE_DELETE=True. Still deadlocks, both ways round. In default ordering the single delete is the victim (... in relation "dojo_finding"). When the cleanup is held back past deadlock_timeout, the cleanup is the victim (... in relation "dojo_test_import_finding_action", the production signature). It reproduces identically on the base tree, so it is pre-existing. See R1.
  3. Delete a stale instance whose row is already gone. delete() returns None quietly. No error and no grading call.
  4. Two single deletes of the same finding at once, with the first paused after it takes the row lock. The second blocks on the lock, then finds no row and returns. Neither raised, and no Test_Import_Finding_Action row survives.
  5. The tests themselves. They exercise the real cleanup (_async_dupe_delete_impl) and the real API delete path (delete_finding_with_conflict_retry) against committed rows on two backends. They do not mock the lock, and they assert the end state (finding and import-history rows gone), not only that no error occurred. No false-coverage concern.

No screenshots: this change has no UI surface.

Findings

  • R1 suggestion pre-existing dojo/finding/helper.py:720: with DD_DUPLICATE_CLUSTER_CASCADE_DELETE=True, finding_delete removes the duplicate cluster with a plain duplicate_cluster.order_by("-id").delete(). The collector deletes children first, so deleting an original while the cleanup removes its excess duplicates hits the same deadlock this PR fixes for the single-finding path (reproduced in attempt 2). The fix is to route it through bulk_delete_findings(duplicate_cluster, order_desc=True), which the comment at dojo/finding/helper.py:861 already claims happens. It is outside the lines this PR changes and reproduces on dev, so it is tracked separately in an internal issue assigned to the author. Folding it into this PR would be cheap, since the harness already covers it.

Recommendation: approve. There are no this-pr findings and one pre-existing suggestion (R1), tracked separately.

Not checked: behaviour under production-scale concurrency (many cleanup chunks and many simultaneous deletes), and any path that calls Finding.delete() inside an outer transaction which already holds child-row locks on other findings. I found no such caller in dojo/, but I verified that by reading the code, not by running it.

@Maffooch

Copy link
Copy Markdown
Contributor Author
  • R1 pre-existing, tracked in an internal issue: the DUPLICATE_CLUSTER_CASCADE_DELETE=True path (duplicate_cluster.order_by("-id").delete() in finding_delete) reproduces on the base tree and is outside the lines this PR changes, so it stays out of this PR and gets its own fix.

@Maffooch
Maffooch added this pull request to the merge queue Oct 27, 2026
Merged via the queue into dev with commit bde034d Oct 27, 2026
48 checks passed
@Maffooch
Maffooch deleted the fix/dupe-delete-deadlock branch October 27, 2026 19:26
@defectdojo-release-tracker defectdojo-release-tracker Bot modified the milestones: 3.5.0, 3.4.302 Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants