Repository navigation
fix(dedupe): stop a single finding delete from deadlocking the duplicate cleanup - #16568
Conversation
…nnot deadlock the duplicate cleanup
A single-finding delete (DELETE /api/v2/findings/{id}/, the UI delete) ran
Django's collector, which deletes the finding's child rows
(Test_Import_Finding_Action and the rest) first and reaches the dojo_finding
row last. The excess-duplicate cleanup (async_dupe_delete) does the opposite:
it locks the duplicate's dojo_finding row FOR UPDATE and then deletes its
children. When a user deleted a duplicate the cleanup was removing, each
side held a row the other needed and Postgres aborted one with 40P01,
"while deleting tuple ... in relation dojo_test_import_finding_action".
The cleanup's per-chunk retry could not absorb this: the chunk runs as a
savepoint inside _delete_excess_duplicates's transaction, which keeps the
finding lock across retries, so every retry re-entered the same cycle.
Finding.delete now takes the finding's row lock (lock_findings_for_delete)
before the tag removal and the cascade, in one transaction, matching the
bulk delete's order. A delete that loses the race waits for the cleanup to
commit and then finds nothing left to delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review verdict: approveTier: full (backend locking change in CI: all 48 checks pass on Regression test: I ran it in an isolated unit-test compose stack (Postgres, real commits, two connections).
What I checked by reading: Break attemptsEach attempt was a real two-connection test reusing the PR's harness (
No screenshots: this change has no UI surface. Findings
Recommendation: approve. There are no Not checked: behaviour under production-scale concurrency (many cleanup chunks and many simultaneous deletes), and any path that calls |
|
Description
The excess-duplicate cleanup task (
dojo.tasks.async_dupe_delete) keeps failing in production with a Postgres deadlock:The other side of the cycle is a single-finding delete (
DELETE /api/v2/findings/{id}/or the UI delete) of a duplicate the cleanup is removing at the same moment. The two take the same rows in opposite order:_delete_excess_duplicates→_bulk_delete_findings_internal) locks the duplicate'sdojo_findingrowFOR UPDATEfirst, then deletes its child rows,dojo_test_import_finding_actionamong them.Finding.deletehands off to Django's collector, which deletes the child rows first and gets to thedojo_findingrow last.So the single delete holds the
Test_Import_Finding_Actionrows and waits on the finding, while the cleanup holds the finding and waits on those rows. Postgres aborts one of them. If it picks the API request, the user gets a 500. If it picks the cleanup, the run aborts.The per-chunk conflict retry in
_bulk_delete_findings_internalcan't recover from this. The chunk runs as a savepoint inside_delete_excess_duplicates's transaction, and that outer transaction keeps holding the finding lock across retries, so each retry runs straight back into the same cycle. The new test confirms it: with the real retry budget, the chunk deadlocks on all four attempts and the task still fails. The singleton guard from #16337 doesn't apply either, since the counterparty here is a user request, not a second cleanup run.Fix
Finding.deletenow takes the finding's row lock withlock_findings_for_delete(the same helper the bulk delete uses) before it removes tags and runs the cascade, all in one transaction. A single delete and a bulk delete now acquire locks in the same order: finding row first, then children. Whichever starts second waits for the other to commit. If the cleanup already removed the finding, the single delete finds no row to lock and returns without doing anything.Moving the tag removal into the same transaction also means the tag count decrements and the row delete commit or roll back together.
No migration, no settings change.
Test results
New class
TestDupeDeleteRacesSingleFindingDeleteinunittests/test_dedupe_delete_commit_race.py. It uses two real Postgres connections. The cleanup pauses after it locks its chunk, and the single delete runs on a second connection until it blocks. Conflict retries are switched off on both sides so a deadlock shows up as an error.test_deleting_a_duplicate_the_dupe_delete_is_removing_does_not_deadlock: the default ordering. Before the fix, the single delete was aborted withdeadlock detected ... CONTEXT: while deleting tuple (0,7) in relation "dojo_finding".test_dupe_delete_is_not_the_deadlock_victim_of_a_single_delete: holds the cleanup back pastdeadlock_timeout, which makes the cleanup the victim. Before the fix this reproduced the production error exactly:deadlock detected ... CONTEXT: while deleting tuple (0,14) in relation "dojo_test_import_finding_action".test_deleting_an_unrelated_finding_during_the_dupe_delete_succeeds: control. Passes before and after.All three pass with the fix. I also ran these modules locally against Postgres 18, and all pass:
test_dedupe_delete_commit_race,test_async_dupe_delete_singleton,test_duplication_loops,test_finding_group_cleanup,test_finding_delete_conflict_retry,test_finding_patch_concurrent_delete,test_importers_deleted_finding_child_rows,test_importers_closeold,test_prepare_duplicates_for_delete,test_tag_utils_bulk,test_importers_test_tags_deadlock,test_deduplication_logic: 273 tests, OK (2 skipped).test_apiv2_delete_preview,test_apiv2_restricted_delete_message,test_async_delete,test_bulk_delete_duplicate_references,test_bulk_delete_findings_deadlock_retry,test_bulk_delete_findings_m2m,test_cascade_delete,test_classic_delete_warning_message,test_delete_with_endpoints_v3,test_finding_model,test_importers_deleted_target,test_rest_framework: 1055 tests, OK (409 skipped).Ruff 0.16.10 passes on both changed files.
Documentation
None needed. Nothing changes from the user's side except that the deadlock is gone.
Checklist
dev.devbranch.🤖 Generated with Claude Code