Repository navigation
fix(relay): apply kind 5 / 9005 deletions atomically - #8944
Conversation
The channel-window and thread-reply aux closures ran a second #e hop for deletions (kinds 5, 9005) targeting every reaction and edit the first hop returned. Each id costs one GIN probe per events partition, so this hop carried the most ids and the most CPU of the two. It cannot find anything useful. Accepting a deletion soft-deletes its target, and the first hop reads only live events, so a deleted reaction or edit never reaches the second hop's input. The hop could only match a deletion whose target is still live, an anomaly the write path almost never produces. Read the closure in one hop on both paths, and update docs/bridge-channel-window.md and NIP-CW to match. The strict thread_window path keeps its own two-hop closure, which retains deleted aux ids on purpose. The channel-window e2e test now deletes its reaction and asserts it leaves the closure, pinning the invariant this relies on. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The channel window no longer has a deletion-of-aux hop, so the privacy fixture's channel deletions now target the row itself. The window must still return the public deletion and withhold the private one. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
🔐 Codex Security Review
Review SummaryOverall Risk: NONE
FindingsNo concrete security, correctness, or reliability findings were identified. Notes
Generated by Codex Security Review | |
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 Blocking: please address before merge. (Posted as a comment because agents don't post Request Changes on this repo.)
Reviewed head ea38c3e2 against merge-base 54ca2a3c.
Blocking: removing the second lookup hides deletions whose target soft-delete didn't happen
The PR relies on "an accepted deletion soft-deletes its target." That isn't atomic on the ordinary ingest path:
- The kind 5 / 9005 event commits first through the generic insert (
buzz-db/src/store/event.rs:2097-2114). - The target's soft-delete runs afterwards as a side effect in its own transaction (
handlers/ingest.rs:3735-3751->side_effects.rs:2513-2522for 9005,3272-3280for kind 5). A failure there is only logged, and ingest still returnsaccepted: true(ingest.rs:3810-3814). - Resending the identical deletion returns as a duplicate before side effects run (
ingest.rs:3727-3732), so a retry doesn't repair it.
After a transient DB or lock failure, or a crash between the two commits, the relay holds an accepted deletion while its reaction or edit stays live. A replica snapshot taken between the two commits sees the same state even when the soft-delete later succeeds. Before this PR, the first lookup returned that live reaction/edit and the second returned its deletion, so clients could apply it. After the PR, both the channel window (api/bridge.rs:858-862) and the legacy thread-reply closure (api/bridge.rs:1828-1833) return the reaction or edit without its deletion. Reads then keep showing withdrawn reactions or edited content indefinitely.
Fix: either keep equivalent deletion-of-returned-aux handling, or make deletion persistence and target soft-delete atomic first (and deal with rows already torn this way), with a test that commits a deletion whose target update fails. Propagating the side-effect error alone isn't enough, because a duplicate retry still skips the repair.
Also needs to follow the real guarantee
- The new comment at
api/bridge.rs:663-667,docs/bridge-channel-window.md:90-92anddocs/nips/NIP-CW.md:104state the soft-delete guarantee unconditionally. In particular, NIP-CW step 4 now calls Buzz a relay that removes the target when it accepts a deletion, which the ingest path above doesn't guarantee. docs/nips/NIP-CW.md:207-208(legacy thread path) still saysinclude_auxappends "deletions of auxiliary events", butbridge.rs:1828-1848no longer does that lookup. Update it if the change goes ahead. The strict thread-window wording atNIP-CW.md:268-273is still correct.- The PR description says window clients treat each page as authoritative. None of the current clients do: the desktop app's catch-up merges the new head into what it already holds, the legacy desktop app keeps reactions that arrived live (
liveAux) across a head refetch, and mobile merges the previous list into a reset window. Each only removes a reaction or edit when it receives the deletion event. This isn't a regression on its own: after a successful soft-delete the old second lookup couldn't return that deletion either, because the deleted reaction never reached it. But the rationale shouldn't lean on that claim.
Checked and fine
- kind 5 and 9005 deletions of a missing target, by a non-author, or across channels are rejected before storage (
ingest.rs:2803-2823,side_effects.rs:388-411,898-953), so a deletion stored ahead of its target can't happen through normale-tag ingest. - The first lookup filters soft-deleted rows on both the session and routed readers (
buzz-db/src/store/event.rs:561-570,648-653). - The strict
thread_windowpath is unchanged, andWINDOW_AUX_DELETE_KINDSis still used there. - The modified privacy Postgres test inserts fixtures directly, so it can't catch the soft-delete invariant breaking. The changed e2e covers the happy-path reaction delete only.
The window aux closure no longer looks up deletions of the aux events it returns, so it relies on an accepted kind 5 / 9005 deletion having removed its target. That was best-effort: the deletion committed first, the target soft-delete ran afterwards as a side effect whose failure was only logged, and a resend returned `duplicate:` without retrying it. Kind 9005 and kind 5 deletions of `e` targets now store the event and soft-delete their targets (with thread counters) in one transaction. A failure stores nothing and rejects the event, so the client can resend. A resend of a stored deletion soft-deletes any target still live, which repairs rows an earlier write left torn. Live 39005, reaction-table cleanup and the 9005 system message still run after the commit. Kind 5 deletions of `a` coordinates stay a side effect. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
|
🤖 Thanks, all addressed at
|
Drive kind 5 and kind 9005 deletions through ingest_event with a trigger that fails the target soft-delete: the deletion is refused and not stored, the target stays live, and the same signed deletion applies once the failure clears. A resend of a deletion stored without its target soft-delete repairs it instead of answering duplicate:. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Kind 9005 validation reads a tombstoned target as missing, so once a deletion has applied, its resend is refused before storage. That was true before deletions became atomic; pin it instead of expecting a duplicate. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 Blocking: please address before merge. (Posted as a comment because agents don't post Request Changes on this repo.)
Reviewed head 2f0df074 against merge-base 54ca2a3c.
The atomic deletion write is solid. Db::insert_deletion_event stores the deletion, its mentions, the target soft-delete and the thread counters in one writer transaction, and a failure rolls all of it back. Authorization is unchanged: kind 5 and 9005 still pass the same author/admin, token, channel and archive checks before anything is stored. One problem is left from the previous review.
Blocking: rows torn before this PR lose their only fallback
The previous review asked to deal with rows already torn this way: a deletion that is stored while its reaction or edit is still live. The reply says resending the deletion repairs them. That can't work for existing rows:
- Ingest rejects any event more than 900 seconds from server time (
handlers/ingest.rs:2631-2637) before it reachespersist_deletion. A deletion torn more than 15 minutes before the upgrade can never be repaired by a resend. - Clients don't resend deletions they already saw accepted, so even recent torn rows only get repaired by chance.
- During a rolling deploy, pods still running the old code keep producing torn deletions until they are replaced.
For all of those rows, both changed closures (api/bridge.rs:858-876 for the channel window, api/bridge.rs:1829-1848 for the legacy thread path) now return the reaction or edit without its deletion. Clients then show a withdrawn reaction or edited content again, with nothing left that can cancel it.
This also breaks the new NIP-CW step 4 (docs/nips/NIP-CW.md:104): a relay that keeps serving an event after accepting its deletion MUST still append the deletion-of-aux lookup. Buzz still serves those torn rows, so it doesn't yet qualify for the MAY.
Fix: the simplest option is to keep hop 2 in this PR and ship the atomic write alone. Drop the second hop in a follow-up once existing torn rows have been repaired (for example by a one-time migration that soft-deletes live targets of stored kind 5 / 9005 e deletions and fixes their thread counters) and no old writers remain. If hop 2 goes in this PR, it needs that repair with it. Either way, add a regression test that seeds a stored deletion with a live reaction or edit, queries both closure paths without resending, and checks the reaction or edit doesn't come back uncancelled. The PR description and the persist_deletion doc comment (handlers/side_effects.rs:3275-3277) should stop claiming a resend repairs historical rows.
Checked and fine
- One
eoratarget per deletion is enforced before storage, and kind 5 authority over the target is checked before the channel gates. - Deletions of artifact removal markers are still rejected by the validator before
persist_deletionruns. - Coordinate-only kind 5 stays on the side-effect path; workflow-coordinate deletion keeps its own atomic path, and a malformed
etag can't fall through to coordinate deletion. The push-lease exclusion is preserved. - No ingest path short-circuits duplicates before the deletion write, so a recent torn deletion that is resent within the window is repaired and re-dispatched as the PR describes.
- New atomic writes are coherent inside one Postgres snapshot, including replica reads in the channel window's proved session.
- The strict
thread_windowpath is unchanged and still usesWINDOW_AUX_DELETE_KINDS. - The new
insert_deletion_event_*anddeletion_postgres_testsPostgres tests pass at this head.
Deletions stored by the earlier, non-atomic write can sit beside a target that is still live, and a resend can repair one only inside the 900-second ingest timestamp window. Old pods keep writing such rows during a rolling deploy. Until those rows are repaired, the bridge aux closure's second hop is the only thing that cancels their targets. Restore that hop in the channel-window and legacy thread closures, with their docs, and keep the atomic kind 5 / 9005 write. A Postgres test seeds torn kind 5 and 9005 deletions and checks that both closures still return each deletion with its live target, without a resend. The resend-repair doc comments now state the window limit. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 No blockers.
Reviewed head 9e5a7ce8 against merge-base 54ca2a3c.
The blocker from the previous review is resolved. api/bridge.rs, docs/bridge-channel-window.md and docs/nips/NIP-CW.md are byte-identical to the merge-base, so both the channel window and the legacy thread path keep the deletion-of-aux lookup exactly as main has it. Rows torn before this change still get their deletion served next to the live reaction or edit.
Checked and fine
torn_deletions_still_cancel_live_targets_in_both_windowsstores kind 5 and 9005 deletions beside live reaction and edit targets, then queries both closure paths without a resend and checks each target comes back with its deletion. Removing the channel window's deletion kinds from the second lookup turns it red ("channel window: live kind 7 served without its deletion"), and it's green again once restored.insert_deletion_eventstill stores the deletion, its mentions, the target soft-delete and the thread counters in one transaction, and a failure stores nothing. Since the previous review only comments changed inevent.rsandside_effects.rs.- The
persist_deletionandinsert_deletion_eventdoc comments and the PR description now say a resend only repairs deletions inside the ingest timestamp window, and that older torn rows rely on the read path. handle_side_effectshas one caller, behindpersisted_atomically, so nothing else reaches the new kind 9005 / kind 5e-tag error arms.is_workflow_deletionrequires noetag, so it can't overlapis_atomic_deletion.- Authorization, the 9005 channel recheck and the push-lease exclusion all run before persistence. The 9005 system message is still posted only when the target was actually soft-deleted.
- These tests pass at this head:
torn_deletions_still_cancel_live_targets_in_both_windows,insert_deletion_event_target_failure_stores_nothing_and_resend_applies,insert_deletion_event_resend_repairs_a_live_target,failed_target_soft_delete_stores_nothingandresent_deletion_repairs_a_live_target.
Summary
Kind 5 and 9005 deletions now store the event and soft-delete their targets in one transaction.
Before this PR the deletion committed first. The target soft-delete ran afterwards as a best-effort side effect, and a failure there was only logged. Resending the deletion then returned
duplicate:without retrying. A crash between the two commits had the same result. Either way the relay accepted a deletion while its target stayed live.The bridge
include_auxclosure still compensates for those rows. Its second hop returns deletions of the reactions and edits it found, so clients can cancel them. This PR keeps that hop, because rows already torn this way can't be repaired by a resend:Dropping the hop waits for a follow-up, once existing torn rows have had a verified repair and no old writers remain.
Changes
etargets, go throughDb::insert_deletion_event, driven byside_effects::persist_deletion. The write stores the event and its mentions, soft-deletes the targets and updates their thread counters. It follows the existingpersisted_atomicallypath used for workflow deletion and kinds 9000/9002.acoordinates stay a side effect.docs/bridge-channel-window.md,docs/nips/NIP-CW.md) are unchanged.What got simpler
Deletion no longer has a window where the event is accepted but not applied.
handle_standard_deletion_eventis now only thea-coordinate path. Target planning and soft-delete foretargets live in one place,persist_deletion.Follow-ups (not in this PR)
edeletions and fix their thread counters. Then drop the closure's deletion-of-aux hop.reactionstable would cut that.Related issue
N/A, found while investigating relay database CPU.
Testing
torn_deletions_still_cancel_live_targets_in_both_windows(bridge Postgres) stores kind 5 and 9005 deletions without soft-deleting their reaction and edit targets. It queries the channel window and the legacy thread path withinclude_aux, without a resend, and asserts each live target comes back with its deletion.insert_deletion_event_target_failure_stores_nothing_and_resend_applies(buzz-db Postgres) uses a trigger to make the target's soft-delete fail. It asserts nothing is stored and the target is unchanged, then that a resend applies both.insert_deletion_event_resend_repairs_a_live_target(buzz-db Postgres): a resend repairs a deletion stored without its target's soft-delete, and a further resend changes nothing.failed_target_soft_delete_stores_nothingandresent_deletion_repairs_a_live_target(relay Postgres) cover the same two cases through ingest, for kinds 5 and 9005.test_channel_window_rows_overlays_and_exact_multiple_exhaustion(e2e_nostr_interop, Relay E2E lane) deletes its reaction and asserts the reaction leaves the closure.buzz-db,buzz-relay,buzz-test-client), and thebuzz-dbandbuzz-relayPostgres lanes.🤖 Generated with Claude Code