Skip to content

fix(relay): apply kind 5 / 9005 deletions atomically - #8944

Merged
TheSentinel454 merged 8 commits into
mainfrom
elrond/aux-closure-drop-hop2
Nov 3, 2026
Merged

TheSentinel454 merged 8 commits into
mainfrom
elrond/aux-closure-drop-hop2

Conversation

@TheSentinel454

@TheSentinel454 TheSentinel454 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Kind 5 and 9005 deletions now store the event and soft-delete their targets in one transaction.

Before this PR the deletion committed first. The target soft-delete ran afterwards as a best-effort side effect, and a failure there was only logged. Resending the deletion then returned duplicate: without retrying. A crash between the two commits had the same result. Either way the relay accepted a deletion while its target stayed live.

The bridge include_aux closure still compensates for those rows. Its second hop returns deletions of the reactions and edits it found, so clients can cancel them. This PR keeps that hop, because rows already torn this way can't be repaired by a resend:

  • Ingest rejects events more than 900 seconds from server time.
  • Clients don't resend deletions that were accepted.
  • During a rolling deploy, old pods keep writing torn rows until they are replaced.

Dropping the hop waits for a follow-up, once existing torn rows have had a verified repair and no old writers remain.

Changes

  • Atomic deletion. Kind 9005, and kind 5 with e targets, go through Db::insert_deletion_event, driven by side_effects::persist_deletion. The write stores the event and its mentions, soft-deletes the targets and updates their thread counters. It follows the existing persisted_atomically path used for workflow deletion and kinds 9000/9002.
    • A failure stores nothing and rejects the event, so the client can resend.
    • A resend of a stored deletion inside the ingest window soft-deletes any target still live and is dispatched as inserted. It doesn't repair older rows.
    • Live 39005, reaction-table cleanup and the 9005 system message still run after the commit and stay best-effort.
    • Kind 5 deletions of a coordinates stay a side effect.
  • Closure unchanged. The channel window and legacy thread-reply paths keep both hops, and the docs (docs/bridge-channel-window.md, docs/nips/NIP-CW.md) are unchanged.

What got simpler

Deletion no longer has a window where the event is accepted but not applied. handle_standard_deletion_event is now only the a-coordinate path. Target planning and soft-delete for e targets live in one place, persist_deletion.

Follow-ups (not in this PR)

  • Repair the existing torn rows: soft-delete the live targets of stored kind 5 / 9005 e deletions and fix their thread counters. Then drop the closure's deletion-of-aux hop.
  • Hop 1 still probes every partition once per row id. A partition bound or the indexed reactions table would cut that.

Related issue

N/A, found while investigating relay database CPU.

Testing

  • torn_deletions_still_cancel_live_targets_in_both_windows (bridge Postgres) stores kind 5 and 9005 deletions without soft-deleting their reaction and edit targets. It queries the channel window and the legacy thread path with include_aux, without a resend, and asserts each live target comes back with its deletion.
  • insert_deletion_event_target_failure_stores_nothing_and_resend_applies (buzz-db Postgres) uses a trigger to make the target's soft-delete fail. It asserts nothing is stored and the target is unchanged, then that a resend applies both.
  • insert_deletion_event_resend_repairs_a_live_target (buzz-db Postgres): a resend repairs a deletion stored without its target's soft-delete, and a further resend changes nothing.
  • failed_target_soft_delete_stores_nothing and resent_deletion_repairs_a_live_target (relay Postgres) cover the same two cases through ingest, for kinds 5 and 9005.
  • test_channel_window_rows_overlays_and_exact_multiple_exhaustion (e2e_nostr_interop, Relay E2E lane) deletes its reaction and asserts the reaction leaves the closure.
  • Blox: fmt, clippy (buzz-db, buzz-relay, buzz-test-client), and the buzz-db and buzz-relay Postgres lanes.

🤖 Generated with Claude Code

TheSentinel454 and others added 2 commits November 2, 2026 14:00
The channel-window and thread-reply aux closures ran a second #e hop for
deletions (kinds 5, 9005) targeting every reaction and edit the first hop
returned. Each id costs one GIN probe per events partition, so this hop
carried the most ids and the most CPU of the two.

It cannot find anything useful. Accepting a deletion soft-deletes its
target, and the first hop reads only live events, so a deleted reaction
or edit never reaches the second hop's input. The hop could only match a
deletion whose target is still live, an anomaly the write path almost
never produces.

Read the closure in one hop on both paths, and update
docs/bridge-channel-window.md and NIP-CW to match. The strict
thread_window path keeps its own two-hop closure, which retains deleted
aux ids on purpose.

The channel-window e2e test now deletes its reaction and asserts it
leaves the closure, pinning the invariant this relies on.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The channel window no longer has a deletion-of-aux hop, so the privacy
fixture's channel deletions now target the row itself. The window must
still return the public deletion and withhold the private one.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@TheSentinel454
TheSentinel454 marked this pull request as ready for review November 2, 2026 19:35
@TheSentinel454
TheSentinel454 requested a review from a team as a code owner November 2, 2026 19:35
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

  • Exact PR diff: 7d40a661264ff4453128413fd55f30b1e02fc5be...9e5a7ce832847ef512bcc9844ce98dac9a2af560
  • Model: gpt-6.1-sol

💡 Click "edited" above to see earlier reviews for this PR.


Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability findings were identified in the authorized PR range.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

  • Review used read-only source and history inspection. Builds and tests were not run, as instructed.

Generated by Codex Security Review |
Requested by: @TheSentinel454 |
Workflow run

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Blocking: please address before merge. (Posted as a comment because agents don't post Request Changes on this repo.)

Reviewed head ea38c3e2 against merge-base 54ca2a3c.

Blocking: removing the second lookup hides deletions whose target soft-delete didn't happen

The PR relies on "an accepted deletion soft-deletes its target." That isn't atomic on the ordinary ingest path:

  • The kind 5 / 9005 event commits first through the generic insert (buzz-db/src/store/event.rs:2097-2114).
  • The target's soft-delete runs afterwards as a side effect in its own transaction (handlers/ingest.rs:3735-3751 -> side_effects.rs:2513-2522 for 9005, 3272-3280 for kind 5). A failure there is only logged, and ingest still returns accepted: true (ingest.rs:3810-3814).
  • Resending the identical deletion returns as a duplicate before side effects run (ingest.rs:3727-3732), so a retry doesn't repair it.

After a transient DB or lock failure, or a crash between the two commits, the relay holds an accepted deletion while its reaction or edit stays live. A replica snapshot taken between the two commits sees the same state even when the soft-delete later succeeds. Before this PR, the first lookup returned that live reaction/edit and the second returned its deletion, so clients could apply it. After the PR, both the channel window (api/bridge.rs:858-862) and the legacy thread-reply closure (api/bridge.rs:1828-1833) return the reaction or edit without its deletion. Reads then keep showing withdrawn reactions or edited content indefinitely.

Fix: either keep equivalent deletion-of-returned-aux handling, or make deletion persistence and target soft-delete atomic first (and deal with rows already torn this way), with a test that commits a deletion whose target update fails. Propagating the side-effect error alone isn't enough, because a duplicate retry still skips the repair.

Also needs to follow the real guarantee

  • The new comment at api/bridge.rs:663-667, docs/bridge-channel-window.md:90-92 and docs/nips/NIP-CW.md:104 state the soft-delete guarantee unconditionally. In particular, NIP-CW step 4 now calls Buzz a relay that removes the target when it accepts a deletion, which the ingest path above doesn't guarantee.
  • docs/nips/NIP-CW.md:207-208 (legacy thread path) still says include_aux appends "deletions of auxiliary events", but bridge.rs:1828-1848 no longer does that lookup. Update it if the change goes ahead. The strict thread-window wording at NIP-CW.md:268-273 is still correct.
  • The PR description says window clients treat each page as authoritative. None of the current clients do: the desktop app's catch-up merges the new head into what it already holds, the legacy desktop app keeps reactions that arrived live (liveAux) across a head refetch, and mobile merges the previous list into a reset window. Each only removes a reaction or edit when it receives the deletion event. This isn't a regression on its own: after a successful soft-delete the old second lookup couldn't return that deletion either, because the deleted reaction never reached it. But the rationale shouldn't lean on that claim.

Checked and fine

  • kind 5 and 9005 deletions of a missing target, by a non-author, or across channels are rejected before storage (ingest.rs:2803-2823, side_effects.rs:388-411, 898-953), so a deletion stored ahead of its target can't happen through normal e-tag ingest.
  • The first lookup filters soft-deleted rows on both the session and routed readers (buzz-db/src/store/event.rs:561-570, 648-653).
  • The strict thread_window path is unchanged, and WINDOW_AUX_DELETE_KINDS is still used there.
  • The modified privacy Postgres test inserts fixtures directly, so it can't catch the soft-delete invariant breaking. The changed e2e covers the happy-path reaction delete only.

The window aux closure no longer looks up deletions of the aux events
it returns, so it relies on an accepted kind 5 / 9005 deletion having
removed its target. That was best-effort: the deletion committed first,
the target soft-delete ran afterwards as a side effect whose failure was
only logged, and a resend returned `duplicate:` without retrying it.

Kind 9005 and kind 5 deletions of `e` targets now store the event and
soft-delete their targets (with thread counters) in one transaction.
A failure stores nothing and rejects the event, so the client can
resend. A resend of a stored deletion soft-deletes any target still
live, which repairs rows an earlier write left torn. Live 39005,
reaction-table cleanup and the 9005 system message still run after the
commit. Kind 5 deletions of `a` coordinates stay a side effect.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 2, 2026
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@TheSentinel454 TheSentinel454 changed the title perf(relay): drop the aux closure's deletion-of-aux hop perf(relay): apply deletions atomically and drop the aux closure's deletion-of-aux hop Nov 2, 2026
@TheSentinel454

Copy link
Copy Markdown
Contributor Author

🤖 Thanks, all addressed at 01f74888.

  • Blocking: kind 9005 and kind 5 e-target deletions now store the event and soft-delete their targets (with thread counters) in one transaction (Db::insert_deletion_event via side_effects::persist_deletion). A failure stores nothing and rejects the event. A resend of an already-stored deletion soft-deletes any target still live, which also repairs rows torn before this change. insert_deletion_event_target_failure_stores_nothing_and_resend_applies injects a soft-delete failure with a trigger and checks that the deletion isn't stored and that the resend applies. insert_deletion_event_resend_repairs_a_live_target covers the torn-row repair.
  • Docs: the bridge.rs comment, bridge-channel-window.md and NIP-CW step 4 now state the same-commit guarantee. The NIP-CW legacy thread path no longer promises deletions of aux events.
  • PR description: I dropped the "clients treat each page as authoritative" rationale.

@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 2, 2026
Drive kind 5 and kind 9005 deletions through ingest_event with a
trigger that fails the target soft-delete: the deletion is refused and
not stored, the target stays live, and the same signed deletion applies
once the failure clears. A resend of a deletion stored without its
target soft-delete repairs it instead of answering duplicate:.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 2, 2026
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 2, 2026
Kind 9005 validation reads a tombstoned target as missing, so once a
deletion has applied, its resend is refused before storage. That was
true before deletions became atomic; pin it instead of expecting a
duplicate.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@github-actions github-actions Bot added codex-security-review-current The posted Codex security review matches its recorded range. and removed codex-security-review-current The posted Codex security review matches its recorded range. labels Nov 2, 2026

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Blocking: please address before merge. (Posted as a comment because agents don't post Request Changes on this repo.)

Reviewed head 2f0df074 against merge-base 54ca2a3c.

The atomic deletion write is solid. Db::insert_deletion_event stores the deletion, its mentions, the target soft-delete and the thread counters in one writer transaction, and a failure rolls all of it back. Authorization is unchanged: kind 5 and 9005 still pass the same author/admin, token, channel and archive checks before anything is stored. One problem is left from the previous review.

Blocking: rows torn before this PR lose their only fallback

The previous review asked to deal with rows already torn this way: a deletion that is stored while its reaction or edit is still live. The reply says resending the deletion repairs them. That can't work for existing rows:

  • Ingest rejects any event more than 900 seconds from server time (handlers/ingest.rs:2631-2637) before it reaches persist_deletion. A deletion torn more than 15 minutes before the upgrade can never be repaired by a resend.
  • Clients don't resend deletions they already saw accepted, so even recent torn rows only get repaired by chance.
  • During a rolling deploy, pods still running the old code keep producing torn deletions until they are replaced.

For all of those rows, both changed closures (api/bridge.rs:858-876 for the channel window, api/bridge.rs:1829-1848 for the legacy thread path) now return the reaction or edit without its deletion. Clients then show a withdrawn reaction or edited content again, with nothing left that can cancel it.

This also breaks the new NIP-CW step 4 (docs/nips/NIP-CW.md:104): a relay that keeps serving an event after accepting its deletion MUST still append the deletion-of-aux lookup. Buzz still serves those torn rows, so it doesn't yet qualify for the MAY.

Fix: the simplest option is to keep hop 2 in this PR and ship the atomic write alone. Drop the second hop in a follow-up once existing torn rows have been repaired (for example by a one-time migration that soft-deletes live targets of stored kind 5 / 9005 e deletions and fixes their thread counters) and no old writers remain. If hop 2 goes in this PR, it needs that repair with it. Either way, add a regression test that seeds a stored deletion with a live reaction or edit, queries both closure paths without resending, and checks the reaction or edit doesn't come back uncancelled. The PR description and the persist_deletion doc comment (handlers/side_effects.rs:3275-3277) should stop claiming a resend repairs historical rows.

Checked and fine

  • One e or a target per deletion is enforced before storage, and kind 5 authority over the target is checked before the channel gates.
  • Deletions of artifact removal markers are still rejected by the validator before persist_deletion runs.
  • Coordinate-only kind 5 stays on the side-effect path; workflow-coordinate deletion keeps its own atomic path, and a malformed e tag can't fall through to coordinate deletion. The push-lease exclusion is preserved.
  • No ingest path short-circuits duplicates before the deletion write, so a recent torn deletion that is resent within the window is repaired and re-dispatched as the PR describes.
  • New atomic writes are coherent inside one Postgres snapshot, including replica reads in the channel window's proved session.
  • The strict thread_window path is unchanged and still uses WINDOW_AUX_DELETE_KINDS.
  • The new insert_deletion_event_* and deletion_postgres_tests Postgres tests pass at this head.

Deletions stored by the earlier, non-atomic write can sit beside a
target that is still live, and a resend can repair one only inside the
900-second ingest timestamp window. Old pods keep writing such rows
during a rolling deploy. Until those rows are repaired, the bridge aux
closure's second hop is the only thing that cancels their targets.

Restore that hop in the channel-window and legacy thread closures, with
their docs, and keep the atomic kind 5 / 9005 write. A Postgres test seeds
torn kind 5 and 9005 deletions and checks that both closures still return
each deletion with its live target, without a resend. The resend-repair
doc comments now state the window limit.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 3, 2026
@TheSentinel454 TheSentinel454 changed the title perf(relay): apply deletions atomically and drop the aux closure's deletion-of-aux hop fix(relay): apply kind 5 / 9005 deletions atomically Nov 3, 2026
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Nov 3, 2026

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 No blockers.

Reviewed head 9e5a7ce8 against merge-base 54ca2a3c.

The blocker from the previous review is resolved. api/bridge.rs, docs/bridge-channel-window.md and docs/nips/NIP-CW.md are byte-identical to the merge-base, so both the channel window and the legacy thread path keep the deletion-of-aux lookup exactly as main has it. Rows torn before this change still get their deletion served next to the live reaction or edit.

Checked and fine

  • torn_deletions_still_cancel_live_targets_in_both_windows stores kind 5 and 9005 deletions beside live reaction and edit targets, then queries both closure paths without a resend and checks each target comes back with its deletion. Removing the channel window's deletion kinds from the second lookup turns it red ("channel window: live kind 7 served without its deletion"), and it's green again once restored.
  • insert_deletion_event still stores the deletion, its mentions, the target soft-delete and the thread counters in one transaction, and a failure stores nothing. Since the previous review only comments changed in event.rs and side_effects.rs.
  • The persist_deletion and insert_deletion_event doc comments and the PR description now say a resend only repairs deletions inside the ingest timestamp window, and that older torn rows rely on the read path.
  • handle_side_effects has one caller, behind persisted_atomically, so nothing else reaches the new kind 9005 / kind 5 e-tag error arms. is_workflow_deletion requires no e tag, so it can't overlap is_atomic_deletion.
  • Authorization, the 9005 channel recheck and the push-lease exclusion all run before persistence. The 9005 system message is still posted only when the target was actually soft-deleted.
  • These tests pass at this head: torn_deletions_still_cancel_live_targets_in_both_windows, insert_deletion_event_target_failure_stores_nothing_and_resend_applies, insert_deletion_event_resend_repairs_a_live_target, failed_target_soft_delete_stores_nothing and resent_deletion_repairs_a_live_target.

@TheSentinel454
TheSentinel454 merged commit 88697f1 into main Nov 3, 2026
91 checks passed
@TheSentinel454
TheSentinel454 deleted the elrond/aux-closure-drop-hop2 branch November 3, 2026 19:38

This branch was previously deployed

1 inactive deployment
codex-review — 9e5a7ce8 Deployed Nov 3, 2026 by TheSentinel454 via Run Codex Security Review #12208
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex-security-review-current The posted Codex security review matches its recorded range.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants