Skip to content

feat(relay): community custom emoji catalog for kind 30180 - #9486

Draft
baxen wants to merge 1 commit into
mainfrom
mbot/emoji-catalog
Draft

baxen wants to merge 1 commit into
mainfrom
mbot/emoji-catalog

Conversation

@baxen

@baxen baxen commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

What

A community can hold 100,000+ custom emoji, and one member can add all of them.

Today the palette is the union of every member's kind:30030 set. One set is one event, so a member is capped by the event size limit (about 550 emoji), and every client re-reads every set to build the palette. Kind:30180 (one emoji per event, #8783) removed the size cap but left names per-author and gave clients no cheap way to hold the whole palette.

This PR makes the relay keep one catalog per community, built from kind:30180 entries:

  • Unique names. A shortcode belongs to the member who first published it. Only that member, or a community owner or admin, may replace or remove it. Anyone else is rejected.
  • Removal is an entry. A kind:30180 event with no emoji tag removes the emoji and frees the name. NIP-09 deletion of an entry is refused, so every catalog change is an entry. A moderation delete of an entry removes the emoji from the catalog in the same commit.
  • Change numbers. Each accepted write gets the community's next change number, allocated under a per-community lock so changes commit in number order. A client that holds everything up to n needs only rows above n.
  • POST /emoji/catalog. {"since": n} returns compact rows [shortcode, change, url, author] changed after n (10,000 per page, gzip). {"shortcodes": [...]} returns the live rows for up to 100 names.

docs/nips/NIP-CE.md is rewritten to match. No client reads kind:30180 yet, so nothing depends on the old per-author semantics.

Why an HTTP endpoint instead of a Nostr filter

Nostr filters page by created_at, which repeats or skips rows written in the same second and cannot express "changes since N". A first read of 100,000 emoji is about 3.4 MB gzipped as compact rows versus about 17 MB as signed events (synthetic benchmark). Writes stay ordinary events through POST /events and the WebSocket.

Database

  • Phase: expand and deploy in one PR. Migration 0089 creates an empty table custom_emoji and one unique index; no backfill, no triggers, functions or foreign keys; lock_timeout set. An older relay ignores the table.
  • Tenant table: keyed by community_id, writes go through begin_community_event_write_transaction, added to EXPECTED_SCOPED_TABLES and PURGE_SCOPED_TABLES, the migration count test and the schema parity list. The community-deletion test covers a refused write after quiesce and the purge.
  • Write cost: one advisory lock, one primary-key read, the existing replaceable-event write, one upsert. Writes to one community's catalog are serialized; other communities and other event kinds are unaffected.
  • Reads: change > $n ORDER BY change LIMIT on (community_id, change), and a primary-key lookup. Both read the primary, so a client never sees a change number ahead of the rows.
  • While old relays still serve during a rollout, community deletion fails closed on the new table, as the rules allow.

Fleet cost (rule 9)

A reconnecting client sends one request with its last change number and usually gets an empty page. The spec tells clients to read on connect and on picker open, and not to poll faster than every 15 minutes.

Testing

  • handlers::ingest::postgres_tests::emoji_entries_build_one_catalog_per_community: ownership, refused takeover, admin takeover retires the previous event, kind 5 refused by coordinate and by id, removal, removal of nothing, reclaim, stale entry refused, paging, cross-community isolation.
  • relay_admin_actions: moderation delete removes the emoji from the catalog.
  • e2e_emoji_catalog: the HTTP journey against a live relay (added to the relay e2e job).
  • Unit tests for request parsing, row encoding, entry validation and the removal builder.
  • Ran locally: the tests above, the migration and deletion tests, the unit lane for buzz-db, buzz-relay, buzz-sdk, clippy and fmt. The e2e journey passed against a local relay. My local Postgres is 14, so 42 unrelated tests (partition maintenance, JSON syntax, media without object storage) fail there; CI on 16 is the authority for the full lane.

Not in this PR

  • Existing emoji. Emoji in kind:30030 sets are not in the catalog. A follow-up adds a one-time backfill script and projects accepted sets into the catalog. The same script must cover any kind:30180 events a deployment already holds: they have no catalog row, their shortcodes can be claimed, and with this PR their authors can no longer delete them with kind 5.
  • Bulk import. Importing 90,000 emoji is limited by media and request rate limits, not by this write path. Follow-up.
  • Clients and CLI. No client or buzz command reads the catalog yet.
  • User deletion. Catalog rows keep the owner's pubkey and image URL; the user-deletion workflow does not touch them yet.
  • Agents. Ownership follows the signing key, so an owner cannot manage their agent's emoji unless they are a community admin.

🤖 Generated with Claude Code

A community can now hold 100,000+ custom emoji. Each accepted kind:30180
entry is stored and applied to a new `custom_emoji` table in one
transaction:

- shortcodes are unique in a community; the first publisher owns one, and
  only the owner or a community owner/admin may replace or remove it
- a removal is a kind:30180 entry with no `emoji` tag; NIP-09 deletion of
  an entry is refused, and a moderation delete removes the emoji from the
  catalog in the same commit
- every accepted write takes the community's next change number under a
  per-community lock, so changes commit in order
- `POST /emoji/catalog` returns compact rows changed since a change
  number, or the live rows for up to 100 shortcodes

NIP-CE is updated to describe the catalog.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Bradley Axen <baxen@squareup.com>

@kalvinnchau kalvinnchau left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Three findings: upgrade/rollout reconciliation, removal retry idempotency, and superseded-event mention cleanup. Findings are based on source inspection; runtime reproductions were blocked.

Some(current.try_get::<Vec<u8>, _>("pubkey")?),
current.try_get::<bool, _>("live")?,
),
None => (None, false),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [P2] Reconcile existing entries before requiring catalog ownership

Migration 0089 starts with an empty catalog, but previously accepted kind-30180 events remain live. An existing author's removal therefore reaches NothingToRemove here, while both kind-5 removal paths are now rejected. Another member can claim the shortcode without retiring the original event because no previous catalog owner exists. Mixed-version rollout also permits old writers/deleters to leave the catalog inconsistent with events. Provide staged reconciliation and a compatible removal path, accounting for old writers (or explicitly quiescing these writes during rollout). Regression scenario: seed an entry through the previous writer, upgrade, remove as its author, and reclaim; assert one resolved live owner. This finding is from control-flow inspection, not an executed reproduction.

),
None => (None, false),
};
if url.is_none() && !live {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [P2] Accept an exact resend of a committed removal

If a removal commits but its OK is lost, resending the same signed event hits NothingToRemove because the row's url is already NULL. This check precedes replace_parameterized_event_in_transaction's duplicate detection, so a successful removal is reported as invalid instead of an accepted duplicate. Select the current event_id and return Unchanged for the identical event before removal/ownership checks. Add an exact-event resend regression; the existing repeated-removal test uses a new event and does not cover this case. Evidence is control-flow inspection; runtime reproduction was unavailable.

}

// Keep one live entry event per shortcode: retire the previous holder's.
if let Some(previous) = owner.filter(|owner| owner.as_slice() != author.as_slice()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [P3] Apply mention cleanup when retiring the previous owner's event

An emoji entry may carry an extra p tag, which insert_mentions_in_transaction indexes. Admin takeover retires that event with this raw UPDATE but leaves its event_mentions rows behind; the other supersede paths use superseded_with_mentions_sql! to remove them. Reuse that cleanup here so repeated takeovers do not retain orphan mention rows and supersede lifecycle rules stay consistent. The observed impact is retained rows, not a demonstrated query leak. Evidence is source inspection.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants