Repository navigation
fix(persistence): isolate stores on KeyValueStore and Redis backings - #9223
Merged
tim-smart merged 8 commits intoOct 15, 2026
Merged
Conversation
🦋 Changeset detectedLatest commit: 0aa2b5b The changes in this PR will be included in the next version bump. This PR includes changesets to release 33 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
tim-smart
reviewed
Oct 15, 2026
tim-smart
enabled auto-merge (squash)
October 15, 2026 09:25
effect-bot
force-pushed
the
v4/next-minor
branch
3 times, most recently
from
October 15, 2026 09:28
b2d972b to
c0aa639
Compare
Merged
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 15, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 16, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
tim-smart
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
effect-bot
added a commit
that referenced
this pull request
Oct 20, 2026
…9223) Co-authored-by: Tim <hello@timsmart.co>
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
This changes the persisted key layout and needs maintainer approval before merge. Entries written by earlier versions are not read back, and there is no legacy read fallback. See "Compatibility" below, particularly the
Chat.layerPersistedimpact.Persistence stores created with different
storeIdvalues were not isolated onlayerKvsandlayerRedis. Clearing one store could delete another store's entries, and some store/key pairs mapped to the same backing key (a/bcandab/con KVS,a/b:canda:b/con Redis).Changes
<storeId>:<key>, with%→%25and:→%3Aapplied to the store id. An encoded id never contains:, so the first colon always ends the store namespace and two store/key pairs can't produce the same key.clearglob-escapes the namespace (\ * ? [ ]) beforeKEYS, soc*orcacheno longer match other stores. TheKEYS→SCANrefactor is left out of this PR.KeyValueStore.clearPrefix(prefix).layerBackingKvsuses it throughKeyValueStore.prefix, whoseclearnow only removes keys under the prefix instead of clearing the whole backing store. Implementations:LIKE … ESCAPE '!'on pg. On mysql and mssql the sameLIKE(so the primary-key index is still used) plus a binary second predicate, because the default collations ignore case and accents:CONVERT(id USING utf8mb4) COLLATE utf8mb4_bin LIKE …on mysql,id COLLATE Latin1_General_100_BIN2 LIKE …on mssql. On sqlite,substr(id, 1, length(?)) = ?, because sqliteLIKEignores ASCII case@effect/platform-browser): delete one exact key rangeKeyValueStore.makethat don't provideclearPrefixget a default that delegates toclearfor""and fails withKeyValueStoreErrorotherwise, so they never silently wipe unrelated data.size/isEmptyon a prefixed view still count the whole backing store. That's unchanged and out of scope.Compatibility
Persistence.layerKvs/layerBackingKvs: every existing entry becomes unreachable. Keys used to be<storeId><key>with no separator. CachedPersistedCache/RequestResolverresults are recomputed.Chat.layerPersistedon aKeyValueStorebacking loses its stored chat histories.Persistence.layerRedis/layerBackingRedis: keys don't change unless the store id contains%or:. Entries for those store ids become unreachable.abccan't be attributed toa/bcorab/c, and sinceclearcan't target legacy keys, a fallback would bring entries back after a clear.<storeId><key>to<encodedStoreId>:<key>.Known limitation: case-insensitive SQL keys on MySQL / MSSQL
This PR does not make store isolation case-sensitive on the SQL-backed stores for MySQL and MSSQL.
KeyValueStore.layerSql(and the SQLPersistencebackings) create their key columns with the server's default collation, which is usually case-insensitive and on MySQL 8 also accent-insensitive. So keys that differ only in case or accents are the same row. That's existing behavior. ThroughPersistence.layerKvson these databases, store idsscopeandSCOPEstill alias for the same entry key (scope:1/SCOPE:1).This PR only makes
clearPrefixexact on those dialects, so clearingscopeno longer deletesSCOPE's entries with different keys.getandsetstill collapse case. Changing the key-column collation needs a decision about existing tables (CREATE TABLE IF NOT EXISTSwon't alter them), so it's tracked separately as EFF-2228. Memory, file system, Web Storage, IndexedDB, Redis, SQLite and Postgres compare keys exactly.Validation
packages/effect/test/persistence/BackingPersistenceTest.ts) now passes forlayerBackingMemoryandlayerBackingKvs. Before this fix, both KVS tests failed.NodeRedis.integration.test.ts. The testcontainers Redis couldn't start in my sandbox (rootless podman,setresuidfailure), so I ran the suite against a localredis-server8.10.2. Both tests pass now and both failed before this fix.layerBackingKvs, backed by file system, SQLite, Postgres, MySQL 8.4, sessionStorage and IndexedDB stores.clearPrefixcheck with%,_,!,[ ]and�in keys against every backend above.packages/sql/mssql/test/KeyValueStore.integration.test.tsruns the sharedKeyValueStoreTestsuite and a store-level check on aLatin1_General_100_CI_ASkey column. In CI run 38709458929 at80ab2ba84fit reported 11 passed, 0 skipped, 0 failed on Node, Bun and Deno. That run is the first execution of theLatin1_General_100_BIN2predicate. It couldn't run locally (no podmanpolicy.json).utf8mb4_0900_ai_ci), the sharedKeyValueStoreTestsuite (including the case-sensitiveclearPrefixregression and the escaping tests),BackingPersistenceTest, and a store-levelscope/SCOPEcheck with different entry keys all pass (23 tests). Before the collation fix, the regression and the store check both failed.pnpm lintpasses, andtsc -bpasses for effect, platform-browser, platform-node and sql-sqlite-node.Closes #9207
Closes EFF-2205