Skip to content

fix(persistence): isolate stores on KeyValueStore and Redis backings - #9223

Merged
tim-smart merged 8 commits into
v4/next-minorfrom
agent/architect-effect-bot/9351522757e7
Oct 15, 2026
Merged

tim-smart merged 8 commits into
v4/next-minorfrom
agent/architect-effect-bot/9351522757e7

Conversation

@effect-bot

@effect-bot effect-bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Warning

This changes the persisted key layout and needs maintainer approval before merge. Entries written by earlier versions are not read back, and there is no legacy read fallback. See "Compatibility" below, particularly the Chat.layerPersisted impact.

Persistence stores created with different storeId values were not isolated on layerKvs and layerRedis. Clearing one store could delete another store's entries, and some store/key pairs mapped to the same backing key (a/bc and ab/c on KVS, a/b:c and a:b/c on Redis).

Changes

  • Both backings store entries under <storeId>:<key>, with % → %25 and : → %3A applied to the store id. An encoded id never contains :, so the first colon always ends the store namespace and two store/key pairs can't produce the same key.
  • Redis clear glob-escapes the namespace (\ * ? [ ]) before KEYS, so c* or cache no longer match other stores. The KEYS → SCAN refactor is left out of this PR.
  • New KeyValueStore.clearPrefix(prefix). layerBackingKvs uses it through KeyValueStore.prefix, whose clear now only removes keys under the prefix instead of clearing the whole backing store. Implementations:
    • memory: walk the map
    • file system: list the directory and decode file names
    • SQL: LIKE … ESCAPE '!' on pg. On mysql and mssql the same LIKE (so the primary-key index is still used) plus a binary second predicate, because the default collations ignore case and accents: CONVERT(id USING utf8mb4) COLLATE utf8mb4_bin LIKE … on mysql, id COLLATE Latin1_General_100_BIN2 LIKE … on mssql. On sqlite, substr(id, 1, length(?)) = ?, because sqlite LIKE ignores ASCII case
    • Web Storage: collect matching keys, then remove them
    • IndexedDB (@effect/platform-browser): delete one exact key range
  • Third-party stores built with KeyValueStore.make that don't provide clearPrefix get a default that delegates to clear for "" and fails with KeyValueStoreError otherwise, so they never silently wipe unrelated data.
  • size / isEmpty on a prefixed view still count the whole backing store. That's unchanged and out of scope.

Compatibility

  • Persistence.layerKvs / layerBackingKvs: every existing entry becomes unreachable. Keys used to be <storeId><key> with no separator. Cached PersistedCache / RequestResolver results are recomputed. Chat.layerPersisted on a KeyValueStore backing loses its stored chat histories.
  • Persistence.layerRedis / layerBackingRedis: keys don't change unless the store id contains % or :. Entries for those store ids become unreachable.
  • Old entries aren't migrated or deleted. A read fallback was rejected on purpose: a legacy KVS key such as abc can't be attributed to a/bc or ab/c, and since clear can't target legacy keys, a fallback would bring entries back after a clear.
  • Users whose store ids are never prefixes of one another can migrate legacy KVS entries by hand: rename <storeId><key> to <encodedStoreId>:<key>.
  • The changeset includes the same warnings.

Known limitation: case-insensitive SQL keys on MySQL / MSSQL

This PR does not make store isolation case-sensitive on the SQL-backed stores for MySQL and MSSQL. KeyValueStore.layerSql (and the SQL Persistence backings) create their key columns with the server's default collation, which is usually case-insensitive and on MySQL 8 also accent-insensitive. So keys that differ only in case or accents are the same row. That's existing behavior. Through Persistence.layerKvs on these databases, store ids scope and SCOPE still alias for the same entry key (scope:1 / SCOPE:1).

This PR only makes clearPrefix exact on those dialects, so clearing scope no longer deletes SCOPE's entries with different keys. get and set still collapse case. Changing the key-column collation needs a decision about existing tables (CREATE TABLE IF NOT EXISTS won't alter them), so it's tracked separately as EFF-2228. Memory, file system, Web Storage, IndexedDB, Redis, SQLite and Postgres compare keys exactly.

Validation

  • The regression suite committed earlier (packages/effect/test/persistence/BackingPersistenceTest.ts) now passes for layerBackingMemory and layerBackingKvs. Before this fix, both KVS tests failed.
  • The same suite is wired into NodeRedis.integration.test.ts. The testcontainers Redis couldn't start in my sandbox (rootless podman, setresuid failure), so I ran the suite against a local redis-server 8.10.2. Both tests pass now and both failed before this fix.
  • I also ran temporary checks that weren't committed:
    • The isolation suite over layerBackingKvs, backed by file system, SQLite, Postgres, MySQL 8.4, sessionStorage and IndexedDB stores.
    • A clearPrefix check with %, _, !, [ ] and � in keys against every backend above.
    • Everything passes except one case-variant key on MySQL. Two keys that differ only in case are one row there; see "Known limitation" above.
  • MSSQL: packages/sql/mssql/test/KeyValueStore.integration.test.ts runs the shared KeyValueStoreTest suite and a store-level check on a Latin1_General_100_CI_AS key column. In CI run 38709458929 at 80ab2ba84f it reported 11 passed, 0 skipped, 0 failed on Node, Bun and Deno. That run is the first execution of the Latin1_General_100_BIN2 predicate. It couldn't run locally (no podman policy.json).
  • Case-insensitive collations: on local MySQL 8.4.11 (utf8mb4_0900_ai_ci), the shared KeyValueStoreTest suite (including the case-sensitive clearPrefix regression and the escaping tests), BackingPersistenceTest, and a store-level scope / SCOPE check with different entry keys all pass (23 tests). Before the collation fix, the regression and the store check both failed.
  • Existing tests pass, 329 across 14 files: effect persistence / Cache / Request / Chat, browser KVS + persistence, node KVS, sqlite-node KVS + persistence.
  • pnpm lint passes, and tsc -b passes for effect, platform-browser, platform-node and sql-sqlite-node.

Closes #9207
Closes EFF-2205

Namespace entries as <storeId>:<key> with % and : percent-encoded in the
store id, and scope clear to the store. Add KeyValueStore.clearPrefix and
make KeyValueStore.prefix(...).clear prefix-scoped.

Closes #9207
@changeset-bot

changeset-bot Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0aa2b5b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 33 packages
Name Type
effect Minor
@effect/platform-browser Minor
@effect/opentelemetry Minor
@effect/vitest Minor
@effect/ai-anthropic Minor
@effect/ai-cloudflare Minor
@effect/ai-openai-compat Minor
@effect/ai-openai Minor
@effect/ai-openrouter Minor
@effect/ai-typesafe Minor
@effect/atom-react Minor
@effect/atom-solid Minor
@effect/atom-vue Minor
@effect/platform-bun Minor
@effect/platform-deno Minor
@effect/platform-node-shared Minor
@effect/platform-node Minor
@effect/sql-clickhouse Minor
@effect/sql-d1 Minor
@effect/sql-libsql Minor
@effect/sql-mssql Minor
@effect/sql-mysql2 Minor
@effect/sql-pg Minor
@effect/sql-pglite Minor
@effect/sql-sqlite-bun Minor
@effect/sql-sqlite-do Minor
@effect/sql-sqlite-node Minor
@effect/sql-sqlite-react-native Minor
@effect/sql-sqlite-wasm Minor
@effect/docgen Minor
@effect/doctest Minor
@effect/graphql-generator Minor
@effect/openapi-generator Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-janitor effect-janitor Bot added bug Something isn't working 4.0 labels Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 38.99 KB 38.99 KB 0.00 KB (0.00%)
basic.ts 7.02 KB 7.02 KB 0.00 KB (0.00%)
batching.ts 10.74 KB 10.74 KB 0.00 KB (0.00%)
brand.ts 6.71 KB 6.71 KB 0.00 KB (0.00%)
cache.ts 10.96 KB 10.96 KB 0.00 KB (0.00%)
config.ts 22.41 KB 22.41 KB 0.00 KB (0.00%)
differ.ts 21.52 KB 21.52 KB 0.00 KB (0.00%)
http-client.ts 23.66 KB 23.66 KB 0.00 KB (0.00%)
http-router.ts 35.19 KB 35.19 KB 0.00 KB (0.00%)
logger.ts 11.28 KB 11.28 KB 0.00 KB (0.00%)
metric.ts 9.09 KB 9.09 KB 0.00 KB (0.00%)
optic.ts 6.94 KB 6.94 KB 0.00 KB (0.00%)
pubsub.ts 15.38 KB 15.38 KB 0.00 KB (0.00%)
queue.ts 12.21 KB 12.21 KB 0.00 KB (0.00%)
schedule.ts 11.41 KB 11.41 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 13.76 KB 13.76 KB 0.00 KB (0.00%)
schema-binary.ts 40.18 KB 40.18 KB 0.00 KB (0.00%)
schema-class.ts 21.20 KB 21.20 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 32.24 KB 32.24 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 27.41 KB 27.41 KB 0.00 KB (0.00%)
schema-string-transformation.ts 14.61 KB 14.61 KB 0.00 KB (0.00%)
schema-string.ts 12.02 KB 12.02 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.19 KB 16.19 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 38.51 KB 38.51 KB 0.00 KB (0.00%)
schema-toCodeDocument.ts 25.53 KB 25.53 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 20.45 KB 20.45 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 20.66 KB 20.66 KB 0.00 KB (0.00%)
schema-toFormatter.ts 20.75 KB 20.75 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 25.38 KB 25.38 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 20.74 KB 20.74 KB 0.00 KB (0.00%)
schema.ts 20.39 KB 20.39 KB 0.00 KB (0.00%)
stm.ts 13.48 KB 13.48 KB 0.00 KB (0.00%)
stream.ts 10.05 KB 10.05 KB 0.00 KB (0.00%)

@effect-bot
effect-bot changed the base branch from main to v4/next-minor October 15, 2026 09:23
@effect-janitor effect-janitor Bot removed the 4.0 label Oct 15, 2026
@tim-smart
tim-smart enabled auto-merge (squash) October 15, 2026 09:25
@effect-bot
effect-bot force-pushed the v4/next-minor branch 3 times, most recently from b2d972b to c0aa639 Compare October 15, 2026 09:28
@tim-smart
tim-smart merged commit 7f8785d into v4/next-minor Oct 15, 2026
12 checks passed
@tim-smart
tim-smart deleted the agent/architect-effect-bot/9351522757e7 branch October 15, 2026 09:31
@effect-bot effect-bot mentioned this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 15, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
effect-bot added a commit that referenced this pull request Oct 16, 2026
tim-smart added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
effect-bot added a commit that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working v4/next-minor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Persistence: stores on layerKvs and layerRedis can clear and read each other's entries

2 participants