[dromagnoli]

Add support for In-memory tile caching and provide Hazelcast implementation.
The in-memory tile cache keeps the most requested tiles in memory, in front of the blob stores. It is meant for layers whose tiles expire quickly and are requested by many clients at once, where reading each tile from disk or from a cloud store costs more than rendering time saved.

Downstream integration failures are expected.
A dedicated GS PR is available for that:
geoserver/geoserver#10214

[aaime]

Thanks for this. The direction looks good, and the outage handling and the Hazelcast usage show a lot of care. The inline comments below fall into four themes:

• Structure and naming. Names that collide with existing GWC concepts, and state and logic split between TileCache and CachingBlobStore. Please follow the existing GWC patterns (BlobStoreInfo as a factory) and
  pull the outage state machine into its own class.
• Consistency of the cache. Read-through races, and truncates during an outage that leave stale tiles behind. A truncate should fail rather than half succeed.
• Cluster locks. Not wired, and more complex than they need to be.
• Configuration. Not reachable through REST.

In addition, What's thHazelcast bulk calls have no timeout. delete, removeAll, aggregate, keySet, addIndex and clear can wait 60 s on a member that doesn't answer, while holding the configLock read lock.
The sorted index is added to the customer's map at runtime. It's undocumented, and if it fails, every truncate fails.

I'll follow up with the corresponding geoserver review

[dromagnoli]

All Feedbacks have been addressed.

[aaime]

Thanks for the rework. The split into MemoryCache and GuardedStore, the KeyValueStoreInfo factories, the wired lock provider and the REST endpoint answer the first round well, and most earlier points are
resolved.

Two points still need a change before merge, both covered inline:

• HazelcastKeyValueStore.putIfAbsent has no deadline. With passthrough on, every read-through can hold a request thread and the blob store read lock for up to 60 s on a silent member.
• Tile and range deletes fail while the cache is down. That stops disk quota, and with passthrough on the blob store fills up during a long outage.

[dromagnoli]

The 2 remaining points have been covered

[geoserver-bot]

The backport to 1.28.x failed:

The process '/usr/bin/git' failed with exit code 1

stderr

error: could not apply 949d5a501... Hazelcast implementation of GWC-distributed in-memory tilecache
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

stdout

Auto-merging geowebcache/core/src/main/java/org/geowebcache/config/XMLConfiguration.java
Auto-merging geowebcache/core/src/main/java/org/geowebcache/layer/TileLayer.java
Auto-merging geowebcache/core/src/main/resources/org/geowebcache/config/geowebcache.xsd
Auto-merging geowebcache/core/src/test/java/org/geowebcache/config/XMLConfigurationTest.java
Auto-merging geowebcache/pom.xml
CONFLICT (content): Merge conflict in geowebcache/pom.xml
Auto-merging geowebcache/web/pom.xml
Auto-merging geowebcache/web/src/test/java/org/geowebcache/jetty/RestIntegrationTest.java
CONFLICT (content): Merge conflict in geowebcache/web/src/test/java/org/geowebcache/jetty/RestIntegrationTest.java

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.28.x 1.28.x
# Navigate to the new working tree
cd .worktrees/backport-1.28.x
# Create a new branch
git switch --create backport-1612-to-1.28.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick 949d5a50139c2107b8708e9787e6282b05290af8
# Push it to GitHub
git push --set-upstream origin backport-1612-to-1.28.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.28.x

Then, create a pull request where the base branch is 1.28.x and the compare/head branch is backport-1612-to-1.28.x.

[dromagnoli]

Manual backport for 1.28.x:
#1618