[deanq]

Summary

VolumeCache hydrate now logs its range retries and reports retry and failure counts. A volume that rejects metadata updates now produces one EPERM log line per operation instead of one per file.

Why

Hydrate retries a byte range up to 8 times, with backoff, on ENOMEM/EIO/EAGAIN, and it logged nothing when it did. A hydrate slowed down by FUSE (geesefs) pressure couldn't be diagnosed from logs; benchmarking had to monkeypatch _copy_range just to count retries.

Changes

• Per-retry debug line: VolumeCache: retrying <src> at offset <off> after ENOMEM (attempt N/8).
• Warning: a hydrate logs one warning when it passes _HYDRATE_RETRY_WARN_THRESHOLD = 64 range retries. Isolated retries are normal on object-store mounts and stay at debug level. Past this point backoff is materially slowing the hydrate.
• Summary: the existing hydrated N file(s) line now ends with (range retries: R, ranges given up: G). It is also logged when retries or failures happened but no file was restored; that case used to log nothing. hydrate() still returns the file count, so the API is unchanged.
• Counters: a per-hydrate _HydrateStats object holds them, guarded by a lock because ranges copy in parallel. Its as_dict() is meant to be folded into structured metrics later. The instrumentation never raises into the caller.
• Given-up count: a range counts as given up when it fails for good, either because retries ran out or because the error isn't transient. One failed file counts once, because its remaining ranges are skipped.
• Metadata EPERM: a per-operation _MetadataPolicy records the first EPERM from utime/chmod, logs it once, and skips both calls for the rest of that sync or hydrate. Errors other than EPERM are still logged per file. Reconciliation compares against the metadata recorded in the manifest, so skipping these calls is safe.
• Docs: docs/serverless/volume_cache.md has a new "Logging" section.

Test plan

• 12 new tests; OS errors are injected at os.pread / os.utime / os.chmod:
  • Retry logging: one debug line per retry, with errno, attempt, file and offset.
  • Summary counts: correct for clean runs, for retries, for exhausted retries and for non-retryable errors.
  • Warning threshold: fires exactly once above the threshold and not at it.
  • Concurrency: counts are exact with 16 parallel workers and 40 injected errors, and the files are byte-exact.
  • Failing instrumentation: a hydrate still succeeds when logging itself raises.
  • EPERM: logged once per operation (including the chmod-only case), detected fresh for each operation, and other metadata errors still logged per file.
• make quality-check: 2835 passed, coverage 94.39%
• Watch logs from a hydrate on a Global Store mount under load

[Copilot]

🟡 Changes recommended

Concurrent range failures can overcount failed files, metadata rejection can prevent manifest mtime restoration, and parallel sync races can make syscall-count tests flaky.

3 open issues

• Count failed files once after the worker pool drains · New
• Restore manifest mtimes after chmod is denied · New
• Serialize metadata-denial tests to avoid races · New

What changed in this PR

This PR improves VolumeCache observability for slow or failed hydrates and reduces repeated metadata-permission logging on network volumes.

Changes:

• Adds per-range retry logs, a retry-threshold warning, and retry/failure counts in hydrate summaries.
• Introduces per-operation metadata handling to suppress repeated EPERM logs and metadata calls.
• Adds tests for retry instrumentation, concurrency, and metadata failures, plus logging documentation.

File	Description
tests/​test_serverless/​test_utils/​test_rp_volume_cache.py	Adds retry-count, logging, concurrency, and metadata-error tests.
runpod/​serverless/​utils/​rp_volume_cache.py	Implements hydrate statistics, retry logging, and shared metadata policy.
docs/​serverless/​volume_cache.md	Documents retry summaries, warnings, and EPERM handling.

_{Review effort: Balanced}

[promptless]

Promptless documentation updates

• Document VolumeCache parallel hydrate options and copy warnings adds a section on reading VolumeCache hydrate logs to the Serverless volume cache page, covering range retry counts, the retry warning, and once-per-operation EPERM logging.