Repository navigation
Open
Labels
api: pubsubIssues related to the Pub/Sub API.Issues related to the Pub/Sub API.priority: p3Desirable enhancement or fix. May not be included in next release.Desirable enhancement or fix. May not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Description
Environment details
- OS: Ubuntu
- Ruby version: 3.1.2
- Gem name and version:
google-cloud-pubsub (2.15.1)
Steps to reproduce
Not sure if this is a bug or intended, but two things:
- If an error happens during publishing, using the async publisher, it is not retried unless you also supply an ordering key:
rescue StandardError => e
items = batch.items
unless batch.ordering_key.empty?
retry if publish_batch_error_retryable? e # why are retries only done when ordering key present?This was added in: #3685. How come retrying in case of errors should not be done if you don't have an ordering key? Or is that a bug?
- If you do have an ordering key, it will retry any of the following errors, indefinitely:
PUBLISH_RETRY_ERRORS = [
GRPC::Cancelled, GRPC::DeadlineExceeded, GRPC::Internal,
GRPC::ResourceExhausted, GRPC::Unauthenticated, GRPC::Unavailable
].freezeConsulting the documentation (https://cloud.google.com/pubsub/docs/reference/error-codes) it looks like some of these are incorrect. For instance GRPC::Unauthenticated should not be retried, and GRPC::ResourceExhausted should be retried with exponential backoff. It seems like it would be a problem if for instance GRPC::Unauthenticated keeps retrying forever?
Metadata
Metadata
Assignees
Labels
api: pubsubIssues related to the Pub/Sub API.Issues related to the Pub/Sub API.priority: p3Desirable enhancement or fix. May not be included in next release.Desirable enhancement or fix. May not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Activity
NivedhaSenthil commentedon Mar 8, 2023
Thanks for reporting this with details. I am trying to find some history behind these changes. Will post here as I get to know more info.
piotrdomagalski commentedon Nov 7, 2025
Is there any update on this? It seems the code is still looking like this in 3.0.3
quartzmo commentedon Oct 13, 2026
TL;DR
The first point is mostly covered already: publishes without an ordering key are retried, with backoff, by the underlying generated client. The second point looks like a bug: for a batch with an ordering key, the async publisher retries immediately and without limit, including on
Unauthenticated. It's unclear whyUnauthenticatedis on the retry list, but retrying without a limit or backoff shouldn't be the behavior either way. Keeping this open to fix the ordered-publish retry behavior.Discussion
1. Retries without an ordering key. Every publish goes through the generated
TopicAdmin::Client#publish(service.rb, via thepublish_internalalias). Its default retry policy retries ABORTED, CANCELLED, INTERNAL, RESOURCE_EXHAUSTED, UNKNOWN, UNAVAILABLE and DEADLINE_EXCEEDED (codes[10, 1, 13, 8, 2, 14, 4]), with exponential backoff (initial 0.1 s, multiplier 4, max 60 s) and a 60 s timeout. So unordered publishes are retried. The extra handwritten retry, added with ordering keys in #3685, applies only to ordered batches.2. Ordered batches retry without limit. On
main, when an ordered batch fails, the async publisher runsretry if publish_batch_error_retryable? ewith no backoff and no limit.PUBLISH_RETRY_ERRORSincludesGRPC::Unauthenticated, which the generated retry policy above doesn't retry. A persistent auth failure on an ordered publish therefore retries forever. This is still true for the latest release, google-cloud-pubsub 3.5.0.Changing this affects ordered delivery: when an ordering key should pause, and what the publish callback reports. So it will need careful requirements analysis rather than a quick patch.
Next steps
Keeping this open as a bug for fixing the ordered-publish retry loop.