Repository navigation
Retrying on SolidQueue::Processes errors doesn't work #786
Description
From #313, I understood that it is expected behaviour that we can configure a job to be retried in this case. This doesn't seem to work however.
From #422 (comment), I understood that it is somewhat expected that you can't actually use retry_on for these errors, but this isn't clearly documented IMO.
We have a setup where we run some jobs on very cheap machines that can be shut down at any moment, so we would like to be able to use retry_on on these jobs (or at least a similar mechanism).
Reacted by Tom Naessens, Nikita Elfimov, Bart Mesuere and Peter Dawyndt
Activity
rosa commentedon Aug 29, 2026
Hey @chvp, thanks for the report.
retry_oncan't catch these, and it's not something we can make work:ProcessPrunedErrorandProcessExitErrorare never raised inside your job. They're recorded by another process (the one pruning the dead worker, or the supervisor reaping a crashed fork), which writes the failed execution after the fact, and Active Job's retry machinery only sees exceptions raised whileperformruns.You're right that the docs were lacking. Since v1.7.0 there's a section on jobs interrupted by non-graceful process death with the reasoning (the job itself may be what's killing the process, so retrying blindly can take down the next worker too) and a recipe for your own recovery policy: subscribe to the
fail_many_claimed.solid_queueevent, which includes the error and the job IDs, and re-enqueue the jobs you know are safe to retry. For preemptible machines, that's the mechanism I'd use. #789 adds an explicit note thatretry_ondoesn't apply. I'll close this one; #591 has more on why this stays loud by default.chvp commentedon Aug 29, 2026
Hi @rosa, thanks for the response.
While I don't mind if that is the conclusion, I do wonder if it wouldn't be possible to actually handle this. It seems to me that at the time the job is marked as failed, Solid Queue has both a job instance and an exception instance. The exception handling code is also just a method in ActiveJob (well, ActiveSupport to be precise, but it's included in the Base job class). So I would think it would be possible to call that method at that time?
As for the reasoning that this could be killing workers, I definitely do get that, but I'd argue that the ActiveJob
retry_ontooling is a much more ergonomic way to prevent that than subscribing to that event and reimplementing fail counters/delays/... oneself. If the Errors used wouldn't inherit from StandardError but from Exception I would think that that'd be warning of "here be dragons" to experienced Ruby developers.Document that retry_on can't intercept process-death errors