You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm wondering if it would be worth it to define Base.norm on ForwardDiff.Dual, and return a subgradient of 0.0 at both 0.0 and -0.0
Also perhaps I missed this, but I think it would be nice to mention somewhere that in generic auto-diffable code sqrt(sum(v.^2)) should be replaced with norm, since sqrt is singular at 0, and produces a NaN when composed with a function with 0 gradient (0*Inf = NaN).
Here's something to think about related to this issue:
using ForwardDiff
using LinearAlgebra
# start with zero valued vector of dual numbers
v =zeros(ForwardDiff.Dual{Nothing, Float64, 1}, 3);
# assume a perturbation of one component exists due to some computational noise
value =1.0e-200# so that value^2 == 0.0 (due to machine precision)
partial =1.0e-100# so that 2*value*partial != 0.0 (due to machine precision)
v[1] = ForwardDiff.Dual{Nothing}(value, partial);
# try out the two methodsnorm(v)
# Dual{Nothing}(1.0e-200,NaN)sqrt(sum(v.^2))
# Dual{Nothing}(0.0,Inf)
Both implementations will result in NaNs propagated throughout the function, even in NaN-safe mode. I encountered this when propagating derivatives through a Newton solve and it took me a lot of time to find the issue.
Here this is well-defined since the dual part gives a specific direction for which to differentiate. (Though I guess only if the number of εs equals the length of the vector....)
I don't think that changing this to return NaNs is an improvement over returning a subgradient. It breaks the common pattern of differentiating norm(x)^2, which works fine if any finite subgradient is returned from norm at zero but now gives NaN.
(In contrast, differentiating abs(x)^2 works fine, because differentiating abs(x) returns a subderivative at zero, and in particular a one-sided derivative.)
Activity
cortner commentedon Mar 2, 2018
Here is an interesting effect that I am guessing is related?
(
u = x -> (1.0 + sum(x.^2))^(-1/2)works fine for both)taylormcd commentedon Oct 8, 2019
Here's something to think about related to this issue:
Both implementations will result in NaNs propagated throughout the function, even in NaN-safe mode. I encountered this when propagating derivatives through a Newton solve and it took me a lot of time to find the issue.
ferrolho commentedon Jan 22, 2020
Could you please share how you ended up working around it?
taylormcd commentedon Apr 13, 2021
I created a new issue related to my comment as it is somewhat tangential to this issue. I'll post my workaround there.
dlfivefifty commentedon Jun 29, 2026
The behaviour seems to have changed:
To avoid an argument on what the gradient is at the origin, we can instead consider the norm of a vector of dual numbers:
Here this is well-defined since the dual part gives a specific direction for which to differentiate. (Though I guess only if the number of εs equals the length of the vector....)
devmotion commentedon Jun 30, 2026
I assume you didn't enable NaN-safe mode? IMO that's the "correct mode", but for historical and performance reasons it's not enabled by default.
dlfivefifty commentedon Jul 1, 2026
I didn't realise there's a NaN-safe mode...
stevengj commentedon Jul 5, 2026
I don't think that changing this to return
NaNs is an improvement over returning a subgradient. It breaks the common pattern of differentiatingnorm(x)^2, which works fine if any finite subgradient is returned fromnormat zero but now gives NaN.See also JuliaLang/LinearAlgebra.jl#1662 (comment)
(In contrast, differentiating
abs(x)^2works fine, because differentiatingabs(x)returns a subderivative at zero, and in particular a one-sided derivative.)stevengj commentedon Jul 5, 2026
See also #794