Skip to content

derivative of norm at 0 #243

Open

Description

@goretkin

norm is not differentiable at 0, so at best you can return a subgradient. It appears that the subgradient is 1.0 at 0.0 (and -1.0 at -0.0).

julia> ForwardDiff.gradient(norm, [0.0, 0.0])
2-element Array{Float64,1}:
 0.0
 1.0

julia> ForwardDiff.gradient(norm, [0.0, -0.0])
2-element Array{Float64,1}:
 -0.0
 -1.0

I'm wondering if it would be worth it to define Base.norm on ForwardDiff.Dual, and return a subgradient of 0.0 at both 0.0 and -0.0

Also perhaps I missed this, but I think it would be nice to mention somewhere that in generic auto-diffable code sqrt(sum(v.^2)) should be replaced with norm, since sqrt is singular at 0, and produces a NaN when composed with a function with 0 gradient (0*Inf = NaN).

Activity

  1. cortner commented on Mar 2, 2018

    @cortner

    Here is an interesting effect that I am guessing is related?

    using ForwardDiff, StaticArrays
    # - ForwardDiff                   0.7.3
    # - StaticArrays                  0.6.6
    u = x ->  (1.0 + norm(x)^2)^(-1/2)
    ∇u = x -> ForwardDiff.gradient(u, x)
    ∇u(zeros(2))
    # 2-element Array{Float64,1}:
    #  -0.0
    #  -0.0
    ∇u(@SVector zeros(2))
    # 2-element SVector{2,Float64}:
    #  NaN
    #  NaN

    (u = x -> (1.0 + sum(x.^2))^(-1/2) works fine for both)

  2. taylormcd commented on Oct 8, 2019

    @taylormcd

    Here's something to think about related to this issue:

    using ForwardDiff
    using LinearAlgebra
    
    # start with zero valued vector of dual numbers
    v = zeros(ForwardDiff.Dual{Nothing, Float64, 1}, 3);
    
    # assume a perturbation of one component exists due to some computational noise
    value = 1.0e-200 # so that value^2 == 0.0 (due to machine precision)
    partial = 1.0e-100 # so that 2*value*partial != 0.0 (due to machine precision)
    v[1] = ForwardDiff.Dual{Nothing}(value, partial);
    
    # try out the two methods
    norm(v)
    # Dual{Nothing}(1.0e-200,NaN)
    sqrt(sum(v.^2))
    # Dual{Nothing}(0.0,Inf)

    Both implementations will result in NaNs propagated throughout the function, even in NaN-safe mode. I encountered this when propagating derivatives through a Newton solve and it took me a lot of time to find the issue.

  3. ferrolho commented on Jan 22, 2020

    @ferrolho
    Contributor

    (...) and it took me a lot of time to find the issue.

    Could you please share how you ended up working around it?

  4. taylormcd commented on Apr 13, 2021

    @taylormcd

    I created a new issue related to my comment as it is somewhat tangential to this issue. I'll post my workaround there.

  5. dlfivefifty commented on Jun 29, 2026

    @dlfivefifty
    Contributor

    The behaviour seems to have changed:

    julia> ForwardDiff.gradient(norm, [0.0,0.0])
    2-element Vector{Float64}:
     NaN
     NaN

    To avoid an argument on what the gradient is at the origin, we can instead consider the norm of a vector of dual numbers:

    julia> norm([Dual(0.0, (1.0,0.0)), Dual(0.0,(0.0,1.0))])
    Dual{Nothing}(NaN,NaN,NaN)
    

    Here this is well-defined since the dual part gives a specific direction for which to differentiate. (Though I guess only if the number of εs equals the length of the vector....)

  6. devmotion commented on Jun 30, 2026

    @devmotion
    Member

    I assume you didn't enable NaN-safe mode? IMO that's the "correct mode", but for historical and performance reasons it's not enabled by default.

  7. dlfivefifty commented on Jul 1, 2026

    @dlfivefifty
    Contributor

    I didn't realise there's a NaN-safe mode...

  8. stevengj commented on Jul 5, 2026

    @stevengj
    Contributor

    I don't think that changing this to return NaNs is an improvement over returning a subgradient. It breaks the common pattern of differentiating norm(x)^2, which works fine if any finite subgradient is returned from norm at zero but now gives NaN.

    See also JuliaLang/LinearAlgebra.jl#1662 (comment)

    (In contrast, differentiating abs(x)^2 works fine, because differentiating abs(x) returns a subderivative at zero, and in particular a one-sided derivative.)

  9. stevengj commented on Jul 5, 2026

    @stevengj
    Contributor

    See also #794

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions