Skip to content

Test the unhappy path - #60

Merged
oxinabox merged 9 commits into
masterfrom
ox/metatest
Oct 9, 2020
Merged

oxinabox merged 9 commits into
masterfrom
ox/metatest

Conversation

@oxinabox

@oxinabox oxinabox commented Oct 6, 2020

Copy link
Copy Markdown
Member

This PR is about testing that we can actually detect when things are wrong.

The code for metatest_get_failures is a pretty nasty hack an on the internals.
I think its clear enough what it is doing.

Two possible alternative implementations:

Alt 1: use a seperate process.

Julia itself does this to test @test.
Writing a test to a file and then kicking it off with run(`julia -e file.jl`)
But three problems are

  1. Slow starting an external process is slow
  2. Very course grained: all that is easy to test is that it caused julia to to exit with a nonzero status
  3. Pretty ugly code, need to redirect stdout still and need to write a file

Alt 2: custom Testset type

We could declare a custom testset type which overloads record and finish so that they don't propate errors upwards.
This might be fiddly, but I have done something like it before in https://github.com/JuliaTesting/ReferenceTests.jl.
so could base it off that.
It should be simplier than that.
It could be pretty clean maybe. Not 100% sure.

Comment thread test/meta_testing_tools.jl Outdated
Invoking it via `metatest_get_failures(f)` will prevent those `@test` being added to the
current testset, and will return a collection of all nonpassing test results.
"""
function metatest_get_failures(f)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this just be called

Suggested change
function metatest_get_failures(f)
function get_failures(f)

?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, extract_failures or fail_results (since they are Test.Fail <: Test.Result) or just failures?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are any non-pass.
So they could be Errors, or some other cause, I don't recall what @test_broken returns.
Arguably we should call it nonpasses

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah nonpasses is better (to avoid the failures = Fail confusion i had)

test_broken also treats all non-passes the same

julia> @test_broken 1 == 2
Test Broken
  Expression: 1 == 2

julia> @test_broken error()
Test Broken
  Expression: error()

Comment thread test/testers.jl Resolved
@nickrobinson251
nickrobinson251 self-requested a review October 7, 2020 10:28

@nickrobinson251 nickrobinson251 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(reviewed with @bencottier)

one design question, and a couple naming comments, but otherwise i think the broadly implementation looks good -- i've asked Ben to follow-up with any comments on the implementation later this afternoon

overall Alt 1 and (probably) Alt 2 both seem less nice to us

Comment thread test/testers.jl Outdated
@testset "unhappy path" begin
alt_double(x) = 2x
@scalar_rule(alt_double(x), 3) # this is wrong, on purpose
failures = metatest_get_failures(()->test_scalar(alt_double, 2.1))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we want to test that test_scalar returns a test failure (rather than a pass or an error), when the @scalar_rule is wrong -- is that right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder if the testset name can be more precise on that, than "unhappy path"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we want to make sure it doesn't succeed.

I can't think of a better name.
Unhappy Path is a term of literature
https://cucumber.io/blog/test-automation/happy-unhappy-paths-why-you-need-to-test-both/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we want to make sure it doesn't succeed.

i am confused by this -- why don't we only care about failures?
(also current naming is confusing to me if it includes more than Test.Fails)
https://github.com/JuliaDiff/ChainRulesTestUtils.jl/pull/60/files#r501079347

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is also why i prefer to be more precise than "unhappy path" ...unless we explicitly document that "unhappy path" means "anything that doesn't result in only Test.Pass"

Comment thread test/meta_testing_tools.jl Outdated
Invoking it via `metatest_get_failures(f)` will prevent those `@test` being added to the
current testset, and will return a collection of all nonpassing test results.
"""
function metatest_get_failures(f)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, extract_failures or fail_results (since they are Test.Fail <: Test.Result) or just failures?

Comment thread test/meta_testing_tools.jl Resolved
Comment thread test/testers.jl Resolved
Comment thread test/testers.jl Outdated
return 2.5 * x, identity_pullback
end

@test !isempty(metatest_get_failures(()->frule_test(my_identity1, (2.2, 3.3))))

@nickrobinson251 nickrobinson251 Oct 7, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do wonder if there's an alternative design (or just another layer of abstraction?)
so we end up with a call more like

@test_failure frule_test(my_identity1, (2.2, 3.3))

or

@test fails(() -> frule_test(my_identity1, (2.2, 3.3))

?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah i think the later makes sense.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

out of interest, why not the macro (that just wraps the call in an anonymous function and passes it to fails)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the macro looks like @test_throws and @test_broken
but it does not at all work like those do.

@test_throws catchs the exception.
@test_broken x is very similar to @test !x + some extra stuff

I would worry that people might think @test_failure x is the same as same as @test !x

@nickrobinson251 nickrobinson251 Oct 7, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

people might think @test_failure x is the same as same as @test !x

yeah, great point.

Comment thread test/testers.jl Resolved
Co-authored-by: Nick Robinson <npr251@gmail.com>
@oxinabox

oxinabox commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

@nickrobinson251 you didn't comment on if either of the Alternative Implementations mentioned in the top post might be better.

@nickrobinson251

nickrobinson251 commented Oct 7, 2020 •

Copy link
Copy Markdown
Contributor

@nickrobinson251 you didn't comment on if either of the Alternative Implementations mentioned in the top post might be better.

We did (above). We think Alt 1 is clearly worse. Alt 2 we didn't think too hard about but I couldn't see what we'd gain over the current implementation

@oxinabox

oxinabox commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

Ah i missed it

overall Alt 1 and (probably) Alt 2 both seem less nice to us

Comment thread test/meta_testing_tools.jl Resolved
_extract_failures(ts::Test.DefaultTestSet) = _extract_failures(ts.results)
function _extract_failures(xs::Vector)
if isempty(xs)
return Test.Result[]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If f actually doesn't call @test, we end up here -- is this the best thing to return in that case?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so?

@nickrobinson251 nickrobinson251 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💚

Comment thread test/meta_testing_tools.jl Resolved
Comment thread test/meta_testing_tools.jl Resolved
Comment thread test/meta_testing_tools.jl Resolved
Comment thread test/meta_testing_tools.jl Resolved
Co-authored-by: Nick Robinson <npr251@gmail.com>
Comment thread test/testers.jl Resolved
@oxinabox

oxinabox commented Oct 9, 2020

Copy link
Copy Markdown
Member Author

Not going to tag a release off this as it is not a functional change

@oxinabox
oxinabox merged commit 1bfe367 into master Oct 9, 2020
@oxinabox
oxinabox deleted the ox/metatest branch October 9, 2020 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants