Skip to content

Testsuite seems to be passing, even though a specific tests fails. #537

Open

Description

@zmiklank
          Strange behaviour.

There is an error in test/run script:

-----------------------------------------------
Running test run_pgaudit_test (starting at 2023-10-03 10:26:59+00:00) ... 
-----------------------------------------------
c644c8223c59eb64a932a2d1cd2e13e9f3bee4a1c7fd79ec69e1df6e37a788c0
Created container c644c8223c59eb64a932a2d1cd2e13e9f3bee4a1c7fd79ec69e1df6e37a788c0
test/run: line 340: $1: unbound variable
Tests finished with EXIT=1

Tests are passing:

==============================================
Tests were run for image quay.io/sclorg/postgresql-15-c9s:1
==============================================
Test cases results:

 [PASSED] for 'postgresql-container_tests' run_container_creation_tests (00:00:19)
 [PASSED] for 'postgresql-container_tests' run_general_tests (00:00:39)
 [PASSED] for 'postgresql-container_tests' run_change_password_test (00:00:07)
 [PASSED] for 'postgresql-container_tests' run_replication_test (00:00:06)
 [PASSED] for 'postgresql-container_tests' run_doc_test (00:00:00)
 [PASSED] for 'postgresql-container_tests' run_s2i_test (00:00:14)
 [PASSED] for 'postgresql-container_tests' run_test_cfg_hook (00:00:08)
 [PASSED] for 'postgresql-container_tests' run_s2i_bake_data_test (00:00:12)
 [PASSED] for 'postgresql-container_tests' run_s2i_enable_ssl_test (00:00:08)
 [PASSED] for 'postgresql-container_tests' run_upgrade_test (00:00:00)
 [PASSED] for 'postgresql-container_tests' run_migration_test (00:00:23)

Originally posted by @phracek in #399 (comment)

Activity

  1. zmiklank commented on Nov 22, 2023

    @zmiklank
    ContributorAuthor

    It is little bit confusing what the log line Tests finished with EXIT=1 is saying to us.
    In reality it means, that the exit code of the test script was not zero, however, this does not mean, that the TESTSUITE_RESULT was success or that the tests finished successfully.
    I do not see the results of run_pgaudit_test in the logs that Peto has found. That could mean, that unbound variable led to exiting the whole testsuite even before the test result itself was recorded.
    This is not really bad behaviour, unfortunately from the log I can not see, if the overal test results were green or red. Everything leads to conclusion that they were red - so I consider this behaviour to be at least partly correct.

    What can we do in order to increase readibility of the logs:

    • change sentence Tests finished with EXIT=1 to something like Tests finished unexpectedly with EXIT=1
    • do not exit test script on unbound variable, just fail the specific testcase if this happens.

    From my POV this problem does not have high priority.
    @phracek Do you want to add something?

  2. phracek commented on Sep 4, 2024

    @phracek
    Member

    @SlouchyButton It looks like you are the expert for this issue. As you did it in s2i-ruby-container and s2i-nodejs-container. If you do not have time, I will fix it.

  3. added a commit that references this issue on Oct 8, 2024
  4. added a commit that references this issue on Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions