Skip to content

HIVE-30250: Compactor tests are flaky because test classes reuse each other's temp directories - #6956

Open
kuczoram wants to merge 1 commit into
apache:masterfrom
kuczoram:HIVE-30250
Open

kuczoram wants to merge 1 commit into
apache:masterfrom
kuczoram:HIVE-30250

Conversation

@kuczoram

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • CompactorTest.setup(): delete the directory before creating it, so every test starts empty.
  • TestCompactionMetrics: add an After that calls compactorTestCleanup().

Why are the changes needed?

Compactor tests that extend CompactorTest can fail depending on which other test classes ran before them in the same Maven run. For example, TestWorker2#minorNoBaseLotsOfDeltas fails with:

java.lang.AssertionError: expected:<12> but was:<15>
    at org.apache.hadoop.hive.ql.txn.compactor.TestWorker.compactNoBaseLotsOfDeltas(TestWorker.java:644)

if it runs after TestCompactionMetrics#testWorkerPerfMetrics in the same split.

How to reproduce:
mvn -pl ql test -Dtest='TestCompactionMetrics#testWorkerPerfMetrics,TestWorker2#minorNoBaseLotsOfDeltas' -Dsurefire.runOrder=alphabetical

Cause
CompactorTest.setup() creates each test's directory as ${test.tmp.dir}/compactor_test_table_, where N comes from a static AtomicInteger. Surefire runs each test class in a new JVM (reuseForks=false), so N starts at 0 in every class. ql/target/tmp is only cleaned once per Maven run, so tests in different classes get the same directory.

TestCompactionMetrics never calls compactorTestCleanup(), so its files stay behind. When a later test reuses that directory and the same table name (mapwb), it finds old base and delta directories. The compactor then marks some deltas as obsolete, and the test's file count assertions fail.

Does this PR introduce any user-facing change?

No, it is just unit test fix

How was this patch tested?

By running the unit tests

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants