Skip to content

Fix flaky test in DeliveryDetailsTest#709

Merged
authorjapps merged 2 commits into
authorjapps:masterfrom
Md-Arif-Hasan:master
Apr 23, 2025
Merged

Fix flaky test in DeliveryDetailsTest#709
authorjapps merged 2 commits into
authorjapps:masterfrom
Md-Arif-Hasan:master

Conversation

@Md-Arif-Hasan

@Md-Arif-Hasan Md-Arif-Hasan commented Apr 22, 2025

Copy link
Copy Markdown
Contributor

Fix Flaky Test in DeliveryDetailsTest.testSerDeser

What is the purpose of this PR?

This pull request addresses a flaky test, DeliveryDetailsTest.testSerDeser, in the zerocode project. The test was failing intermittently due to assumptions about the ordering of JSON fields during serialization and deserialization.

Why the test fails

The DeliveryDetailsTest.testSerDeser test relied on strict JSON field ordering during serialization. Gson, the JSON library used in the project, does not guarantee field order by default. This caused the test to fail when the field order changed, especially under NonDex shuffling, which explores non-deterministic behaviors.

How to reproduce the test failure

Run the test with NonDex to simulate shuffling of field order:

mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -pl core -Dtest=org.jsmart.zerocode.core.kafka.DeliveryDetailsTest#testSerDeser

Before the fix, the test failed intermittently with errors similar to:

Expected: is "{\"status\":\"Ok\",\"message\":\"test message\",\"size\":10}"
     but: was "{\"message\":\"test message\",\"size\":10,\"status\":\"Ok\"}"

Expected results

The test should pass consistently, regardless of the order of fields in the serialized JSON.

Actual results

Before the fix, the test failed intermittently due to changes in the order of JSON fields.

Description of fix

  1. Relaxed Assertions:
    • Replaced strict JSON string comparisons with JSONAssert assertions using the LENIENT mode. This ensures that field order does not affect the test results.
    JSONAssert.assertEquals("{\"status\":\"Ok\",\"message\":\"test message\",\"size\":10}", json, JSONCompareMode.LENIENT);
  2. Deterministic Behavior:
    • Configured Gson to ensure consistent serialization behavior using GsonBuilder:
    Gson gson = new GsonBuilder().serializeNulls().create();
  3. Improved Assertions:
    • Where applicable, compared deserialized objects directly instead of relying on string comparisons.

Validation

Test Results

  • Tests run normally: Passed
    mvn -pl core test -Dtest=org.jsmart.zerocode.core.kafka.DeliveryDetailsTest#testSerDeser
  • Tests run with NonDex: Passed consistently across multiple seeds
    mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -pl core -Dtest=org.jsmart.zerocode.core.kafka.DeliveryDetailsTest#testSerDeser

Additional Notes

  • The fix ensures the test is robust to non-deterministic behaviors of JSON serialization and compatible with future updates to Gson or related dependencies.

Let me know if further improvements or clarifications are needed!

@authorjapps

Copy link
Copy Markdown
Owner

Thanks for the PR @Md-Arif-Hasan. Once this CI build goes Green, we should be able to merge it.

@authorjapps authorjapps self-requested a review April 22, 2025 22:41
@authorjapps authorjapps merged commit 82c187c into authorjapps:master Apr 23, 2025
@Md-Arif-Hasan

Copy link
Copy Markdown
Contributor Author

@authorjapps Thank you so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants