Tired of flaky tests failing intermittently every time you run your automation pack? In this article, we will explore how to utilize the retry mechanism in Karate to automatically rerun failed scenarios and improve the overall efficiency of your testing process.
Below is an example feature file including 10 scenarios. Each scenario will randomly generate a number, test will fail if it’s an even number.
Feature: Retry
Scenario: Test 1
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 2
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 3
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 4
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 5
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 6
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 7
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 8
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 9
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
Scenario: Test 10
* def n = utils.randomUtil.getNumber(2)
* if (n % 2 == 0) karate.fail("odd numbers expected")
In TestRunner.java, from line 10 to 23, that’s where the magic happens:
- At line 11, we gather all the failed scenarios from the first run.
- attempt variable determines maximum how many times we want to retry those failed tests, or until there’s no more failed tests.
- At line 16, we loop through each scenario.
- At line 19, we retry the failed scenario.
- At line 20, we update the test results.
- At line 25, we generate the test report based on returned results.
- At line 26, we expect there should be no failed scenarios at all.
public class TestRunner {
@Test
void runTest() {
Results results = Runner.path("classpath:demo/retry.feature")
.outputCucumberJson(true)
.outputJunitXml(true)
.tags("~@ignore")
.parallel(5);
// Reference: https://github.com/karatelabs/karate/blob/develop/karate-core/src/test/java/com/intuit/karate/core/retry/RetryTest.java
List<ScenarioResult> failedScenarioResults = results.getScenarioResults().filter(s -> s.isFailed()).collect(Collectors.toList());
int attempt = 5;
while (failedScenarioResults.size() > 0 && attempt > 0) {
System.out.println(String.format("There is %d failed scenario(s)", failedScenarioResults.size()));
attempt--;
for (ScenarioResult failedScenarioResult : failedScenarioResults) {
Scenario scenario = failedScenarioResult.getScenario();
System.out.println("Retrying: " + scenario.getName());
ScenarioResult scenarioResult = results.getSuite().retryScenario(scenario);
results = results.getSuite().updateResults(scenarioResult);
}
failedScenarioResults = results.getScenarioResults().filter(s -> s.isFailed()).collect(Collectors.toList());
}
generateReport(results.getReportDir());
assertEquals(0, results.getFailCount(), results.getErrorMessages());
}
public static void generateReport(String karateOutputPath) {
Collection<File> jsonFiles = FileUtils.listFiles(new File(karateOutputPath), new String[] { "json" }, true);
List<String> jsonPaths = new ArrayList<>(jsonFiles.size());
jsonFiles.forEach(file -> jsonPaths.add(file.getAbsolutePath()));
Configuration config = new Configuration(new File("target"), "PartnerIntegrationAutomation");
ReportBuilder reportBuilder = new ReportBuilder(jsonPaths, config);
reportBuilder.generateReports();
}
}
Now let’s run the test and see how it goes.
At the 1st run, only 2 tests passed, while there are 8 tests failed.
---------------------------------------------------------
feature: classpath:demo/retry.feature
scenarios: 10 | passed: 2 | failed: 8 | time: 0.2201
---------------------------------------------------------
>>> failed features:
odd numbers expected
classpath:demo/retry.feature:7
odd numbers expected
classpath:demo/retry.feature:11
odd numbers expected
classpath:demo/retry.feature:15
odd numbers expected
classpath:demo/retry.feature:23
odd numbers expected
classpath:demo/retry.feature:31
odd numbers expected
classpath:demo/retry.feature:35
odd numbers expected
classpath:demo/retry.feature:39
odd numbers expected
classpath:demo/retry.feature:43
<<<
There is 8 failed scenario(s)
Retrying: Test 1
---------------------------------------------------------
feature: classpath:demo/retry.feature
scenarios: 10 | passed: 3 | failed: 7 | time: 0.1883
---------------------------------------------------------
...
Karate version: 1.3.1 | env: uat
======================================================
elapsed: 5.77 | threads: 5 | thread time: 0.05
features: 1 | skipped: 0 | efficiency: 0.00
scenarios: 10 | passed: 8 | failed: 2
======================================================
>>> failed features:
odd numbers expected
classpath:demo/retry.feature:23
odd numbers expected
classpath:demo/retry.feature:43
<<<
There is 2 failed scenario(s)
Retrying: Test 5
...
Karate version: 1.3.1 | env: uat
======================================================
elapsed: 5.77 | threads: 5 | thread time: 0.05
features: 1 | skipped: 0 | efficiency: 0.00
scenarios: 10 | passed: 10 | failed: 0
======================================================
Loading...