In this post, we will learn to rerun failed test cases using TestNG. We will explore the two approaches to achieve this, namely - using the testng-failed.xml file and by implementing the testNG IRetryAnalyzer.
Sometimes after some bug fixes, as a test automation engineer we are required to run only the failed tests reported by test automation suite. Running only the failed tests validate the bug fixes quickly.
Running only the failed tests is fairly simple as TestNG provides inherent support for this. Whenever a test suite is run using the testng.xml file then after the test execution, a testng-failed.xml file gets created in the test-output folder. Later on, we can run this file just like we run the testng.xml file. As this file only keeps track of the failed tests, so running this file, runs the failed tests only.
At times, the test execution report comes up with some failures that are not because of the issues in the application. The underlying cause of these issues might be related to the test environment setup or some occasional server issue. In order to make sure that the failure reported in the test report are genuine and not just one-off cases, we can retry running the failed test cases to eliminate false negative tests results in our test reports.
For retrying the failure test runs automatically during the test run itself, we need to implement the IRetryAnalyzer interface provided by TestNG. The IRetryAnalyzer interface provide methods to control retrying the test runs. Here, we will override the retry() method of IRetryAnalyzer to make sure the test runs in case of failure with specififed number of retry limit. The comments in the code snippet make it self-explainatory.
For the demo, I am creating a dummy test method below and intentionally failing it using the assert.fail() method. Here, we will set the @Test annotation's retryAnalyzer attribute with RetryAnalyzer.class that we created above.
Here, we can observe that the number of runs for the method are displayed as 2 with one failure and one skipped. The first run of the test method when failed will be marked as Skipped and then the test will run again due to the logic specified in the RetryAnalyzer.
Now, there is just one problem, we need to set the retryAnalyzer attribute in each of the @Test annotation. To deal with this we can implement IAnnotationTransformer interface. This interface provides the capability to alter the testNG annotation at runtime. So, we will create a class implementing the IAnnotationTransformer and make it set the RetryAnalyzer for @Test annotations.
Having created a listener, we can specify it in the testNG.xml file like this-
Now, we don't have to set the retryAnalyzer in each @Test annotation. Having the listener specified in the testng.xml file will make it work for all the tests.
PS: If you want to add retry functionality to only limited set of test method than just set the @Test annotations with retryAnalyzer attribute with RetryAnalyzer.class, there is no need to implement IAnnotationTransformer and adding the listener in the testng.xml file.