Optimizing Regression Testing Through Test Prioritization

Why Strategize Testing?

Regression testing is just one of many forms of testing that can take days or even weeks to complete. Regression testing intends to make sure updated functionalities work the same way as they did earlier for your software.

Imagine you are installing a new music system in your car, but then the heating stops working. Regression helps to identify such problems in the software development process to ensure a seamless transition between updates. The time taken for a regression suite is one of the major challenges and not the only one. However, the critical aspect is the discovery of faults at the later end of the regression. Most uncalled delays are due to late fault discovery and fix that overshoot the planned timelines. The strategy is to plug in predictability in the discovery of faults early in the regression cycle. Consider a situation where we have 20000 test cases to complete in a month. If 50 percent of the faults are discovered during the last ten days of the regression run, the first twenty days will result in underutilized resources. Strategizing enables good delivery, effective resource utilization, and reduction in delays.

Best Approaches for Test Prioritization

Regression testing is driven by both requirement or structural (e.g., bug fixes, etc.) changes. Testing based on structural changes is also termed as corrective regression. Test Case Prioritization for corrective regression is an important step in your arsenal in faults discovery to have early in your DevOps cycle. This enables early troubleshooting and bug fixing for a timely release. In situations with time constraints, partial execution can be carried out to maximize the detection of critical bugs or issues. Various approaches have been proposed for prioritization like traditional test code coverage to prioritize tests with the most coverage, requirement correlation, and weights. However, these techniques suffer from a common problem – data. Code coverage requires analysis at the code level and requirements-driven approach asks for assigning priority to requirements through customer interaction and then mapping the test cases for each of the requirements. Although the techniques are beneficial it incurs a repetitive cost to prepare the data and improve with a new set of specifications thrown in at regular intervals. What we have used is test case historical execution data which is usually captured in the testing tools and is relatively easy to obtain.

Why is software testing so important?

It is an undeniable fact that bugs exist in small or large software systems with varying degrees of functional complexity. This is where the testing helps uncover issues and formulate a fixing strategy plan. Bugs or issues discovered at the late stages of software development adds to the cost as bugs identification, assignment, unit testing, integration testing & regression must be carried out before the system goes live. But how often, can the projects afford a repetition of the process. This is expensive and not feasible. In large software systems, regression consumes a large chunk of testing time and plays a vital role in an almost bug- free system. An unoptimized regression can derail releases by delays due to faults getting detected towards the end of the cycle. Prioritization helps to overcome this problem by introducing predictability in the regression execution. Predictably means that a certain percentage of faults are always detected by executing a certain percentage of test cases. This means more time for a developer to fix issues during the execution of regression itself and executing important test cases early.

Test Implementation Design Techniques

Test Case Prioritization leverages Genetic and Swarming techniques to find an optimal ranking of the test cases based on its historical execution. Genetic algorithms are good at taking huge search spaces and navigating them. These algorithms then look for optimal combinations of solutions. In test case prioritization, the search space is made up of the test cases historical execution data and the outcome is based on testing the optimal combinations of cases repetitively until the search space is exhausted and rated against a fitness criterion. Fitness criteria in TCP are the maximum faults covered by combining different sets of test cases.

ABB Robotics Norway used an open data set for an experiment, the results of which portrayed early fault detection for each of the tested cycles. Average Percentage of Faults Detected (APFD)1 metric has been used for validating the improvement in the fault discovery after the prioritization results.  The metrics range from 0 to 1, with values closer to 1 means a good fault detection early in the regression cycle.

Test Case Analysis

The below charts highlight the improvement when the test cases were run using the prioritization order with respect to default ordered the execution. The prioritized ordered test cases have been divided into 5 buckets P1 to P5 by percentile basis. P1 is executed first and the rest follow. It can be seen invalidation executions below, that by P3 bucket execution; roughly 95 percent of the faults are detected while default execution is barely able to reach 65 percent. It’s not only the percentage that tells the story but the reliability and predictability the solution brings in.

[1] https://ieeexplore.ieee.org/abstract/document/6724209/

Optimizing Regression Testing Through Test Prioritization

Optimizing Regression Testing Through Test Prioritization

Source: Altran DevOps

Optimizing Regression Testing Through Test Prioritization

Source: Altran DevOps

Conclusion

Test Case Prioritization is an efficient technique to ensure that important tests are executed well in advance thereby improving the product quality, performance of corrective regression testing and avoiding delays in regression. The graphs clearly depict improvement in fault detection during initial execution itself along with APFD comparison. The benefits far outweigh the effort required for data collection and preparation using traditional methods of prioritization.



Gaurav Pahwa Principal Systems Engineer
Contact us

MEET OUR EXPERTS

For more information about regression testing services or test prioritization services
contact an Altran DevOps expert today.

GET IN TOUCH