Why Regression Testing Matters

Regression testing is the practice of re-running functional and non-functional tests to ensure that previously developed and tested software still performs after new code commits are submitted. Inevitably, as more functions are added and more code is integrated into existing codebases, this integration of the old and new can result in new mistakes. When previously tested software does not perform successfully against previously run tests, it is called a “regression”.

While the practice may seem simple and trivial in concept, it is a significant effort that, nonetheless, pays off in dividends in practice. In a 2019 Black Hat Europe speaking session, Google’s OSS-Fuzz program revealed that 40% of their thousands of bugs are actually regressions.

Ensuring previously squashed bugs stay dead is crucial to customer satisfaction. When updates are made to software, the user expectation is to gain more features, more functionality, more convenience. When users are no longer able to fulfill the jobs they were able to before an update, it prompts unflattering questions about this so-called “upgrade”, leading to loss of trust and, in severe cases, breach of contracts in B2B relationships.

If it’s so important, why aren’t we talking about regression testing more? Why aren’t we excited about regression testing more? Truth be told, because it’s boring. Developers and QA are most inspired by new features, whether it’s building them or testing them. They’re creative individuals who want to either solve problems with new features or find problems in new features. It’s uninteresting to test and retest old code again, and again, and again before every. single. release.

Not to mention that with each release, the regression test suite grows larger and larger, leading to increasingly longer regression testing, where more time is spent testing old features rather than testing and influencing the build out of high-anticipated, new features before they hit the door. Then there’s the administrative tasks ensuring the test suites are continuously optimized. Those regression test suites aren’t going to maintain themselves!

‍

How to Make Regression Testing a Snap

Mayhem autonomously builds and optimizes test suites, making regression testing quick and easy.
‍

We’ve made significant performance improvements to the Mayhem Analysis Engine, increasing the analysis speed by at least 3x for regression tests and test case reporting. For example, what used to take 25 minutes for 10,000 test cases now takes just 4 minutes. These improvements help shorten the time to results, enabling users to quickly identify and fix critical defects before moving on to the next iteration. Customers have shared that faster regression testing is key to accelerating their CI/CD pipeline, and we’re committed to continually improving the Mayhem Analysis Engine for even greater speed and performance, especially on more complex targets.

Our mission to our customers is simple: Allow teams -- whether it’s development, security, performance, and/or quality assurance -- to focus on what’s most strategic to them, allowing autonomous testing to take care of the rest.

Interested in learning more about Mayhem? Request a personalized chat with our team of security experts here.