Why You Need Test and Evaluation (T&E)

David Brumley
March 18, 2021
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

In part three of the series, I will discuss the role of test and evaluation in your organization.

Download the complete white paper Build a Test and Evaluation Plan with Advanced Fuzz Testing here.


Part three of a three part series. Part one discussed open source software. Part two discussed the consequences of not analyzing open source software.

The Saga Continues 

So far, we’ve discussed the challenges of the software supply chain at the developer-level. But, that’s only the tip of the iceberg. There still remains the challenges of software supply chain management at the vendor-level. 

An organization relies on a single application to operate their business. There is a complex network of hardware and software at the heart of every company, flowing necessarily data to the appropriate departments and powering their organization’s productivity. Like developers, consumers of software -- typically organizations -- rely on a vast software supply chain for their IT ecosystem. 

These organizations face the same challenge as the builders of software: It only takes a single vulnerability in a software component to put the entire business and IT ecosystem at risk. In July 2018, upstart bank Monzo noticed an uptick in fraudulent claims, especially among their customers who had recently made purchases on Ticketmaster. Monzo was onto something.

Test and Evaluation

Ticketmaster was in fact a cause, but not the root issue. Ticketmaster had subcontracted Inbenta Technologies for their chatbot technology, allowing for a personalized purchasing experience for Ticketmaster visitors. Months prior, threat group Magecart had uncovered a vulnerability within Inbenta that allowed the hackers to modify code and implement a digital card skimmer, affecting 40,000 of Ticketmaster’s European customers. Though the faulty software was developed by Inbenta Technologies, Ticketmaster faces a 6.5 million dollar lawsuit today. Had Ticketmaster conducted acceptance testing prior to implementing software into their IT ecosystem, they may have been able to avoid the issue or proven they had conducted due diligence prior to acquisition. Software composition analysis scanners are powerful and can offer many benefits to a security-conscious organization. However, they come with a key drawback: it takes a backwards-looking approach, identifying issues that has already been reported. As a result, organizations are forced to assume a reactive stance, always playing the catch-up game. Luckily, there is a way to get ahead of the curve: proactively uncover zero-day vulnerabilities. In the following sections, we’ll share a proven and accepted framework, Test & Evaluation, and testing technique, advanced fuzz testing


Test and Evaluation

Test and Evaluation (T&E) is a fundamental process within the government for managing risk and assuring quality of critical systems. While this term is more common in the federal sector, the concepts are universal and relevant across any industry that relies on software for operation. This evaluation process assures quality, security, and resiliency of products before acquisition or operation. There are two types of T&E performed: 


  • Developmental Test and Evaluation (DT&E). DT&E is testing performed in parallel with product development and designed to analyze the maturation of product development through verification of technical progress and implementation of risk management controls, as well as to certify readiness for initial operational testing. DT&E may sound familiar to readers in the enterprise sector. DT&E has several similarities with the shift left movement, which calls for early and often security testing throughout the development lifecycle to control remediation costs, ensure security, and prevent time-to-market delays. 


  • Operational Test and Evaluation (OT&E). OT&E is a fielded test, under realistic combat conditions, of any item or component of a weapons system, equipment, or munitions to determine operational effectiveness and suitability for combat. Concepts behind OT&E could be found in security savvy industries, such as telecommunications and networking, but implementation of such practices are few and far in between. Security-first enterprise organizations conduct acceptance testing of software or systems before implementing them to their labs or IT ecosystems.


T&E involves evaluating a product from the component level, to integration and even full system of systems. Testing is conducted across multiple phases of the acquisition lifecycle and across various teams. See diagrams below for further details:

Test and Evaluation is conducted across multiple phases of the acquisition lifecycle

Download the complete white paper Build a Test and Evaluation Plan with Advanced Fuzz Testing here.

Share this post

Fancy some inbox Mayhem?

Subscribe to our monthly newsletter for expert insights and news on DevSecOps topics, plus Mayhem tips and tutorials.

By subscribing, you're agreeing to our website terms and privacy policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Add Mayhem to Your DevSecOps for Free.

Get a full-featured 30 day free trial.

Complete API Security in 5 Minutes

Get started with Mayhem today for fast, comprehensive, API security. 

Get Mayhem

Maximize Code Coverage in Minutes

Mayhem is an award-winning AI that autonomously finds new exploitable bugs and improves your test suites.

Get Mayhem