Artificial Intelligence is accelerating software test creation, allowing teams to generate new scenarios faster. But as the ability to produce tests increases, a new question emerges: how can we ensure they actually identify the issues they were designed to catch?
This is the starting point for Noesis' new article, published in AutomationSTAR, which explores the challenges of validating AI-generated tests.
1. AI doesn't know the application it is testing
A passing test doesn't necessarily mean it's effective. AI models can generate scenarios based on patterns and examples, but they may lack the context needed to understand an application's or business's specific rules.
This creates a risk of false confidence: a test suite may return positive results without detecting certain defects. As AI makes it possible to generate more tests, ensuring their effectiveness becomes as important as expanding test coverage.
2. Does a passing test mean an effective test?
Validating AI-generated tests is therefore essential. One way to assess their effectiveness is to deliberately change the code and check whether the test detects it. Known as mutation testing, this approach helps determine whether a test can actually fail when it should. Validation can also be supported by a second AI model that checks whether the assertions defined in the test align with the requirements and identifies tests that are too similar to one another. This helps prevent the test suite from growing without adding meaningful coverage.
This does not mean human review is unnecessary. In scenarios with greater implications for the business and its customers, such as payments, pricing, or authentication, closer scrutiny of tests remains important. Combining automated validation with human review makes it possible to assess test quality without applying the same level of effort to every scenario.
3. testingON explores Agentic AI for test validation
While many AI test-generation tools focus on creation, testingON's evolution aims to address an equally important step: ensuring tests can identify real defects. This is where an Agentic AI initiative comes in: an agent designed to validate generated tests and assess their effectiveness beyond simply confirming they run successfully.
This approach is part of a broader vision for testingON, in which different agents support complementary stages of the testing process: generating tests from new user stories, validating their ability to detect defects, and running tests and analyzing the results, including performance. The goal is to connect these stages in an integrated process, rather than treating them as separate, standalone tools.
These capabilities are part of the testingON roadmap and are not yet available as product features. They reflect a focus on test automation quality, at a time when generating tests faster is not enough: it is also essential to ensure they can identify the issues that really matter.
