Qodo’s fully autonomous agent tackles the complexities of regression testing

Code is continuously evolving in the software development process, requiring ongoing testing for quality and maintainability. This is the root of regression testing, in which existing tests are re-run to ensure that modified code continues to function as intended.

However, regression testing can be time-consuming and complex, and may often be neglected in lieu of other priorities. 

Qodo (formerly CodiumAI) says it can ease headaches around the process with the release today of its new fully autonomous AI regression testing agent, Qodo Cover. Its agent creates validation suites to ensure that software applications are, essentially, behaving. The 2-and-a-half-year-old startup announced its new tool at AWS re:Invent, where it also pitched as a finalist in an AWS Unicorn Tank competition.

“We’re fast approaching a point where the vast majority of code will be AI-generated, fundamentally changing how software is built,” said Qodo CEO Itamar Friedman. “It’s critical that we keep up by leveraging AI not just for code generation, but for maintaining and improving code quality.”

Only retaining code that meets specific criteria

Qodo explained earlier this year at VentureBeat Transform that it is approaching AI agents in an incremental fashion — taking on competitors such as Devin that offer more end-to-end suites. The Israeli startup offers numerous small agents that handle specific tasks within software development workflows. 

Qodo Cover is the newest of these. The fully autonomous agent analyzes source code and performs regression tests to validate it as it changes throughout its lifecycle. The platform ensures that each test runs successfully, passes and increases the amount of code it covers — and only keeps those that meet all three criteria. 

Demonstrating its ability to generate production-quality tests, a pull request generated fully autonomously by Qodo Cover was recently accepted into Hugging Face’s PyTorch Image Models repository. Pull requests are a means of quality control in software development, allowing collaborators to propose and review changes before they are integrated into a codebase. This can keep bad code and bugs out of the main codebase to ensure quality and consistency.

The acceptance by Hugging Face validates Qodo’s offering and exposes it to more than 40,000 projects in the popular machine learning (ML) repository. 

“Qodo Cover represents a significant step toward autonomous software development by ensuring every piece of code, whether human or AI-written, is properly tested and maintainable,” said Friedman. 

Qodo Cover is built on an open-source project that Qodo launched in May. That project was based on TestGen-LLM, a tool developed by Meta researchers to fully automate test coverage. To overcome challenges with large language model (LLM)-generated tests, the researchers set out to answer specific questions: 

• Does the test compile and run properly?
• Does the test increase code coverage?

Once those questions are validated, it’s important to perform a manual investigation, Friedman writes in a blog post. This involves asking: 

• How well is the test written?
• How much value does it actually add?
• Does it meet any additional requirements?

Users provide several inputs to Qodo Cover, including: 

• The source file for code to be tested
• Existing test suite
• Coverage report
• Command for building and running suites
• Code coverage targets and maximum number of iterations to run
• Additional context and prompting options

Qodo Cover then generates more tests in the same style, validates them using the runtime environment (i.e., do they build and pass?), reviews metrics such as increased code coverage and updates existing test suites and coverage reports. This is repeated until code either reaches the coverage threshold or the maximum number of iterations. 

Giving devs full control, provides progress reports

Qodo’s agent can be deployed as a comprehensive tool that analyzes full repositories to identify gaps and irregularities and extend test suites. Or, it can be established as a GitHub action that creates pull requests automatically to suggest tests for newly-changed code. Qodo emphasizes that developers maintain full control and have the ability to review and selectively accept tests. Each pull request also includes detailed coverage progress reports. 

Qodo Cover supports all popular AI models, including GPT-4o and Claude 3.5 Sonnet. The company says it delivers high-quality results across more than a dozen programming languages including JavaScript, TypeScript, C++, C#, Ruby, Go and Rust. It is intended to integrate with Qodo Merge, which reviews and handles pull requests, and coding tool Qodo Gen.