Results & benchmark data
AGEST ZERO TEST detects regressions — defects introduced by a change — automatically, from nothing but the URL of the target system. Here we publish the quantitative evidence of the productivity and quality gains it delivered in real verification work.
We present objective results measured across 24 industry test environments and 14 real, publicly available sites: elapsed time against designing and running tests by hand, a benchmark of defect-detection performance against people and against general-purpose AI, and a comparison of test-case generation volume against a large LLM (Claude).
* All data shown here was measured against internal verification environments or publicly available real sites. Actual results vary with your own system architecture and test requirements.
Where it differs from doing it by hand
On a sales management system seeded with known defects, the tool designed 45 test cases and ran them against two versions — before and after the change. Here it is side by side with a person creating and running the same 45 cases.
Automated test design 11.5 min + observing the pre-change screens 7.6 min + running the post-change version and comparing 19.8 min
* Everything runs automatically, so nobody has to attend the run (unattended execution).
Test design (writing up 45 cases) approx. 11 h + test execution (two runs, before and after) approx. 4 h
Scale: 0 — 15 hours
The 38.9 minutes breaks down as 11.5 minutes of test design, 7.6 minutes observing the pre-change version, and 19.8 minutes running the post-change version and comparing. All of it automatic, and unattended while it runs.
The roughly 15 hours by hand is about 11 hours of design and about 4 hours of execution, assuming 45 cases written up one at a time and run twice, before and after.
With the tool, all a person does is enter the URL and the test account, start the run, and check the differences at the end. And runs can go in parallel: one night we ran 17 systems unattended in a single night.
* The tool's 38.9 minutes is an actual measurement taken on 18 August 2026 on our own verification bench (target: a sales management system).
* The roughly 15 hours for a person is an estimate, not a measurement. It applies standard design and execution effort to the breakdown of the 45 cases the tool generated (20 screen-transition-only, 21 with form input, 4 screens with many input fields). Basis of calculation: "Estimation basis for test design and execution effort, 2026-08-20".
* Because the comparison contains an estimate, we make no categorical claims in the form of a simple multiple (for example "cuts working time to one Nth" or "N times faster"). We present the actual figures and a visual comparison only.
People, AI and the tool, on the same problem
We also wanted to check that it isn't just fast. The same sales management system was verified independently by a senior engineer, by generative AI and by this tool — none of them told the answers in advance. The defects fall into 8 classes.
4 of 8 classes found
Explored the screens for 45 minutes without creating test cases.
5 of 8 classes found
One class more than the others: a date format discrepancy.
4 of 8 classes found
The 4 it found, and the 4 it missed, were exactly the same as the senior engineer's.
The 4 classes it found, and the 4 it missed, were exactly the same as the senior engineer's. We take that to mean it is not looking at the system differently from a person — it is automating the way a person looks at it.
* The 45 minutes for the person here is time spent exploring the screens looking for defects, without creating test cases. The roughly 15 hours in the previous section is for creating and running 45 test cases — a different measurement.
On the 3 classes none of the three found
"Deletion consistency" and "session" are operations the tool does not perform by default. That is a design choice to avoid damaging an environment identical to production, not a limit on what it can find.
"Input validation" is a genuine weakness. We are continuing to improve it.
What the test looked like (tool side, 18 August 2026)
Compared with AI a second way
So far the comparison has been "give them the same problem and line up the results". We also measure it another way: have the AI design tests without letting it see the screens at all, then line those up against the tests we generate automatically from the URL alone.
Across 5 industries, taking the AI's 107 items as the denominator, 76.6% were covered by the same operations. (Measured internally, 16 August 2026.)
Close on breadth of design — and on our side, execution and comparison are automatic too. What the AI produces is a design document; making it run still takes human hands.
Three numbers from our development verification
We measure in two ways. One: we build business systems for 24 industries ourselves, seed defects in them deliberately, and see whether it finds them. The other: how far it can get into real external sites that we did not build.
Of 46 defects seeded in advance across test environments built for 24 different industries, it correctly detected 45. (Measured 19 August 2026.)
Comparing two healthy versions with no defects, we confirmed at the same time the stability of zero false detections — no incorrect changes or defects reported.
The proportion of elements the automated exploration was able to reach and operate across 14 publicly available external web applications we had no part in developing (CRM, ERP, learning management, issue tracking, enquiry management, e-commerce and others). Combined figure for two nights of measurement, 17–18 August 2026.
* ① The one defect that was not detected stems from an internal data state — "negative overtime hours" — that cannot be entered through screen operations. Because this tool works from UI operations it cannot reach that in principle; that area is covered by the sibling products AGEST ZERO TEST for Static and for Data.
* ② This measurement is an actual figure taken on a static verification site, under conditions where changing banners such as build numbers were excluded from the comparison in advance. It is not a guarantee that false detections will never occur under any conditions.
* ③ 6 of the 14 sites recorded 100% coverage of all detected operable targets. Out of respect for other companies' design rights and for confidentiality, the sites are named by system category rather than by individual product name.
On 6 of the 14 sites it reached every operable target it found. The targets are public sites we did not build — CRM, ERP, learning management, issue tracking, enquiry management, e-commerce and so on. On sites that require login, providing a test account widens the range it can reach.
The per-site breakdown, and the method, conditions and limits of the measurement, are set out in the technical paper.
Test design volume compared with a large LLM (Claude)
We gave a large LLM (Claude Opus 4.8) exactly the same screen data this tool observed, and compared the tests each of them designed.
Input combinations
It achieves high coverage on the mechanical, exhaustive design of complex input patterns — equivalence partitioning, state transition, boundary values.
Understanding context
On perspectives that need an understanding of context and of what words mean — consistency of calculation formulas, screen display formats, complex permission checks — the large LLM showed better design accuracy.
This tool is not designed to compete with an LLM.
The two complement each other, each covering the other's weaknesses.
Three metrics, three different opponents
Test design volume is compared against a large LLM; screen coverage against a person operating a browser; detection power against a senior test engineer. Different opponents, different conditions — three separate measurements.
Screen coverage on a standard business web application was 82–100% of all site screens, on a par with a person operating it by hand. The one screen of difference comes from a setting that, for safety, does not press confirm-type actions by default.
Detection power: it found the seeded defect classes with 0 false detections. Because every screen it reaches is compared on every run, there is no bias in what gets missed.
* The test design volume comparison (1.26×) is a result measured against a large LLM. Screen coverage was compared against "a person operating a browser manually" and defect detection power against "a senior test engineer". This is not a claim that every metric beat the same opponent (Claude).
* On calculation verification the large LLM is superior. This comparison does not show that the tool is better than an LLM on every dimension.
Seen the numbers — now try it on your own system
Download the measurement conditions and the breakdown data, or apply for the free proof of concept.