AI Didn’t Kill Manual QA — It Killed the Boring Half of It

By Pavel

Around 2023, when large language models first started writing plausible code, a prediction hardened into conventional wisdom almost overnight: software testers would be among the first casualties. The reasoning looked clean. If AI can write code, it can write test code. If it can write test code, it can run it, read the results, and file the bug report. The manual tester — the person clicking through the same checkout flow for the four-hundredth time — was described, in conference talks and LinkedIn think-pieces alike, as a role with an expiration date.

Three years later, the data is in, and it tells a stranger story. Eighty-nine percent of organizations are now piloting or deploying generative AI somewhere in their quality engineering practice, according to the World Quality Report 2025-26 — the seventeenth edition of the industry’s longest-running QA survey, published by Capgemini, Sogeti and OpenText. And yet remote QA and SDET job listings have grown more than 40 percent since 2023. Postings that require AI literacy pay 15 to 25 percent more than those that don’t, according to industry job-market analyses. The people who were supposed to be automated out of existence are, on average, busier and better paid than before the automation arrived.

Something did die, though. It just wasn’t the profession. It was a specific slice of the work — the repetitive, mechanical, judgment-free slice — and understanding exactly where that line fell explains more about the future of software quality than any headline about AI replacing testers ever could.

The Adoption Mirage: 89% Trying, 15% Succeeding

Start with the number that sounds like total victory for the automation thesis: nearly nine in ten organizations pursuing generative AI in quality engineering. On its own, that statistic seems to settle the argument. Look one layer deeper and it falls apart.

Of that 89 percent, only 37 percent have anything in actual production. A further 52 percent are stuck in pilot phases — proofs of concept, sandboxed experiments, tools evaluated but never rolled out. And the number that should give every “AI-first QA” vendor pause: just 15 percent of organizations have deployed AI in quality engineering at enterprise scale. The distance between “we’re using AI in testing” and “AI is actually load-bearing in how we test” turns out to be enormous, and most of the industry is still standing on the wrong side of it.

More telling still: the share of organizations that aren’t adopting AI in QA at all *rose* from 4 percent in 2024 to 11 percent in 2025. That’s not laggards failing to start. That’s early experimenters quietly walking their pilots back — companies that tried, measured, and concluded the technology wasn’t ready for their context.

The World Quality Report’s respondents were specific about why. Sixty-seven percent cite data privacy risks — feeding proprietary application data into AI tools remains a compliance minefield. Sixty-four percent cite integration complexity: an AI test generator that doesn’t talk to your CI pipeline, your test management system, and your issue tracker is a demo, not a tool. Sixty percent cite hallucination and reliability concerns, which in a testing context is uniquely corrosive — a test suite you can’t trust is arguably worse than no test suite, because it manufactures false confidence. And half of organizations report they simply lack the AI/ML expertise to run these systems well, a figure unchanged from the year before.

None of this means the technology doesn’t work. It means the gap between a compelling demo and a dependable practice is wide, and crossing it costs more than most adoption headlines admit.

Why Pilots Die

The 52 percent stuck in pilot purgatory deserve a closer look, because the way these initiatives stall follows a pattern specific to testing — different from why AI pilots fail in, say, marketing or customer support.

A typical trajectory: a team adopts an AI test-generation tool and points it at their web application. Within days it produces hundreds of tests — a volume that would have taken the team months by hand. The demo to leadership goes beautifully. Then the tests start running against real release candidates, and three problems surface in sequence.

First, volume turns out to be the wrong metric. Five hundred generated tests that all exercise variations of the login form aren’t five hundred units of protection; they’re one unit of protection with four hundred ninety-nine echoes. Coverage *breadth* — the thing that catches real regressions — requires understanding which paths through the application carry business risk, and the model generating tests from the UI has no idea that the billing reconciliation flow matters a thousand times more than the profile-picture uploader. A human has to encode that risk map, and in most pilots, nobody budgeted time for it.

Second, the failure triage load doesn’t disappear — it moves. When a generated test fails, someone must decide: is this a product bug, a flaky test, an environment problem, or the model hallucinating an assertion that was never a real requirement? That last category is new, and it’s poisonous. A human-written test encodes a human’s understanding of intended behavior; when it fails, the failure means something. A generated test encodes a model’s guess about intended behavior, and a percentage of those guesses are simply wrong — the 60 percent of organizations citing hallucination concerns in the World Quality Report aren’t being paranoid, they’re describing Tuesdays. Teams that didn’t staff for this triage work found their engineers spending the time they’d saved on authoring tests adjudicating them instead.

Third, integration debt compounds quietly. The pilot ran in a sandbox. Production QA runs in a CI pipeline wired to test management, issue tracking, deployment gates, and reporting dashboards — the 64 percent citing integration complexity are describing the moment a tool that worked in isolation has to join that circulatory system. Every seam is glue code someone owns forever.

None of these failure modes is fatal. The 15 percent that scaled solved all three — with risk-based coverage models, explicit triage ownership, and real integration engineering. But all three require exactly the resource the pilot was supposed to conserve: skilled human judgment, applied continuously. That’s the quiet irony of the operationalization gap. Getting AI to *reduce* your dependence on QA expertise turns out to require quite a lot of QA expertise.

What AI Actually Took

Inside the organizations where AI-augmented testing genuinely operates, a consistent pattern has emerged about which work disappeared first. It wasn’t the most skilled work. It was the most repetitive.

Regression testing went first — the ritual of re-running hundreds of known scenarios after every release to confirm nothing old broke. This was always the least loved part of the job: essential, mind-numbing, and mechanical by definition, since the whole point is checking that behavior *hasn’t* changed. AI-driven tools now generate these checks, run them continuously, and — through so-called self-healing — repair their own test scripts when a UI element moves or a selector changes, instead of failing with a false alarm that a human then spends an afternoon triaging. Script maintenance, historically the silent killer of test automation initiatives, is exactly the kind of pattern-matching chore machine learning handles well.

Boilerplate test authoring followed. Boundary value checks, form validation, basic CRUD coverage — the tests that any competent engineer writes the same way every time — are now routinely generated from user stories or tickets. The World Quality Report found that over 60 percent of organizations using AI in QA apply it to autonomous test script creation and test data generation. Synthetic test data specifically jumped from 14 percent usage in 2024 to 25 percent in 2025, the single fastest-growing GenAI use case in the field — unsurprising, since generating realistic-but-fake data is a task with no judgment component at all, just tedium.

Notice the shape of this list. Everything AI absorbed is work where the correct output was already fully specified and the human was acting, essentially, as a slow and expensive execution engine. That’s the boring half. It’s gone, or going, and almost nobody who did that work mourns it.

What It Couldn’t Take

The other half of the job was never about executing known checks. It was about finding the problems nobody thought to check for.

Exploratory testing — the practice of probing an application the way a curious, slightly malicious, easily confused real user would — remains stubbornly human. Not because AI can’t click around an interface (it can, tirelessly), but because the value of exploration lives in judgment: noticing that a flow is *technically correct but confusing*, that an error message is accurate but unhelpful, that a feature works exactly as specified and the specification is wrong. A model can verify that the checkout button functions. It takes a human to notice that customers keep missing it.

Business logic validation sits in the same category. Knowing that a discount rule, a tax calculation, or an insurance eligibility check is wrong requires knowing what *right* means in a specific business context — knowledge that lives in domain experts’ heads and half-outdated wikis, not in anything a model was trained on. UX evaluation, accessibility judgment, and risk-based test prioritization — deciding what *not* to test when time is short, which is most of the time — all resist automation for the same reason: they’re not verification tasks, they’re judgment tasks.

And then there’s the newest entry on the list, with a certain poetic quality to it: testing AI systems themselves. Applications with LLM components fail in ways deterministic software never did — non-reproducibly, plausibly, differently every run. Validating them requires exactly the skeptical, adversarial, “what if I phrase this weirdly” instinct that exploratory testers have been cultivating for decades. The technology that was supposed to replace testers has instead become one of the hardest things they’re asked to test.

The World Quality Report captured this duality in its skills data: generative AI is now the top-ranked skill organizations want in quality engineers, cited by 63 percent — but core quality engineering skills sit right behind it at 60 percent. The market isn’t choosing between the old skill set and the new one. It’s demanding both in the same person, which goes a long way toward explaining that 15-to-25-percent pay premium.

The Agentic Wave: From Writing Tests to Supervising Them

The current frontier goes by the name agentic testing, and it’s worth being precise about what makes it different, because the term is already being stretched over everything from autocomplete to full autonomy.

AI-assisted testing — the previous wave — meant a human decided what to test and the AI helped do it faster. Agentic testing inverts the loop: the system plans what to test, generates the tests, executes them, interprets the results, and heals what breaks, with no human in the loop for each step. Platforms like Mabl and Functionize have been racing toward this model, adding test generation straight from user stories and Jira tickets. Investors have noticed — devtools companies raised over $1 billion in a year, with the biggest checks flowing specifically to AI agents, QA infrastructure, and DevOps automation.

For the engineer, this shifts the job description again. Supervising a fleet of testing agents is real work, but it’s different work: defining coverage goals instead of writing scripts, auditing what the agents chose to test and — more importantly — what they silently chose not to, and treating the agents’ output with the same skepticism previously reserved for a junior engineer’s first pull request. The skill that appreciates in value is exactly the one the boring half of the job never exercised: knowing what quality means for this product, these users, this risk profile, and being able to tell when a green dashboard is lying.

That last part is not hypothetical. The most instructive software failure of 2026 so far happened at a company with one of the most sophisticated engineering cultures on the planet, and it slipped straight through a green dashboard.

GitHub, April 23: A Case Study in the Gap

On April 23, 2026, GitHub shipped a change to how merge base computation worked for merge queue operations. The new code path was supposed to be gated behind a feature flag for an unreleased feature. The gating was incomplete. For four hours and 38 minutes, squash-merge groups in merge queues received an incorrect three-way merge — meaning GitHub, the system of record for much of the world’s source code, was quietly producing corrupted merge commits. By the time it was over, 230 repositories and 2,092 pull requests were affected.

Two details make this incident a perfect specimen for the question this article is asking.

First: the regression sailed through internal validation because, in GitHub’s own words, existing test coverage primarily exercised single-PR merge queue groups — which didn’t trigger the faulty calculation. The tests ran. The tests passed. The tests were testing the wrong thing. No amount of execution speed fixes a coverage judgment error; running the wrong checks faster just produces false confidence sooner.

Second, and more damning: the issue wasn’t caught by monitoring at all. Because it corrupted correctness rather than availability — nothing went *down*, the output was just silently wrong — automated systems saw nothing amiss. GitHub became aware of the problem three and a half hours in, through customer support inquiries. Which is to say: the detection layer that finally worked was humans, outside the company, noticing that something felt wrong with their merges and saying so.

It’s hard to design a cleaner natural experiment. A world-class engineering organization, heavily automated, extensively tested — and the failure mode was precisely the one automation is structurally blind to: a confident, plausible, wrong answer. That’s not an argument against automated testing, which catches thousands of failures for every one it misses. It’s an argument about what the remaining human half of the job actually is. Someone has to ask “but what happens with *multiple* PRs in the queue?” before production does.

The Bug Factory Feedback Loop

Zoom out from any single incident and a broader, stranger dynamic comes into focus: AI is simultaneously the biggest new source of software defects and the top-selling remedy for them.

Industry analyses of 2026 bug patterns report that functional regressions now ship more often than any other defect category — driven, specifically, by AI-accelerated teams pushing code to production multiple times a day, faster than test coverage grows to match. The bottleneck in software delivery used to be writing the code; generative tools have blown that constraint open, and the pressure has moved downstream to verification. More code, written faster, reviewed more shallowly, lands on test suites designed for a slower world.

The economics of getting this wrong were staggering before AI accelerated anything. The Consortium for Information & Software Quality estimated the cost of poor software quality in the United States at $2.41 trillion for 2022 alone — with accumulated technical debt accounting for roughly $1.52 trillion of the total. That figure predates the generative-AI coding boom entirely. Every trend since points in the direction of more code, produced faster, by fewer people per line — which makes quality the discipline absorbing all of the resulting pressure.

This is the context that makes the “AI replaces testers” prediction look not just wrong but backwards. The prediction assumed the amount of verification work was fixed, so automating some of it would shrink the human share. Instead, AI expanded the total volume of software — and therefore of verification — faster than it automated any of it. The testers didn’t lose the race to the machine. The machine moved the finish line.

What the Job Looks Like Now

Put the pieces together and the transformed role comes into focus — visible already in how job listings have shifted, with “Manual Tester” postings declining while “Quality Engineer,” “SDET,” and AI-assisted QA roles grow to replace them.

The engineer in this new shape of the role spends little time executing scripted checks — agents and pipelines do that — and much more time on the judgment layer above them: deciding what the agents should cover, auditing what they missed, exploring the product the way an unpredictable human would, validating the business logic no model has context for, and testing the AI features that fail in ways deterministic software never did. The consensus that has settled over the industry compresses into one sentence: AI won’t replace QA engineers, but it will replace QA engineers who refuse to work with it.

The World Quality Report’s authors call the destination “collaborative intelligence” — human expertise and machine capability composing a quality practice neither could sustain alone. Strip away the consultancy phrasing and the underlying observation is sound, because it matches what the failure data shows: the machine half executes at a scale and speed no human team can match, and the human half catches the category of failure the machine half is structurally incapable of seeing — the test that passes while the product is wrong.

A Field Guide: Sorting Your Own Two Halves

For engineering leaders trying to apply any of this, the useful exercise isn’t asking “should we adopt AI testing” — at 89 percent industry adoption, that question has answered itself — but sorting your team’s actual workload into the two halves this whole story turns on. A few filters do most of the work.

Ask, for each recurring QA activity: *if this check fails, does anyone need to think?* Regression runs, smoke tests, form validation, cross-browser sweeps — a failure there routes mechanically to whoever broke the build. That’s the automatable half, and if humans are still executing it by hand in 2026, that’s no longer diligence; it’s waste dressed as thoroughness. The World Quality Report’s fastest-moving adoption numbers — autonomous script creation above 60 percent, synthetic data nearly doubling year over year — are concentrated precisely here, because this is where AI’s error tolerance is forgiving: a badly generated regression test wastes minutes, not trust.

Then ask the inverse: *which failures would only a person even notice?* The confusing-but-functional checkout flow. The discount rule that’s computationally correct and commercially insane. The AI feature that answers fluently and wrongly. The merge that succeeds and corrupts. This half doesn’t shrink with better tooling — GitHub’s April incident is what it looks like when world-class automation meets a question nobody assigned a human to ask. Staff this half deliberately: exploratory testing time protected on the calendar rather than squeezed into release gaps, domain experts pulled into test design, and at least one person whose explicit job is auditing what the automated layer *isn’t* checking.

And one habit borrowed from the organizations that made it to scale: treat every green dashboard as a claim, not a fact. The most expensive failures of the AI-testing era — silent corruption, plausible hallucination, coverage gaps behind passing suites — share a single signature: everything looked fine. A standing ritual as simple as a monthly hour spent trying to falsify your own test results — pick a passing suite, ask what it would fail to catch, try to make that happen — is the cheapest insurance the current era offers.

The 15 percent of organizations that scaled AI-augmented testing didn’t get there by buying a tool; they got there by having the test architecture, the data discipline, and the people capable of supervising automation critically. Choosing an automation stack is itself a judgment-heavy engineering decision — one that determines whether the next incomplete feature flag gets caught by a test someone thought to write, or by 2,092 broken pull requests.

The boring half of QA is dead, and deserves no mourning. The interesting half — skepticism, curiosity, the instinct that a green dashboard might be lying — just became the most valuable skill in the room.