The Uncomfortable Truth About Agentic Quality Engineering

I'm actively using AI agents to help with quality work across the entire software development lifecycle. Not as a theoretical exercise – as my daily workflow.

I use agents to check requirements, create test plans, generate test scenarios, automate test cases, verify tests and code, assess regression risks between releases, and validate deliverables. I'm dogfooding the framework on a dozen different projects, constantly improving the agents as I use them.

Some of these agents are genuinely helpful. They speed up tedious work, catch things I'd miss, and provide insights I wouldn't have thought to look for. Others? They create more work than they save, generate deliverables that need heavy editing, or miss the point entirely.

But here's what nobody talks about: Agentic QE isn't a silver bullet. It's a framework that works brilliantly for some tasks and contexts, and terribly for others.

This article exists to help you figure out which category your work falls into – before you waste time building the wrong thing.

What Makes Quality Engineering "Agentic"?

Before we discuss value, let's clarify what we mean. Agentic Quality Engineering isn't about replacing humans with robots or automating everything. It's about building quality systems that can operate autonomously within defined boundaries while collaborating with humans.

The Agentic QE Framework is built on PACT principles:

• Proactive: Anticipate and prevent issues before they reach production
• Autonomous: Operate independently within safe boundaries
• Collaborative: Work alongside humans and other agents
• Targeted: Focus on what actually matters to the business

These aren't theoretical principles. They're patterns from actively using agents across dozens of projects – from checking requirements to validating code, from test generation to regression risk assessment.

The Value Patterns You Can Actually Expect

Let's talk about what happens when agents actually help. Not hypothetical benefits – real outcomes from using them across the software development lifecycle.

The value isn't about replacing humans or automating everything. It's about augmenting the work that developers, testers, and quality engineers do every day – from solo developers to full teams.

Speed Without Losing Quality

Traditional quality work is tedious. You check requirements manually, write test plans line by line, create test scenarios one at a time, and automate tests, case by case. Each task takes hours or days.

With well-designed agents, a lot of this routine work can be accelerated. A requirements agent can spot gaps and ambiguities. A test plan agent can generate comprehensive coverage strategies. Test scenario agents can explore edge cases you'd miss. Automation agents can generate test code from scenarios.

But – and this is critical – the agents don't replace your judgment. They generate drafts, suggest approaches, and find patterns. You still review, refine, and decide what's actually valuable.

I use agents from the Agentic QE Fleet to handle routine tasks across a dozen projects. Sometimes the output is 80%-90% ready and needs light editing. Sometimes it's 50%-60% and needs heavy rework. Rarely, it's wrong, and I start over.

The value isn't perfection – it's starting from something rather than nothing.

Coverage Across the Entire SDLC

Manual quality work doesn't scale well. One person can only review so many requirements, write so many test plans, create so many test cases, and review so much code.

Agents can help with tasks across the entire software development lifecycle – if you pick the right patterns for each task.

The agent design patterns aren't academic theory. They're patterns that work for specific quality tasks:

• Scouts explore requirements and find gaps before development starts
• Validators check tests and code for correctness and consistency
• Generators create test plans, scenarios, and automation code
• Reviewers analyze deliverables and provide improvement suggestions
• Synthesizers find insights by correlating information across artifacts
• Assessors evaluate regression risks when changes are made

Each pattern has a specific purpose and works best for certain types of tasks. Use a Generator for test plan creation, a Validator for test verification, a Reviewer for code analysis, and an Assessor for risk evaluation.

Mix them wrong, and you get agents that try to do everything and master nothing.

Shift-Left Quality Assistance

Traditional QE is reactive: find bugs after code is written. Better QE shifts left: catch issues during requirements and design.

Agents can help shift quality left across the SDLC:

• Requirements agents spot ambiguities, gaps, and contradictions before design starts
• Test planning agents identify coverage gaps and suggest test strategies early
• Scenario agents generate edge cases and negative paths that are often missed in manual planning
• Risk assessment agents evaluate what could go wrong based on code changes

I use a quality gate agent to check the status of deliverables across projects. A regression agent assesses risks from changes between releases. Review skills validate that outputs actually match what was requested.

But here's the reality: this only works if you validate the agent outputs. They're not oracles. They miss things, make wrong assumptions, and sometimes generate plausible-sounding nonsense.

The value is in having a draft to critique, not a finished product to accept.

Human Judgment, Amplified

The biggest misconception about agentic QE is that it replaces humans. The reality is different – it augments human work by handling the routine parts so you can focus on the judgment calls.

The human-in-the-loop patterns show how this works in practice. Agents generate drafts, you review and refine. Agents suggest approaches, and you decide which fits your context. Agents find patterns, you interpret what they mean.

The framework describes five levels of human involvement:

• Level 0: Fully manual (human does everything)
• Level 1: Agent suggests, human decides
• Level 2: Agent acts with approval
• Level 3: Agent acts, human audits
• Level 4: Agent acts, alerts on edge cases
• Level 5: Fully autonomous (human sets policy only)

Most practical implementations stay at levels 1-2. The agent does the grunt work, you provide the expertise.

This isn't about achieving full autonomy. It's about having agents do what they're good at (pattern matching, draft generation, routine checking) while humans do what they're good at (contextual judgment, strategic thinking, creative problem-solving).

What Actually Goes Wrong (And What to Watch For)

Let me be honest about what happens when agents don't work well. These aren't dramatic production disasters – they're everyday friction points.

When Agents Miss the Context

I use agents to generate test scenarios. Sometimes they're spot-on and find edge cases I'd miss. Other times, they generate plausible-sounding scenarios that miss the business context entirely.

For example, an agent might generate exhaustive test cases for a feature flag used internally by only three people. Technically correct, massive waste of time.

Lesson: Agents can't read your mind about what matters. You need to provide context explicitly, and you still need to review outputs critically.

When "Complete" Doesn't Mean "Correct"

Agents are excellent at appearing complete. They generate comprehensive test plans with all the right sections, detailed test cases with all the expected fields, and thorough code reviews with proper formatting.

But comprehensive ≠ useful. I've seen agents generate test plans that cover everything except the actual risk areas. Test cases that check syntax but miss logic errors. Code reviews that point out style issues while missing algorithmic problems.

Lesson: Agents optimize for appearing complete, not for being correct. This is what I call "completion theater" – it looks done, but it's not actually right.

When Agents Contradict Themselves

I use multiple agents on the same project – one checks requirements, another generates tests, and another reviews code. Sometimes they contradict each other.

The requirements agent says a feature is clearly defined. The test generation agent can't figure out what to test because the requirements are ambiguous. The code review agent approves code that doesn't match either interpretation.

Lesson: Agents don't have a shared understanding. Each one interprets context independently. You're still the integration layer.

When You Can't Explain the Output

The worst failures aren't wrong answers – they're when you can't tell if an answer is correct.

An agent generates a complex test strategy. It looks good. But you can't trace back why it made specific choices. Another engineer asks, "Why this approach?" and you realize you're just trusting the agent without understanding.

Lesson: If you can't explain why the agent suggested something, you shouldn't use that suggestion. Black box outputs are technical debt.

How to Know If Agentic QE Fits Your Context

Not every team should adopt agentic QE. Here's how to evaluate if it fits your context.

Green Flags: You're Probably Ready

You should consider agentic QE if:

• You have repetitive quality processes that follow predictable patterns
• Your QE team is a bottleneck in the delivery pipeline
• You have data from tests, deployments, incidents, and monitoring
• Quality issues are discovered late in the process or in production
• Your system is complex enough that humans can't track all quality signals
• You're willing to invest in building proper foundations (telemetry, observability, data pipelines)
• Your team embraces experimentation and learning from failures

Red Flags: You're Probably Not Ready

You should wait on agentic QE if:

• Your quality processes are still forming and change frequently
• You don't have basic test automation in place yet
• Your data is scattered, inconsistent, or missing
• You need predictable, deterministic behavior with zero surprises
• Your team has no experience with AI/ML systems
• You can't tolerate failures during the learning phase
• You want a turnkey solution that works immediately without tuning

The Context-Driven Question

The most important question isn't "Should we use agentic QE?" It's "What problem are we actually trying to solve?"

If your answer is "we want to look modern" or "everyone's doing AI now" – don't start. You'll waste time and money.

If your answer is "we spend 40% of our time on manual deployment validation" or "our flaky tests are killing team productivity" – you might have a case.

The framework is context-driven by design. There are no universal best practices, only practices that fit specific contexts.

Where to Start: Assessment Before Action

If you've read this far and still think agentic QE might fit your context, start with assessment.

The Agentic QE Assessment evaluates your current state across four PACT dimensions:

Proactive Capability

• Do you have failure prediction mechanisms?
• Can you identify risks before they materialize?
• Are you doing trend analysis and early warning?

Autonomous Capability

• What's your current automation coverage?
• Do you have self-healing systems?
• How much requires human intervention?

Collaborative Capability

• How well do your tools integrate?
• Do you have effective knowledge sharing?
• Are your feedback loops fast?

Targeted Capability

• Is your quality work aligned with business priorities?
• Do you have effective prioritization methods?
• Can you measure the quality's business value?

The assessment gives you a maturity score (0-100) for each dimension and identifies specific gaps. This tells you where to focus – and whether agentic QE makes sense at all.

Your First 30 Days: The Getting Started Guide

If the assessment shows you're ready, the Getting Started Guide provides a practical 30-day roadmap:

Week 1-2: Foundation

• Audit your current quality data sources
• Identify one specific, high-pain problem
• Build baseline metrics
• Get team buy-in

Week 3-4: First Agent

• Start with a Scout or Validator pattern (low risk, high learning)
• Deploy in shadow mode (agent observes but doesn't act)
• Collect decision agreement rates
• Learn from disagreements

Month 2-3: Orchestration

• Add complementary agents
• Implement human-in-the-loop workflows
• Graduate from shadow mode to supervised execution
• Build trust through demonstrated competence

Month 4+: Scale

• Increase autonomy based on performance
• Add more sophisticated patterns
• Implement continuous learning loops
• Share lessons across the team

The Templates and Tools You'll Actually Use

Theory is cheap. Implementation is expensive. That's why the framework includes ready-to-use templates:

Agent Decision Log (YAML) – Track every agent decision for audit and learning. Includes context, analysis, alternatives considered, final decision, execution results, and human interaction.

PACT Assessment Scorecard (Markdown) – Comprehensive assessment template for measuring PACT maturity over time. Shows evidence, gaps, and next steps for each dimension.

Agent Implementation Checklist (Markdown) – Ensures you don't skip critical steps: clear purpose, defined boundaries, explainable actions, graceful failure, learning capability, human override, audit trail, and success metrics.

Code Templates (Python) – Basic agent skeleton and orchestrator templates that handle the tedious infrastructure work so you can focus on the quality logic.

These aren't theoretical templates – they're working artifacts from active projects.

The Real Value: Learning What Works in YOUR Context

Here's the truth about agentic QE: the framework's value isn't in the agents themselves. It's in the systematic approach to building, testing, and evolving autonomous quality systems.

You'll learn:

• Which quality tasks are ripe for automation, vs which need human judgment
• How to build trust gradually rather than demanding it immediately
• What good orchestration looks like for your specific architecture
• Where humans add the most value (spoiler: it's not the repetitive stuff)
• How to measure and prove the quality's business impact

The framework provides patterns, templates, and guardrails. But the real work – understanding your context, adapting patterns, measuring outcomes – that's on you.

What Success Looks Like (And What It Doesn't)

Success with agentic QE doesn't look like:

• ❌ Firing your QE team because agents do everything
• ❌ Zero human involvement in quality decisions
• ❌ Perfect accuracy from day one
• ❌ A universal solution that works everywhere

Success looks like:

• ✅ QE engineers spending time on interesting problems, not repetitive tasks
• ✅ Faster feedback loops without sacrificing quality
• ✅ Catching issues earlier in the development process
• ✅ Quality insights you couldn't see before (from pattern synthesis)
• ✅ Gradual improvement over time as agents learn
• ✅ Humans doing what humans do best, agents doing what agents do best

The Hard Questions You Should Ask

Before you start, ask yourself:

1. What specific problem are we solving? (Not "we want AI in QE")
2. What does success look like, in measurable terms? (Not "better quality")
3. What can we afford to get wrong while learning? (Be honest)
4. Who owns the agents? (Build vs buy, maintenance, evolution)
5. How will we know if it's working? (Define metrics up front)
6. What's our rollback plan? (When, not if, things go wrong)
7. Are we prepared for the cultural shift? (Trust, transparency, new ways of working)

If you can't answer these clearly, you're not ready. And that's okay – it's better to wait than to rush into expensive failure.

Why I Built This Framework (And Why I'm Sharing It)

I spent eight years leading quality engineering at Alchemy. Before that, I have over 29 years in various IT roles. I've lived through the evolution from manual testing to test automation to CI/CD to the current state of AI-assisted quality work.

I built this framework because I started using agents myself and realized there were patterns to what worked and what didn't:

• Some agent types consistently help with specific tasks
• Human involvement needs to match the agent's capability and the task's risk
• Orchestrating multiple agents requires intentional design
• "Complete-looking" outputs often hide incomplete thinking
• Context is everything – what works for one project fails for another

The framework captures what I've learned from actively using agents across my own projects and helping others do the same. Not vendor hype, not conference-talk promises – practical patterns from actually doing the work.

I'm sharing it because quality engineering is at an inflection point. AI agents will either amplify our effectiveness or become another overhyped technology that disappoints.

The difference is how we use them.

Your Next Step: Take the Assessment

If you've read this far, you're serious about understanding if agentic QE fits your context.

Here's what I recommend:

1. Take the assessment (15 minutes)

• Get your current PACT maturity scores
• Identify specific gaps and opportunities
• See where you rank across the four dimensions

2. Review the results with your team

• Do the gaps resonate with your experience?
• Which dimension needs the most work?
• Is there alignment on priorities?

3. If you're ready, start with the Getting Started Guide

• Pick one specific problem to solve
• Choose the right agent pattern for that problem
• Build in shadow mode first
• Measure everything

4. If you're not ready, that's valuable information too

• Focus on building the foundations first
• Get basic observability and data pipelines in place
• Mature your test automation
• Reassess in 3-6 months

The Framework Is Open, The Journey Is Yours

The Agentic QE Framework is open and community-driven. It's not a product you buy – it's a methodology you adapt.

You'll find:

• Agent design patterns that work (and anti-patterns to avoid)
• Orchestration strategies for coordinating multiple agents
• Human-in-the-loop workflows for graduated autonomy
• Templates and tools to accelerate implementation
• Assessment tools to measure progress

But the framework can't tell you what to build. Only you know your context, your constraints, your quality challenges, and your team's capabilities.

What the framework can do is give you a fighting chance at succeeding where most fail.

A Final Word: Context Trumps Everything

There are no universal best practices in quality engineering. Only practices that fit specific contexts.

Agentic QE works brilliantly for some teams and terribly for others. The difference isn't the technology – it's the context, the problem, the team, the culture, and the willingness to learn from failures.

The framework exists to help you discover if it works for you – not to convince you it should.

Take the assessment. Be honest about the results. If you're ready, start small. If you're not, build the foundations first.

Either way, you'll be making an informed decision based on your context, not someone else's hype.

---

About the Author: Dragan Spiridonov (Profa) is the Founder of Quantum Quality Engineering and an Agentic Quality Engineer with 29+ years in IT and 12+ years specializing in quality engineering. He previously served as VP of Quality Engineering at Alchemy for 8 years before starting his consultancy. He's establishing the Serbian Agentic Foundation Chapter and is a member of the Global Agentics Foundation. He practices context-driven testing, TDD (both London and Chicago schools), XP methodologies, RST, and the Holistic Testing Model evolved with PACT principles.

Visit agentic-qe.dev to get started.

Ready to evaluate if Agentic QE fits your context? Take the assessment and start your journey.