Directing an AI System That Tests, Documents, and Learns on Its Own
This is the system I designed, code, and lead at Paysafe. Instead of writing and running every test case by hand, I built an AI-driven workflow that reads requirements straight from Jira and Confluence, keeps a live knowledge base always up to date, designs functional and automated test cases from it, runs them across web, API, and performance layers, and documents the evidence — all connected in real time via MCP to the tools the team already uses. My work today is less about executing tests one by one, and more about directing this system: deciding what it should look at, checking its judgment, and improving it as it goes.
The diagram above is a simplified view — zoom or drag to explore it. Below, I break down each part individually: what it does, why it exists, and what it actually replaces in day-to-day QA work.
1. The live AI knowledge base
Everything starts here. The system pulls technical and business context from Jira and Confluence, and uses AI to synthesize it — leaving a clear, business-friendly summary in Confluence itself, and generating a change spec that gives the update a clear purpose based on existing documentation and rules.
New information is then linked to what's already known, building a graph of interconnected concepts rather than a pile of disconnected documents. I review that graph periodically in Obsidian, where it's easy to spot which topics still lack context. Nothing gets folded in automatically without being validated against the person who owns the task — that's what keeps the base accurate instead of just large.
2. Designing the test cases
With the knowledge base as context, the system designs functional, security, usability, and non-functional test cases — and separately, automation candidates chosen through impact, variability, stability, and risk analysis. The goal isn't coverage for its own sake: it's identifying what's actually worth testing before writing a single line of automation.
Automated cases follow best practices defined in reusable skills, matched to whichever tool generates them, so the same logic can serve more than one process instead of being rebuilt from scratch every time a similar case comes up.
3. Running the tests
Automated cases run with Playwright (web and API, using the Page Object Model) and K6 for performance, validating against real databases and AWS environments in real time — not static copies of data that may no longer reflect what's actually there.
When something breaks, the system doesn't just report a failure — it iterates on its own to figure out whether it's a real issue or something in the test that needs adjusting, because it understands the goal of the test, not just its steps. I also share prompts and agents with development teams so their repositories get standardized "data-test" attributes automatically, making elements easier to locate for everyone, not just for my own automation.
4. Evidence and traceability
Every run leaves a trace: evidence gets logged and merged into the same test documentation used for manual cases, then synced with QMetry to keep everything organized by technology across environments. By the time a cycle is ready to move forward, the documentation is already done — no one has to reconstruct it after the fact.
This also removes a subtle risk from manual QA: relying on "dummy" data that behaves differently across environments. Test data is generated and validated per environment, so the evidence reflects what's actually true there — not assumptions carried over from somewhere else.
Questions I asked myself before building this
Can the AI get it wrong?
Yes. It's not infallible, and I don't treat it that way. The system works because the knowledge base is validated against real sources and refined continuously — but I still review what it produces, especially for anything high-risk. This replaces repetitive manual work, not judgment.
What happens when the business logic changes?
The knowledge base is designed to stay current, not fixed. It re-checks itself against Jira and Confluence regularly, so outdated information gets flagged and corrected instead of quietly going stale.
Does this replace manual testing entirely?
No. Exploratory and manual testing still have a place, especially for edge cases and usability. What changed is how much of the repetitive, well-defined work — writing cases, running them, documenting results — no longer needs a person doing it step by step.
Could this work outside of Paysafe?
The core idea — a living knowledge base driving test design, execution, and documentation — isn't tied to any one company. The specific integrations would change, but the approach translates to most teams working across Jira, Confluence, and a mix of environments.