Why We Built One QA Runtime for Multiple Test Engines
Keep specialized test frameworks. Standardize the execution contract, the evidence and the independent verification step.
Status. ReDevOps QA is pre-release and its repository is public: github.com/redevops-io/redevops-qa. There is no tagged release, nothing is published to PyPI and there is no one-command installer, so you work from a checkout. It is source-available, not open source. Everything below is described from the code and from public CI runs, and each claim says what kind of evidence stands behind it.
The problem is not a lack of frameworks
Most teams that test seriously already have several engines: Playwright or Selenium for the browser, Robot Framework for keyword suites, Appium for mobile. Each is good at its job. The trouble starts when a release decision needs all of them. Each engine has its own setup, its own artifacts and its own idea of what a pass means, so the decision is assembled by hand from reports that do not line up.
Replacing the engines is the wrong fix. The part worth sharing is the layer around them: how a run is requested, what it may touch, what evidence it leaves, and who decides that the evidence shows the outcome that was asked for. ReDevOps QA is that layer. It does not translate one engine's tests into another's, and a project can use more than one engine and read the results through one contract.
What rAIls adds to the QA engine stack
The engines run tests. They do not give you a shared request format, a record of a run that can be checked afterwards, a rule about who may say a test passed, or a network policy that applies the same way to every engine. That layer is built on the rAIls core runtime packages, and this is what the code contains, with where to read it:
| Piece | What it does | Where |
|---|---|---|
| Shared contracts | The qa/v1 schemas: a test spec whose assertions are frozen and referenced by hash, a plan, run records, a run manifest with assertion accounting, and the adapter protocols. | runtime-contracts, pinned by release tag and digest in pins.toml |
| Run harness | One spec in one isolated session, a run state machine, a fail-closed action policy, bounded recovery, redaction, and an integrity check over the receipt chain, the event chain and the artifact hashes, so a run's record preserves or traces transactional integrity. | agent-harness docs/qa.md |
| Independent verifier | Loads the frozen assertions by their pinned hash, recomputes the hash and refuses on any difference. Adapters see no expected value, and nothing in a run can rewrite an assertion. | same document |
| Egress control | A gateway that every browser request passes through: default-deny allow-list, the resolved address pinned, each redirect hop decided again before it is followed. | qa-egress |
| Adapter contract | A shared supervisor (deadline, output limit, process-group cancellation, executable and path allow-lists, an environment built from nothing, secret redaction) and models that keep an engine's native result apart from a verdict. | qa-adapter-kit |
| Conformance | A suite that checks the contracts and the harness mirror agree on hashes, enums, protocols and refusals, and writes a compatibility receipt that can be verified later. | qa-bridge |
| Deterministic checks | Hashed, versioned programs that only observe a value and never see the expected one. | qa-checks |
For agent-driven use, the harness defines a model-candidate recovery mode. It is off unless a run enables it, it sits behind an injected client, and the module holds no model SDK or key.
Native PASS is not a verified outcome. An engine's own PASS counts as evidence for an assertion only when the request maps it explicitly and the test declares the same assertion id. Even then the status stays UNVERIFIED until the independent verifier supplies a verdict that cites the digest of that exact evidence. An adapter cannot verify its own result.
Two limits on this table. The verified install profiles below install the adapter packages only; none installs the contracts bridge or the harness verifier, which have their own CI jobs. And nothing here is a guarantee, or a claim that a result is semantically correct.
Installing and using it today
The honest path is a checkout of the public repository at a pinned commit. There is no release tag, no PyPI package and no one-command installer.
The static configurator at redevops.app/qa/install turns a choice of engines, host and channel into a validated, pinned list of steps and a manifest. It runs in your browser, runs nothing for you and sends nothing anywhere. It refuses combinations that have no profile and marks what is not offered, and it never calls a profile production-ready.
| Profile | Status | What backs it |
|---|---|---|
| Robot Framework, Linux x86_64 | Verified in CI | run 41306373018, job 124847489639: install, smoke and check, then uninstall, from the pinned commit in a clean virtual environment |
| Selenium, Linux x86_64 | Verified in CI | same run, job 124847489672, with the runner's Chrome and ChromeDriver 156 |
| Robot on Linux arm64 and macOS, Selenium on macOS | Experimental | steps written, no CI job runs them on that host |
| Appium Android | Experimental | Python-side steps run in CI; the device step is manual and has never run in CI |
| Playwright, Appium iOS, Docker Compose, Kubernetes, Windows and WSL | Not offered | each has a reason in the data |
The independent verifier is not offered either: no profile installs it. A worked example, with the commands exactly as the generator prints them for Robot Framework on Linux x86_64:
git clone https://github.com/redevops-io/redevops-qa.git redevops-qa
git -C redevops-qa checkout --detach 5e9c8da6248f9c4e170b77ab4ff202b519c970a8
python3.12 -m venv .venv-qa
.venv-qa/bin/python -m pip install --disable-pip-version-check pytest==9.2.0 pyyaml==6.0.3 robotframework==7.5
.venv-qa/bin/python -m pip install --disable-pip-version-check --no-deps redevops-qa/packages/qa-adapter-kit redevops-qa/packages/qa-robot
.venv-qa/bin/python -m pytest -q -p no:cacheprovider -c redevops-qa/pyproject.toml -o pythonpath= --junitxml=qa-evidence/junit-robot-real.xml -m robot_real redevops-qa/packages/qa-robot/tests/real
.venv-qa/bin/python redevops-qa/tools/check_junit.py qa-evidence/junit-robot-real.xml
The generator also prints the other smoke and check steps, an import check and an uninstall step. The reports it leaves show the adapter and the real engine passing their own tests. They are not an independent verdict, and pip resolves transitive dependencies at install time without pinning them. Host coverage is being extended; there is nothing to report from that yet.
What we have and have not validated
| Engine | What has been shown | Where |
|---|---|---|
| Playwright | Real Chromium lane: 28 tests, Playwright 1.64.1, Ubuntu 24.04 | CI (job) |
| Selenium | Real Chrome lane: 11 tests, Selenium 4.52.0, Chrome 156.0.8078.47 | CI (job) |
| Robot Framework | Real framework lane: 10 tests, Robot Framework 7.5 | CI (job) |
| Appium server | Real Appium 3.9.0 server, UiAutomator2 8.7.3, no device; drives no app | CI (job) |
| Appium on Android | One test passed on an Android 17 (API 37, arm64) emulator, three runs | Local only, never in CI |
| Appium on iOS | Nothing; the adapter reports UNAVAILABLE | No run |
These engine lanes are from commit 938eb48 on 1 November 2026 (UTC), in the public run, and they will move. The Selenium, Robot Framework and Appium server lanes are not yet required checks. The Android pass is one emulator on one machine; hosted CI runners have no emulator, and server startup is not a device session. So Appium is experimental, iOS is not available, and we do not claim that every adapter is production-ready. The benefits we are aiming for (less duplicated integration work, mixed-engine workflows, consistent evidence, and a separation between an engine finishing and the outcome being verified) are design goals. We have not measured a time saving or a reliability gain.
What comes next
Each item below is recorded in the repository's data as not offered, or in our plan as later. None is built, and none has a date.
- A tagged release and a package. Today there is neither.
- A Docker Compose profile, once a packaging is merged and run in CI.
- A Kubernetes profile, once there is a chart to test.
- A Playwright install profile, which needs a tested path for the pinned harness tree and the browser download.
- Windows, WSL and Intel macOS profiles, and more verified hosts.
- A device-backed CI lane for Appium. The matrix defines the level, and no engine has reached it.
- Installing the independent verifier through a tested profile.
- Later: a downloadable GitHub Actions workflow, a self-hosted worker, an engine registry and a hosted evidence viewer. A managed control plane would come only after demand and a security review.
Frequently asked questions
Does it replace Playwright? No. It runs the engines under one contract.
Do I need an LLM? Not for the engine adapters: they run an engine and import its report. Model-assisted recovery in the harness is off by default.
Does PASS mean verified? No. A native PASS is evidence only through an explicit mapping, and verification checks recorded evidence against the assertions a request names. It does not prove semantic correctness.
Can I run iOS in a Linux container? No. iOS needs a macOS host with Xcode that you supply, and no iOS session has been run.
Can one workflow mix engines? Yes, at the level of requests and evidence. One engine does not run another's tests unchanged.
Try it
The ReDevOps QA page has the current status table. Clone the repository, use the configurator, or join the pilot to try it on your own suite with us. For how we test the agent layer above it, read Stress-Testing Agent Harnesses Where Models Cannot Hide Their Weaknesses.