Run OpenClaw E2E QA Tests: Automate Your Workflow

Ever felt the frustration of unit tests passing while the actual user experience is broken? It is a common headache when building complex agentic systems where you need to see how things actually play out in a chat window.

The OpenClaw QA E2E automation stack is designed to exercise the system in a realistic, channel-shaped way. This private stack allows you to move beyond basic tests and observe how your Gateway and agents handle real-world interactions.

The current setup includes several key pieces:

1. extensions/qa-channel: A synthetic message channel that handles DMs, threads, reactions, edits, and delete surfaces.
2. extensions/qa-lab: A debugger UI and QA bus for observing transcripts, injecting messages, and exporting reports.
3. qa/: A collection of repo-backed assets for kickoff tasks and baseline scenarios.
4. Associated scripts: Tools to manage the lifecycle of the QA environment.

The current QA operator flow uses a two-pane QA site:

1. Left: Gateway dashboard (Control UI) with the agent.
2. Right: QA Lab, showing the Slack-ish transcript and scenario plan.

Run OpenClaw QA E2E with Docker

You can start the full environment to see the agent in action across a two-pane dashboard. This setup gives you a Control UI on the left and the QA Lab on the right to track the scenario plan.

1. Run the following command in your terminal:

pnpm qa:lab:up

This command builds the QA site, starts the Docker-backed Gateway lane, and exposes the QA Lab page. From there, an operator or an automation loop can give the agent a QA mission, observe real channel behavior, and record what worked, what failed, what stayed blocked, and what needs attention.

If you want to iterate on the QA Lab UI quickly without rebuilding the Docker image every time, you can use a bind-mounted bundle. This approach keeps your workflow fast and responsive by auto-reloading the browser when assets change.

1. Build the image and the lab bundle.
2. Start the stack in fast mode.
3. Watch for changes to trigger auto-reloads.

pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

The qa:lab:up:fast command keeps the Docker services on a prebuilt image and bind-mounts the distribution folder into the container. The qa:lab:watch command rebuilds that bundle on change to keep your environment up to date.

Test Matrix and Telegram Transports

Testing real-world connectivity is straightforward with dedicated smoke lanes for different platforms. These lanes help you verify that your plugins work correctly with actual messaging protocols.

1. To run a smoke lane for Matrix, use the following command:

pnpm openclaw qa matrix

This process sets up a disposable Tuwunel homeserver in Docker and registers temporary users to test the real Matrix plugin. It writes structured report artifacts and logs into the selected Matrix QA output directory. If you want to capture the outer build and launcher output, you can set the OPENCLAW_RUN_NODE_OUTPUT_LOG environment variable to a local file path.

If you need to test Telegram, you will target a real private group instead of a disposable server. This requires specific bot tokens and group IDs to be configured in your environment.

1. Ensure you have OPENCLAW_QA_TELEGRAM_GROUP_ID, OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN, and OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN set.
2. Run the Telegram QA command:

pnpm openclaw qa telegram

The SUT bot must have a Telegram username, and bot-to-bot observation works best when both bots have Bot-to-Bot Communication Mode enabled in @BotFather. Live transport lanes share a contract to ensure consistent behavior across transports, as shown in the table below:

Lane	Canary	Mention gating	Allowlist block	Top-level reply	Restart resume	Thread follow-up	Thread isolation	Reaction observation	Help command
Matrix	x	x	x	x	x	x	x	x
Telegram	x								x

Execute QA Suites with Multipass

If you prefer a clean Linux environment without using Docker directly in the QA path, Multipass is a great option. This method boots a fresh guest and builds OpenClaw inside it for a truly isolated test run.

1. Execute the suite using the Multipass runner:

pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

This command boots a fresh Multipass guest, installs dependencies, and runs the qa suite. It then copies the normal QA report and summary back into the host artifacts directory. Host and Multipass suite runs execute multiple selected scenarios in parallel with isolated Gateway workers by default. You can use --concurrency <count> to tune the worker count or set it to 1 for serial execution. Live runs forward supported auth inputs like env-based provider keys, the QA live provider config path, CODEX_HOME, and the output directory.

Manage Test Plans with Repo-backed seeds

Keeping your test scenarios in the repository ensures that the QA plan is visible to everyone on the team. These assets live in the qa/ directory and act as the source of truth for your automated runs.

1. qa/scenarios/index.md
2. qa/scenarios/*.md

The QA Lab is designed to be a generic JSON and markdown runner. Each scenario file defines the metadata, code references, optional plugin requirements, and the executable qa-flow needed for the test. The reusable runtime surface that backs qa-flow stays generic, allowing markdown scenarios to combine transport-side helpers with browser-side helpers.

The baseline scenarios are broad enough to cover:

1. DM and channel chat interactions.
2. Thread behavior and message action lifecycles.
3. Cron callbacks and memory recall.
4. Model switching and subagent handoff.
5. Repo-reading, docs-reading, and small build tasks.

Plug in New Channels with Transport adapters

The architecture separates scenario execution from the specific transport details to keep things flexible. This means you can add new channels without having to rewrite your QA runner logic.

At the architecture level, the split is organized as follows:

1. QA Lab owns generic scenario execution, worker concurrency, and artifact writing.
2. The transport adapter owns Gateway config, readiness, and inbound/outbound observation.
3. Markdown scenario files define the test run.
4. The reporting system handles the final output.

If you are looking to add a new channel, check out the maintainer-facing guidance in the Testing documentation.

Generate Insights with Markdown Reporting

After a run, the system exports a detailed Markdown report that tells you exactly what happened. This report is essential for understanding what worked, what failed, what stayed blocked, and what follow-up scenarios are worth adding.

1. To evaluate character and style across multiple models, run the character-eval command:

pnpm openclaw qa character-eval \
  --model openai/gpt-5.4,thinking=xhigh \
  --model openai/gpt-5.2,thinking=xhigh \
  --model openai/gpt-5,thinking=xhigh \
  --model anthropic/claude-opus-4-6,thinking=high \
  --model anthropic/claude-sonnet-4-6,thinking=high \
  --model zai/glm-5.1,thinking=high \
  --model moonshot/kimi-k2.5,thinking=high \
  --model google/gemini-3.1-pro-preview,thinking=high \
  --judge-model openai/gpt-5.4,thinking=xhigh,fast \
  --judge-model anthropic/claude-opus-4-6,thinking=high \
  --blind-judge-models \
  --concurrency 16 \
  --judge-concurrency 16

This command runs local Gateway child processes to compare how different models handle personas and tasks. Character eval scenarios set the persona through SOUL.md and run ordinary user turns like chat, workspace help, small file tasks, and code reviews. The command preserves each full transcript, records basic run stats, and asks judge models to rank the runs by naturalness, vibe, humor, and helpfulness.

When comparing providers, you can use --blind-judge-models to replace candidate names with neutral labels. Candidate runs default to high thinking, but you can override this with ,fast, ,no-fast, ,fast=false, or ,thinking=level inline. Candidate and judge durations are recorded for benchmark analysis, though judges are instructed not to rank by speed.

Explore More in Related docs

If you want to find more details about how testing works or how specific channels are configured, check out these resources. They provide more context on the internal workings of the system.

1. Testing
2. QA Channel
3. Dashboard
4. API Reference

AI Setup Assistant

Next Steps

• Learn more about Testing
• Explore the QA Channel documentation
• Check out the Dashboard features