# Live-app QA & driving UIs Source: https://flow-next.dev/guides/live-qa/ The live pass that drives the running app against the spec, and the driver ladder underneath it - surface detection, the web rungs, the native rung, and graceful degradation. Every other review surface in Flow-Next reads code or specs. A live pass exercises the deployed app. It drives the **running** thing like an unforgiving customer, derives its scenarios directly from the spec, files findings with evidence, and ends with a ship verdict. It is **forbidden from marking PASS by reading source**, because a scenario passes only on captured evidence (screenshot, console, URL) and never on inspection. [`/flow-next:qa`](https://flow-next.dev/skills/qa/) is the **cheap first live pass**: the app already runs on the developer’s machine during `/flow-next:work`, so this drives it like a real user before a human opens the PR. Its findings are advisory, riding the PR and the bug-memory track, and the human reviewer plus the [land](https://flow-next.dev/autonomy/land/) gate decide. Use it to shrink the QA humans have to do, not to remove them from the loop. CI, staging, and manual QA still happen downstream. When a repo has a committed feature map at `.flow/features/` (seeded by [`/flow-next:features`](https://flow-next.dev/skills/features/)), QA reads the index and the one matching feature file. It uses the file for **navigation only**: how a user reaches the surface, the preconditions to get there, and the gotchas that waste a run. The spec still supplies this run’s acceptance criteria and R-IDs, and captured live evidence is still the only basis for a PASS. Discovery is an existence check, so an absent map, or a per-target miss inside a present one, falls back to the normal derivation with nothing to configure either way. A route the run finds stale is filed as a `feature-map-drift` memory entry, then the run finds the route live and continues; QA never edits the map mid-run. Open drift notes make the map due a maintain pass, and work updates the routes its own change moved: [Keep the feature map current](https://flow-next.dev/guides/keep-feature-map-current/). Flow-Next runs fully without this. It costs a live-app drive pass per spec, plus a running deploy and a configured driver. Set `pipeline.qa` to `on` when nobody will exercise the app before merge, to `auto` when only some specs have a drivable surface, or invoke `/flow-next:qa` manually on the changes that deserve it. ## The spec-as-intent advantage Spec-less QA tools have to reconstruct what the app is *supposed* to do from READMEs and landing pages. Flow-Next already has the spec, so QA derives scenarios straight from it: * **Acceptance criteria → test scenarios.** * **R-IDs → a coverage table** - the same requirement ids [`make-pr`](https://flow-next.dev/skills/make-pr/) traces in its coverage lines. * **Boundaries → what NOT to test** - suppresses false bugs. * **Decision context → expected behavior.** The result is bidirectional traceability: spec AC ↔ scenario ↔ finding ↔ R-ID. ## Lifecycle position QA runs after `/flow-next:work`, **before** [`make-pr`](https://flow-next.dev/skills/make-pr/). There are three ways to reach it: * **User-invoked** - run `/flow-next:qa ` by hand whenever you want a live pass. * **The optional pipeline stage** - wire it in with `flowctl config set pipeline.qa on` (default **off**). When on, [`/flow-next:flow`](https://flow-next.dev/skills/flow/) and [`/flow-next:flow --auto`](https://flow-next.dev/autonomy/pilot/) insert a `qa` stage at the **all-tasks-done** juncture, one live pass over the complete build just before make-pr (plan and plan-review run before work only on a plan signal or a named design risk): ```text work → qa → make-pr ``` The stage is **evidence-aware** (it leans on what `work` already verified - it never re-runs a deterministic test/lint/build check, but always live-runs every runtime/UI/integration AC), **idempotent** (a `head_sha` freshness gate runs it at most once per branch head, and a pass after a fix [re-checks only what changed](https://flow-next.dev/skills/qa/#re-check) since the last receipt), and **autonomy-safe**: it never prompts, the `flow --auto` gate routes on `qa_outcome`, `SHIP`/`NA` advance cleanly, and **`NEEDS_WORK` and `BLOCKED` still advance** to the PR - their findings become open items on a draft PR, plus the bug-memory track and a tracker comment when the bridge is active. QA never hard-blocks the loop; merge stays the human’s + land’s decision. With the gate **off**, the unattended stage set is byte-for-byte unchanged. * **`auto`, judged by the same rule attended and unattended** - `flowctl config set pipeline.qa auto` runs the pass under [`/flow-next:flow`](https://flow-next.dev/skills/flow/) and [`flow --auto`](https://flow-next.dev/autonomy/pilot/) only when the spec’s acceptance describes UI behaviour on a drivable surface and a target can be started (a documented start command, a deploy URL, or the feature map). Otherwise the stage records `skipped(config: pipeline.qa=auto: )` and the route advances to make-pr. `flow --auto` reads the key through the same gate-selection reference attended flow reads. Setup asks the question once. Whatever the answer, it also recommends [`/flow-next:features`](https://flow-next.dev/skills/features/) when the repo has no feature map (or a due one), since QA reads `.flow/features/` to navigate the app. The two halves of that `auto` rule are split by who can answer them: code resolves the startable target (a documented start command, a deploy URL, or the features map), and the agent judges whether the acceptance criteria describe UI behaviour on a drivable surface. The same holds with or without a TypeSafe key; the QA gate never asks [Jev](https://flow-next.dev/guides/jev-judge/). The stage line reads `ran (target: )` or names the failing half, `skipped(config: pipeline.qa=auto: no UI-observable criteria)`. It never hard-blocks a run. When the tracker bridge is configured, the verdict can post to the linked issue (opt-in via `tracker.perEvent.qa`). ## The driver underneath QA never re-implements driving. It consumes the surface-aware driver ladder below, which is also available on its own for navigating a site, verifying a deployed change, capturing a baseline screenshot, or driving a desktop app. ## Step 1: Detect the surface, then branch | Surface | What it is | Path | | --------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | **Web app** | A URL in a browser (localhost dev server, staging, production) | **Web ladder** | | **Chromium-backed desktop app** | Electron / Windows WebView2 - Chromium under the hood, exposes a CDP debug port | **Web ladder**, attaching over CDP to the app’s remote-debugging port | | **True-native / non-CDP surface** | macOS AppKit/SwiftUI, Catalyst, or a webview exposing no CDP (macOS WKWebView, which Tauri uses on macOS) | **Native rung** → Cua Driver / Computer Use / Cua Sandbox | Per-platform caveat: **Windows WebView2 is CDP-drivable** (web ladder); **macOS WKWebView generally is not** (native rung). When unsure whether a desktop app exposes CDP, probe for the web ladder first (try to launch/attach with a debug port); if no port is reachable, fall to the native rung. ## Step 2: The universal flow (all surfaces) Whatever driver the environment has, the work is the same shape: ```text observe / navigate to the target snapshot → fresh element refs (REQUIRED before each act) act → click / fill / type / press / scroll verify → confirm the expected text / state appeared capture → screenshot + console/errors (and on failure) release → close the tab / end the session when fully done ``` Refs (`@e1`, `@e2`, …) go **stale** after any navigation, click, or form submit - always re-snapshot. “ref not found” or “pointer-events: none” almost always means a stale snapshot, not a real bug. ## Step 3: Web ladder (web apps + Chromium-backed desktop apps) Probe availability top-down and use the **highest rung that passes**; fail soft to the next; the terminal rung is manual. | Rung | Driver | Use when | | ------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 (default) | **agent-browser** CLI | Always assumed present. CDP-based, headless-safe, no extra install. Drives web apps; drives Electron / WebView2 over CDP (`--cdp ` / `--auto-connect`). | | 2 | **chrome-devtools-mcp** | You want built-in auto-wait, DevTools-grade network/console inspection, Lighthouse, or to **attach to your real signed-in Chrome** (`--browser-url`) so bot defenses don’t challenge an automated profile. | | 3 | **Playwright** | The repo already has Playwright configured, or you need a headless CI-style / cross-browser regression run. | | 4 | **cursor-ide-browser** MCP | On Cursor: no install. Probe by id `cursor-ide-browser` (a catalog miss is not absence). If that probe fails in an attended session, type `@Browser` (no space) or open the Browser pane until it shows connected, then the agent re-probes once. Snapshot YAML + `browser_cdp`. Cannot capture console/network from the driven page - a QA pass here records BLOCKED, not PASS. | | 5 (terminal) | **Manual + screenshot relay** | No browser driver available - drive yourself, paste console errors and screenshots into chat. | The same ladder drives Electron / WebView2 apps by attaching to the app’s remote-debugging port. Launch the app with a dedicated debug port and user-data-dir; treat the open debug port as a security exposure (any local app can drive that session). On Cursor, rung 4 needs no extra package. If the probe fails, type `@Browser` (no space) or open the Browser pane until it shows connected, then the agent tries once more. Unattended runs skip that ask and fall through. A mid-run vanished MCP is a partial pass - `@Browser` does not restore it. Recovery: [Install → Cursor](https://flow-next.dev/install/#cursor), [Troubleshooting](https://flow-next.dev/reference/troubleshooting/#cursor-in-ide-browser-mcp-missing-cursor-ide-browser). ## Step 4: Native rung: Cua Driver → Computer Use → Cua Sandbox A genuinely native app (or a non-CDP webview) has no browser tab to attach to - the only way to drive it is to operate the live machine. This rung is **provider-agnostic**: probe for the best available driver, prefer the highest that passes, degrade to the next. None of these is ever a hard dependency. | Probe | Driver | What it is | | ---------------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cua-driver` MCP registered / `command -v cua-driver` (real display) | **Cua Driver** | [trycua/cua](https://github.com/trycua/cua) (MIT) - provider-agnostic, **background** (no focus steal), **macOS / Windows** (Linux is **pre-release/experimental** - Wayland-only limits; fall to the sandbox or documented-limitation there), **accessibility-tree-based** (drives structured `element_index` elements, not pixels). Preferred when present. | | Codex CU available, or a Claude Computer-Use harness present | **Computer Use** | Codex Computer Use (macOS / Windows) / Anthropic Claude Computer Use (the API `computer` tool via its own harness). Screen-takeover. | | **Headless / CI** (no display) + a sandbox backend (`lume` / Docker / QEMU, or opted-in cloud) | **Cua Sandbox** | Drives inside an isolated VM/container - the **only** native option with no real screen. Opt-in per run, torn down each run; local backend default, `cua.ai` cloud explicit opt-in. | The actuation differs from the web ladder, but the universal flow (Step 2) is identical - `observe → snapshot → act → verify → capture`, described as goal + success state, not pixel coordinates. **Attended vs headless splits the precedence.** On a real display, prefer the background **Cua Driver** → **Computer Use** (screen-takeover) → documented-limitation. On a **headless / CI** host (no screen), the **Cua Sandbox** is the only native option - local Computer Use *and* the local Cua Driver both need a real display. Web/Chromium surfaces (Electron / WebView2) are unaffected - they stay on the web ladder by CDP-attach even in CI, never on this rung. **Determining headless / CI is empirical, not env-sniffing.** `$CI` ⇒ headless; otherwise ask the driver - `cua-driver call get_screen_size` reports a real display (positive width **and** height). Two guards keep it honest: a **missing driver is *unknown*, not headless** (don’t route an attended machine without `cua-driver` to the sandbox), and **TCC grants are not a display signal** (missing Accessibility → fall to Computer Use; missing Screen Recording → AX-only evidence). `$DISPLAY` is unset on a fully-displayed Mac, so it is a Linux-only hint. ### Cua Driver: install + wiring (detect-and-instruct, never auto-installed) Flow-Next **detects and instructs**; it never runs these for you - the same no-auto-install consent rule `/flow-next:map` applies to `clawpatch`. The base install stays zero-dependency; `agent-browser` remains the only assumed-present driver. **1. Install the driver** (upstream installers): ```bash # macOS / Linux /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" ``` ```powershell # Windows (PowerShell) irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex ``` These are upstream’s official installers, piped from the **mutable `main` branch** - on sensitive or CI machines, review the script first (drop the pipe and inspect) or pin a release tag instead of `main`. **2. Register the MCP server with your host** (host-specific): ```bash # Claude Code CLI claude mcp add --transport stdio cua-driver -- cua-driver mcp ``` ```toml # Codex - add to ~/.codex/config.toml [mcp_servers.cua-driver] command = "cua-driver" args = ["mcp"] ``` Any other MCP host: register a stdio MCP server whose command is `cua-driver mcp`. **3. Grant macOS permissions (one-time) - the load-bearing step.** macOS TCC grants **Accessibility** and **Screen Recording** *separately*, and they unlock different things: **Accessibility** unlocks *driving* (launch / click / type / read the live AX tree); **Screen Recording** unlocks *screenshots*. Request **both**: ```bash cua-driver permissions status # report which grants are present cua-driver permissions grant # walk you through both grants ``` The grant is attributed to `com.trycua.driver`. **The daemon must restart to pick up a newly-granted permission.** (Linux / Windows have no TCC split - skip this step there.) **4. (Optional) Run the daemon** so the MCP tools share one driver process: ```bash cua-driver serve # macOS / Linux, and Windows from an interactive desktop session ``` Without `serve`, tools fall back to an in-process driver with a warning. **On Windows from SSH or a non-interactive shell**, a bare `cua-driver serve` lands in Session 0 (the GUI tools return empty arrays - installed but undriveable); use `cua-driver autostart enable` then `cua-driver autostart kick` to run the daemon in the user’s interactive session instead. For sandbox / cloud operations and the full API, see upstream [`libs/cua-driver/README.md`](https://github.com/trycua/cua/blob/main/libs/cua-driver/README.md). ### The macOS permission split: AX-only evidence Because Accessibility and Screen Recording grant independently, the driver can **drive fully while screenshots stay empty**. When Screen Recording is absent, the rung still drives and captures the **accessibility tree as structured live-state evidence**, and surfaces *“Screen Recording not granted ⇒ AX-only evidence, no screenshot”* - it never presents a missing screenshot as a captured one. `/flow-next:qa` (downstream) decides whether AX-only evidence meets its bar. ### Cua Sandbox: the headless/CI surface The **Cua Sandbox SDK** drives an app inside a disposable VM/container (any OS) - the one surface the local driver and Computer Use can’t reach, because both need a real display. It’s **opt-in per run** and **torn down each run** (no leaked VMs). Two backends, very different profiles: * **Local** (`lume` / QEMU / Docker) - a VM/container on your own hardware. **The default sandbox backend** - for a fully offline run, set `CUA_TELEMETRY_ENABLED=false` (the `cua` SDK ships anonymous usage telemetry **on by default**). * **Cloud** (`cua.ai`) - a managed VM on Cua’s infrastructure. **Explicit opt-in only**: it bills and the driven screen + data leave the machine (data egress); requires a `CUA_API_KEY`. Never auto-selected. The SDK installs with `pip install cua` (**Python 3.12+** - Flow-Next itself needs 3.11+, so use a dedicated venv: `python3.12 -m venv .cua && .cua/bin/pip install cua`). For the persistent (non-ephemeral) form, delete explicitly with `Sandbox.delete(name, local=True)` - the default `local=False` targets the cloud namespace, so omitting it after a local create leaves the VM running. The first `lume pull` of a macOS image downloads a multi-GB disk image (\~30 GB) - a one-time “coffee break,” not a hang. ### Licensing The **default driving path - the background `cua-driver` MCP - uses only MIT components** and needs none of the vision/OmniParser stack. The optional `cua-agent[omni]` pulls **ultralytics (AGPL-3.0)** and **OmniParser** is **CC-BY-4.0**; the rung documents which extras are MIT-safe vs AGPL/CC-BY and **never auto-installs** any of them. ## Graceful degradation The native rung never blocks - it falls through, never failing silently: * **Cua Driver not installed / MCP not registered** → print the install + wiring commands (Flow-Next can’t install for you), fall to **Computer Use** → documented-limitation. * **macOS, Accessibility not granted** → only no-grant reads work; guide you through `cua-driver permissions grant` + a daemon restart, fall to Computer Use meanwhile. * **macOS, Screen Recording not granted** → drives fully, **AX-only evidence** (surfaced explicitly, never an empty screenshot). * **Headless / CI, no local sandbox backend and no opted-in cloud** → no native driver reachable (local driver + Computer Use both need a display); document the limitation and print the sandbox install. Never auto-select the billing cloud. * **Chromium-backed app** (Electron / WebView2) → **not this rung** - drives via the web-ladder CDP attach, even headless/CI. (Shell-level integration - system tray, native menus, OS dialogs - can’t be reached over CDP; surface that limitation.) * **Genuinely native app with no native driver at all** → document the limitation rather than fail. `agent-browser` stays the only assumed-present driver. No MCP server, Cua Driver, or Computer Use is ever a hard install dependency, and `flowctl` never imports any of them. ## Where it stops * Not a test framework. There is no assertion DSL, no parallel runner, no flake retry. * Not a scraper for restricted sites. Respect robots.txt and the target’s terms of service. * It orchestrates drivers - it does not reimplement Playwright or Computer Use. ## Common pitfalls * Forgetting to re-snapshot after a click or form submission - refs become stale (the Cua Driver’s `element_index` values go stale the same way after any UI mutation; re-read `get_window_state` before each act). * Routing an Electron / WebView2 app to the native rung - it’s Chromium, drive it over CDP via the web ladder (even though Cua’s `launch_app` exposes the debug ports). * Treating any native driver as a default - most environments lack one; the Cua Driver / Computer Use / Cua Sandbox is the native-rung fallback, not the common case. * Skipping `@Browser` on Cursor when the in-IDE browser probe fails - a catalog miss is not absence; one `@Browser` (no space) or a connected Browser pane, then one re-probe. Do not loop. A mid-run MCP drop is a different flake. * Reaching for the local Cua Driver in CI - it needs a real display; the headless/CI native surface is the **Cua Sandbox** rung. * Treating a present `CUA_API_KEY` as consent to use the billing cloud - the cloud sandbox is opt-in per run; the zero-network local backend is the default. Invocation, the findings format, the verdict table, and requirements are on the [QA skill reference](https://flow-next.dev/skills/qa/) and [Flow-Next Drive](https://flow-next.dev/skills/flow-next-drive/).