App UI testing

OpenFactory can run reusable browser workflows in managed desktop tester VMs. The application may be deployed by OpenFactory or hosted anywhere the tester VM can reach.

Scenarios combine semantic actions such as open, click, type, key, wait, and assert with screenshots, browser diagnostics, and a recorded verdict. They are best for a known user flow; use the autonomous walker for bounded discovery.

Create and run a scenario

Get the app’s tester VM:


ensure_tester_vm(app="my-app", app_url="https://staging.example.com")

Then save a scenario whose steps use meaningful on-screen labels:


[
  { "action": "open_url", "value": "${APP_URL}" },
  { "action": "type", "target": "email field", "value": "${EMAIL}" },
  { "action": "type", "target": "password field", "value": "${PASSWORD}" },
  { "action": "click", "target": "Sign in" },
  { "action": "assert_text", "expect": "Dashboard" },
  { "action": "assert_no_error" }
]

Use create_app_scenario, then run the returned scenario ID with run_app_scenario. The first run resolves elements through OmniParser. Later runs can reuse the hardened element cache and re-resolve only changed steps. Review whether each step was cached or parsed when diagnosing a result.

What a passing run means

A report counts as validation only when:

its status is passed;
it targeted the intended environment and revision;
any staging-protection and application login completed; and
assertions ran inside the product, not on a login or protection page.

A passing scenario demonstrates only the declared flow and assertions. It does not establish complete application correctness, accessibility, security, or cross-browser behavior.

Variables and secrets

Steps reference values as ${VAR} and TOTP seeds as ${totp:VAR}. Non-secret defaults may live with the scenario. Secrets can be stored in the owner’s encrypted key store when the scenario is created; single-run overrides passed to run_app_scenario are merged for that run and are not persisted.

Do not place literal passwords, tokens, or TOTP seeds in step text, scenario descriptions, screenshots, or issue exports. For emailed one-time codes, the email_otp action requires an active, explicitly connected Gmail integration for the scenario owner.

Useful assertions

assert_text verifies expected visible text.
assert_no_error checks built-in or supplied error phrases.
assert_visual compares a perceptual hash with a stored baseline.
assert_stable samples a region for unexpected flicker/disappearance.
assert_progress verifies that a streaming region continues changing for a minimum period.
network can interrupt and restore the tester VM network for recovery tests.

Visual and temporal assertions are heuristics. Tune thresholds against stable test data and inspect captured frames before treating drift as a product defect.

Direct and recorded testing

For exploratory work, drive the VM with desktop_open_url, desktop_screenshot, desktop_click, desktop_type, and related desktop tools. Use start_app_test, record_app_test_step, and finish_app_test when that manual sequence should become an auditable run.

annotate_screenshot can add labeled boxes to evidence. An annotation explains what a reviewer should inspect; it does not itself prove the element worked.

Operational notes

Tester VMs are reused per app/host. Scale the tester pool for parallel work instead of creating arbitrary duplicates.
The default tester has no extra persistence disk. Request persistence only when state must survive a reboot and accept the longer first boot.
A client timeout does not necessarily cancel backend work; inspect the run before retrying.
Keep reports private unless their screenshots and diagnostics have been reviewed for sensitive data.