Testing and application verification
OpenFactory has two related surfaces:
- image scenarios that boot a built artifact and run bounded guest assertions; and
- an app-platform preview that creates immutable app variants plus UI-test and walker workflows.
Availability differs substantially across app-platform modules. Read the status on each page before using it.
App-platform availability
| Topic | Current boundary |
|---|---|
| App Deployment | Queues a Git source through the immutable variant pipeline; follow the asynchronous deployment and health evidence. |
| App UI Testing | Stores and runs bounded semantic scenarios against reachable targets; results prove only the stated actions and assertions. |
| Prompt-assisted deployment | Requires an existing template Git URL; the brief is provenance, not source generation. |
| App environment variables | Encrypted storage and deployment handoff are implemented when required keys/tokens are configured; rotation and runtime evidence remain operator concerns. |
| Managed Databases | Stub contract only; no database is provisioned. |
| Object Storage | Stub contract only; no bucket is provisioned. |
| Custom Domains | Public-DNS ownership verification only; custom TLS serving/routing is not active. |
| Checkpoints | Stub identifiers only; no recoverable snapshot exists. |
| Observability | Manual event storage and VM allocation are real; ingest, probes, samples, logs, and dashboard are incomplete. |
| Web IDE | Stub binding only; no editor or private route is provisioned. |
| App Auth | Stub binding only; no identity provider, issuer, or real token flow is provisioned. |
| Templates and Remix | Creates app records from manifests or eligible source lineage; does not deploy or automatically create declared services. |
| Autonomous Walker | Bounded UI discovery with important coverage, authorization, and side-effect limits. |
| Walker Diffs | Compares stored walks and exports ticket-shaped payloads; it does not file external tickets by itself. |
| Walk and Fix | Creates fix intent; legacy live-VM patching conflicts with the immutable deployment model. |
Do not chain a stub adapter into a production workflow because its API returned success.
Image scenarios
Image tests run only when the selected build/scenario enables them and the required test infrastructure is available. Keep these states separate:
- artifact construction;
- guest provisioning and boot;
- assertion execution;
- evidence finalization; and
- certification or publication policy.
A build can succeed while tests are disabled, pending, failed, or incomplete.
Test design
Built-in tests
Built-in names such as boot, login, packages, network, and services provide a baseline. Inspect Default Tests for their precise limitations.
Custom assertions
Use Custom Assertions for service, port, HTTP, file, command, process, and supported GUI observations. Every assertion should include a description, expected result, target, timeout, and failure meaning.
Benchmarks
Benchmark catalogs are structured check sets, not compliance determinations. Match the exact OS/version, retain applicability and failures, and read CIS benchmark evidence.
Evidence checklist
For a decision-bearing run, retain:
- recipe, source, build, artifact, VM, scenario, and run IDs;
- exact artifact and source digests;
- test environment and runner version;
- every assertion result and raw output;
- screenshots only where justified and safely handled;
- missing, skipped, or not-applicable checks;
- timestamps and terminal status; and
- reviewer disposition and approval for the stated decision.
When a test fails, diagnose the failing layer before rebuilding. A duplicate build can hide an ownership, deployment, test-runner, or evidence-finalization defect rather than fixing it.