Custom Assertions

Custom assertions turn a requirement into a command the VM harness can execute. Use them for behavior that the small default suite does not cover.

Structure

Each custom-test group needs a description and one or more structured assertions:

{
  "description": "Verify the web service",
  "category": "application",
  "assertions": [
    {
      "type": "service_running",
      "description": "Nginx is active",
      "params": { "service": "nginx" },
      "expected_visual": "The evidence panel shows nginx active"
    },
    {
      "type": "http_responds",
      "description": "Local health endpoint responds",
      "params": { "url": "http://localhost/health", "status": 200 }
    }
  ]
}

description is required both on the group and on each assertion. on_vm selects a VM ID in a multi-VM test environment; without it the runner uses primary. expected_visual supplies the intended screenshot state when the visual evidence gate is active.

Design assertions around outcomes

• Check a service with service_running, not merely that its package exists.
• Check a local endpoint before testing an external route.
• Use file_contains for one stable configuration fact, not a complete-file snapshot that breaks on harmless formatting.
• Use command_succeeds only with deterministic, non-interactive commands.
• Give long commands timeout_seconds only when needed. The executor clamps command timeouts to 1–1,800 seconds.
• Never put credentials in commands, descriptions, expected output, or URLs; these fields can appear in evidence and logs.

Failure semantics

An unknown type is currently accepted by the recipe model but becomes an error when the executor cannot find a handler. Known assertions that omit required parameters can be dropped during test-plan assembly with a warning. Therefore, validate the recipe and inspect the materialized test plan before starting a build.

If on_vm names a VM that is not present, the assertion is skipped. Do not treat skipped or error as successful verification.

Multi-VM example

{
  "description": "Client reaches the API node",
  "assertions": [
    {
      "type": "http_responds",
      "description": "API health is reachable from the client",
      "on_vm": "client",
      "params": { "url": "http://api:8080/health", "status": 200 }
    }
  ],
  "environment": {
    "vms": [
      { "vm_id": "client", "role": "client", "networks": ["lan"] },
      { "vm_id": "api", "role": "server", "networks": ["lan"] }
    ],
    "networks": [
      { "network_id": "lan", "type": "isolated", "dhcp": true }
    ]
  }
}

VM-name resolution depends on the test topology and its discovered addresses. Confirm the rendered topology and the assertion’s resolved command in the run evidence.

For every supported type and parameter, see Assertion types.