Custom Assertions

Define your own test criteria to verify specific functionality in your build.

Assertion Structure


{
  "description": "Verify my application",
  "assertions": [
    {
      "type": "service_running",
      "params": { "service": "myapp" }
    }
  ]
}

Assertion Types

user_exists

Verify a user account exists with expected configuration.


{
  "type": "user_exists",
  "params": {
    "username": "deploy",
    "groups": ["docker", "sudo"]
  }
}

Parameter	Required	Description
`username`	Yes	Username to check
`groups`	No	Expected group memberships
`shell`	No	Expected login shell
`home`	No	Expected home directory

service_running

Check that a systemd service is active.


{
  "type": "service_running",
  "params": {
    "service": "nginx"
  }
}

Parameter	Required	Description
`service`	Yes	Service unit name

service_enabled

Check that a service is enabled at boot.


{
  "type": "service_enabled",
  "params": {
    "service": "docker"
  }
}

package_installed

Verify a package is installed.


{
  "type": "package_installed",
  "params": {
    "package": "nginx",
    "version": "1.24"
  }
}

Parameter	Required	Description
`package`	Yes	Package name
`version`	No	Expected version (prefix match)

port_listening

Check that a port is open and listening.


{
  "type": "port_listening",
  "params": {
    "port": 443,
    "protocol": "tcp"
  }
}

Parameter	Required	Description
`port`	Yes	Port number
`protocol`	No	`tcp` or `udp` (default: tcp)
`address`	No	Bind address (default: any)

http_responds

Verify an HTTP endpoint responds correctly.


{
  "type": "http_responds",
  "params": {
    "url": "http://localhost:8080/health",
    "status": 200,
    "contains": "healthy"
  }
}

Parameter	Required	Description
`url`	Yes	URL to request
`status`	No	Expected HTTP status (default: 200)
`contains`	No	Response body must contain
`method`	No	HTTP method (default: GET)
`timeout`	No	Request timeout in seconds

file_exists

Check that a file exists.


{
  "type": "file_exists",
  "params": {
    "path": "/etc/myapp/config.yml"
  }
}

Parameter	Required	Description
`path`	Yes	File path
`mode`	No	Expected permissions (e.g., “0644”)
`owner`	No	Expected owner
`group`	No	Expected group

file_contains

Verify file contents.


{
  "type": "file_contains",
  "params": {
    "path": "/etc/ssh/sshd_config",
    "content": "PermitRootLogin no"
  }
}

Parameter	Required	Description
`path`	Yes	File path
`content`	Yes	String or regex to find
`regex`	No	Treat content as regex

command_output

Run a command and check its output.


{
  "type": "command_output",
  "params": {
    "command": "docker --version",
    "contains": "Docker version",
    "exit_code": 0
  }
}

Parameter	Required	Description
`command`	Yes	Command to execute
`contains`	No	Output must contain
`exit_code`	No	Expected exit code
`timeout`	No	Command timeout in seconds

process_running

Check that a process is running.


{
  "type": "process_running",
  "params": {
    "name": "nginx",
    "user": "www-data"
  }
}

Parameter	Required	Description
`name`	Yes	Process name
`user`	No	Expected user
`count`	No	Expected process count

mount_exists

Verify a mount point exists.


{
  "type": "mount_exists",
  "params": {
    "path": "/data",
    "fstype": "ext4"
  }
}

Parameter	Required	Description
`path`	Yes	Mount point
`fstype`	No	Filesystem type

cis_benchmark

Run CIS benchmark checks. See CIS Benchmarks.


{
  "type": "cis_benchmark",
  "params": {
    "level": 1,
    "profile": "server"
  }
}

GUI & Display Testing

gui_application_opens

Launch a GUI application and verify its window appears.


{
  "type": "gui_application_opens",
  "params": {
    "command": "firefox",
    "window_title": "Firefox",
    "timeout": 10
  }
}

Parameter	Required	Description
`command`	Yes	Command to launch application
`window_title`	Yes	Expected window title
`timeout`	No	Seconds to wait for window (default: 10)

gui_window_visible

Check if a window with specific title is visible.


{
  "type": "gui_window_visible",
  "params": {
    "window_title": "EDC Login"
  }
}

Parameter	Required	Description
`window_title`	Yes	Window title to check

gui_execute_command

Execute a command and capture a VNC screenshot.


{
  "type": "gui_execute_command",
  "params": {
    "command": "weasis /opt/test-data/image.dcm",
    "wait_seconds": 5
  }
}

Parameter	Required	Description
`command`	Yes	Command to execute
`wait_seconds`	No	Wait time before screenshot (default: 2)

Automatically captures screenshot + metadata (PNG + JSON) for visual validation and FDA audit trails.

gui_application_process

Verify an application process is running.


{
  "type": "gui_application_process",
  "params": {
    "process_name": "firefox"
  }
}

Parameter	Required	Description
`process_name`	Yes	Process name to check

gui_screenshot_matches

Compare current screenshot against a reference image for visual regression testing.


{
  "type": "gui_screenshot_matches",
  "params": {
    "reference_id": "login_screen_v1",
    "threshold": 0.95,
    "ignore_regions": [[10, 10, 200, 30]],
    "save_as_reference": false
  }
}

Parameter	Required	Description
`reference_id`	Yes	Unique identifier for baseline image
`threshold`	No	Similarity threshold 0.0-1.0 (default: 0.95)
`ignore_regions`	No	List of [x, y, width, height] regions to mask
`match_area`	No	[x, y, width, height] to crop both images
`save_as_reference`	No	Save current screenshot as new baseline
`wait_before`	No	Seconds to wait before capture (default: 1.0)

Use for: Detecting UI regressions, verifying layouts, validating visual changes

gui_click_element

Click a UI element at specified screen coordinates.


{
  "type": "gui_click_element",
  "params": {
    "x": 600,
    "y": 400,
    "button": "left",
    "double_click": false
  }
}

Parameter	Required	Description
`x`	Yes	X coordinate
`y`	Yes	Y coordinate
`button`	No	Mouse button: left/right/middle (default: left)
`double_click`	No	Click twice (default: false)
`wait_after`	No	Seconds to wait after click (default: 0.5)
`verify_click`	No	Capture verification screenshot (default: true)

Use for: Navigating menus, clicking buttons, interacting with custom controls

gui_form_fill

Fill form fields and optionally submit.


{
  "type": "gui_form_fill",
  "params": {
    "fields": [
      {"name": "username", "value": "testuser", "method": "tab"},
      {"name": "password", "value": "pass123", "method": "tab"}
    ],
    "submit_button": {"x": 600, "y": 500}
  }
}

Parameter	Required	Description
`fields`	Yes	List of field objects (see below)
`submit_button`	No	`{x, y}` coordinates to click after filling
`wait_after_submit`	No	Seconds to wait after submit (default: 1.0)
`verify_submission`	No	Capture screenshot after submit (default: true)

Field object: {name, value, method, x, y, wait_after}

Use for: Automated data entry, form validation testing, workflow automation

gui_text_visible

Check if text is visible on screen using OCR.

Requires: pytesseract and tesseract-ocr system package


{
  "type": "gui_text_visible",
  "params": {
    "text": "Login Successful",
    "case_sensitive": false
  }
}

Parameter	Required	Description
`text`	Yes	Text to search for
`region`	No	[x, y, width, height] to crop before OCR
`case_sensitive`	No	Case-sensitive search (default: false)
`lang`	No	Tesseract language (default: eng)
`wait_before`	No	Seconds to wait before capture (default: 1.0)

Use for: Verifying messages, checking labels, validating text in legacy apps

Natural Language Examples

Web Server Verification


Test that:
- nginx is running
- Port 80 is listening
- http://localhost returns 200
- The homepage contains "Welcome"

Docker Setup


Verify Docker:
- docker service is running
- docker.sock exists
- User 'deploy' can run docker commands
- Can pull hello-world image

Security Checks


Security assertions:
- SSH is on port 2222
- Root login is disabled
- UFW is active
- Only ports 22, 80, 443 are open

Complete Test Example


{
  "tests": [
    {
      "description": "Verify web application",
      "assertions": [
        {
          "type": "service_running",
          "params": { "service": "nginx" }
        },
        {
          "type": "port_listening",
          "params": { "port": 80 }
        },
        {
          "type": "port_listening",
          "params": { "port": 443 }
        },
        {
          "type": "http_responds",
          "params": {
            "url": "http://localhost/health",
            "status": 200
          }
        }
      ]
    },
    {
      "description": "Verify database",
      "assertions": [
        {
          "type": "service_running",
          "params": { "service": "postgresql" }
        },
        {
          "type": "port_listening",
          "params": { "port": 5432 }
        }
      ]
    }
  ]
}

Test Failure Behavior

By default, test failures:

Are reported in detail
Don’t prevent ISO download
Are flagged with warnings

To make tests blocking:


These tests must pass before the build completes:
- Application health check
- Security baseline

Debugging Failed Tests

View test logs in Build Details
Check assertion details for specific errors
Use VM Management to manually inspect the system
Add debug commands like journalctl output