Running tests

This section will show you how to run your tests and the available command line arguments

Test execution

Run the python script test_runner.py with the --platform= parameter:

python test_runner.py --platform=android

With this minimum setup the test runner will execute all available tests for the given platform

In order for a test to be discoverable, the test method name must have the prefix: test_ For example: test_create_new_report

Tests methods that do not have that prefix will not be recognized by the test runner as test methods

Configuration with local_settings.py

Note

All integrations require proper configuration in your local_settings.py file. This file should be created from the local_settings.py.example template in your test repository. Never commit local_settings.py to version control as it contains sensitive credentials.

The framework uses a local_settings.py file to manage all configuration and credentials. This approach:

Keeps sensitive data out of version control
Uses environment variables for CI/CD environments (Jenkins, GitHub Actions, etc.)
Provides a consistent configuration pattern across all integrations

Setting up local_settings.py:

# Copy the example file
cp local_settings.py.example local_settings.py

# Edit with your credentials
# Add your API keys, tokens, and configuration values

The test runner validates that all required settings are present based on the flags you use. If any required configuration is missing, the framework will exit with an error message indicating which settings need to be added.

Command line arguments

Test execution can be customized in many ways and integrations can also be enabled or disabled through command line arguments. These options can be activated by the short flag or the full command name.

Platform -p or --platform (REQUIRED)

Options: ios, android, android_web, ios_web, chrome, firefox

Defines the target platform for test execution.

Examples:

# Mobile native apps
python test_runner.py --platform=android
python test_runner.py --platform=ios

# Mobile web browsers
python test_runner.py --platform=android_web
python test_runner.py --platform=ios_web

# Desktop browsers
python test_runner.py --platform=chrome
python test_runner.py --platform=firefox

Test classes or test methods should be decorated with platform-specific decorators:

@unittest.skipUnless(TestRunnerUtils.running_on_android(),
                   TestRunnerUtils.running_on_android_message)
def test_offline_visit(self):
    # Test implementation

Single test -t or --test

Default: None (runs all tests)

Execute a specific test case instead of running the entire test suite. Multiple formats are supported:

Examples:
# Run all tests in a class (simplest format)
python test_runner.py -p chrome -t TestHomepage

# Run a specific test method
python test_runner.py -p chrome -t TestHomepage.test_login

# Explicit module path (if needed)
python test_runner.py -p chrome -t test_homepage.TestHomepage.test_login
The framework automatically converts class names to module names (e.g., TestHomepage → test_homepage.TestHomepage).

Slack integration -s or --slack

Default: False

Enables Slack notifications for test results. Posts test execution summaries to a configured Slack channel.

Required configuration in local_settings.py:
SLACK = {
    'SLACK_AUTH_TOKEN': os.getenv('SLACK_TOKEN', None),
    'CHANNEL_ID': 'C123456789',
}

PROJECT_CONFIG = {
    'PROJECT_NAME': 'My Project',
    'RELEASE': '1.0.0',
    'ENVIRONMENT': 'QA',
}
Example:
python test_runner.py -p android -s

TestRail integration -r or --testrail

Default: False

Enables TestRail integration to create test runs and report results automatically.

Required configuration in local_settings.py:

TESTRAIL = {
    'BASE_URL': 'https://yourcompany.testrail.io',
    'USERNAME': os.getenv('TESTRAIL_USERNAME', None),
    'API_KEY': os.getenv('TESTRAIL_API_KEY', None),
    'PROJECT_ID': 1,
    'SUITE_ID': 1,
}

Example:

python test_runner.py -p android -r

SauceLabs integration -l or --saucelabs

Default: False

Executes tests on SauceLabs cloud platform using virtual or real devices.

Required configuration in local_settings.py:

SAUCELABS = {
    'USERNAME': os.getenv('SAUCELABS_USERNAME', None),
    'ACCESS_KEY': os.getenv('SAUCELABS_ACCESS_KEY', None),
    'URL': 'https://{}:{}@ondemand.us-west-1.saucelabs.com:443/wd/hub',
}

# Also requires platform-specific capabilities with _saucelabs suffix
SELENIUM = {
    'CAPABILITIES': {
        'android_saucelabs': {...},
        'ios_saucelabs': {...},
    }
}

Example:

python test_runner.py -p android -l

Report on fail -f or --report-on-fail
Default: False

By default, reports are only sent on successful test runs. This flag enables reporting regardless of test outcome. Automatically activates Slack integration.

Example:
python test_runner.py -p android -s -f
Note

Without this flag, Slack reports are only sent when all tests pass. Use -f to receive notifications for both passing and failing test runs.

Managed drivers -d or --managed
Default: False

Enables automated driver management for browser-based testing. The framework will automatically download and manage WebDriver binaries.

Example:
python test_runner.py -p chrome -d

Headless mode --headless
Default: False

Runs the browser in headless mode (no visible UI). Useful for CI/CD pipelines where no display is available.

Example:
python test_runner.py -p chrome --headless

Environment selection --env

Default: Value of PROJECT_CONFIG['ENVIRONMENT'] in settings.py (typically STAGING)

Selects which environment configuration to use from settings.ENVIRONMENTS. The value is case-insensitive and will be uppercased automatically. The framework validates that the specified environment key exists in settings.ENVIRONMENTS and exits with an error listing available environments if it does not.

Required configuration in settings.py:
ENVIRONMENTS = {
    'STAGING': {
        'BASE_URL': 'https://staging.example.com/',
        'USERNAME': os.getenv('STAGING_USERNAME'),
        'PASSWORD': os.getenv('STAGING_PASSWORD'),
    },
    'PRODUCTION': {
        'BASE_URL': 'https://production.example.com/',
        'USERNAME': os.getenv('PROD_USERNAME'),
        'PASSWORD': os.getenv('PROD_PASSWORD'),
    }
}
Tests access environment data dynamically via config.CURRENT_ENVIRONMENT:
import settings
import qlty.config as config

class MyTestClass(QLTYTestCase):
    base_url = settings.ENVIRONMENTS[config.CURRENT_ENVIRONMENT]['BASE_URL']
Examples:
# Uses default environment (from PROJECT_CONFIG['ENVIRONMENT'])
python test_runner.py -p chrome -t TestHomepage

# Explicitly target staging
python test_runner.py -p chrome -t TestHomepage --env staging

# Target production
python test_runner.py -p chrome -t TestProductionDeployments --env production

Exclude tests --exclude
Default: None (no tests excluded)

Excludes specific test classes from execution by name. Accepts a comma-separated list of class names. Useful for running the full test suite while skipping certain tests (e.g., production-only tests when running against staging).

The filter applies to both full suite discovery and single test execution. It matches against the test class name exactly (case-sensitive).

Examples:
# Run all tests except one class python test_runner.py -p chrome --exclude TestDynamicRegistration # Exclude multiple test classes python test_runner.py -p chrome --exclude TestDynamicRegistration,TestProductionDeployments # Combine with other flags for a full staging run python test_runner.py -p chrome --env staging -r -s -f --exclude TestDynamicRegistration

Tag filter --tag

Default: None

Run only test classes decorated with the specified tag. Use the @tag() decorator on test classes to assign tags:
from qlty.classes.core.qlty_testcase import QLTYTestCase, tag

@tag('production')
class TestProductionFeature(QLTYTestCase):
    ...

@tag('smoke', 'fast')
class TestQuickChecks(QLTYTestCase):
    ...
When --tag is used, only classes whose tags contain the specified value are included. This bypasses DEFAULT_EXCLUDE_TAGS.

Only applies to full suite runs — using -t to run a specific test bypasses all tag filtering.

Examples:
# Run only production-tagged tests
python test_runner.py -p chrome --tag production --env production

# Run only smoke tests
python test_runner.py -p chrome --tag smoke

Exclude tag --exclude-tag
Default: None

Exclude test classes decorated with the specified tag. The inverse of --tag.

Example:
# Run all tests except those tagged 'slow' python test_runner.py -p chrome --exclude-tag slow

Default exclude tags PROJECT_CONFIG['DEFAULT_EXCLUDE_TAGS']
Tags listed in settings.PROJECT_CONFIG['DEFAULT_EXCLUDE_TAGS'] are automatically excluded from full suite runs when neither --tag nor --exclude-tag is specified. This avoids needing --exclude for environment-specific tests.

Configuration in settings.py:
PROJECT_CONFIG = { 'DEFAULT_EXCLUDE_TAGS': ['production'], # Auto-excluded from default runs }
With this configuration, @tag('production') classes are skipped during default runs but included when using --tag production.

Combining flags:

Multiple flags can be combined to enable different features:

# Run on Android with Slack and TestRail integrations
python test_runner.py -p android -s -r

# Run specific test on Chrome with Slack notifications
python test_runner.py -p chrome -t tests.web.test_login.TestLogin.test_valid_credentials -s

# Run on SauceLabs with all integrations and report on failure
python test_runner.py -p ios -l -s -r -f

ChromeDriver Management

The framework includes a utility to automatically download the correct ChromeDriver version matching your installed Chrome browser. This is useful when Chrome auto-updates and you need a matching ChromeDriver.

# Check Chrome and ChromeDriver versions
./scripts/fetch_chromedriver.sh --check

# Download matching ChromeDriver to ./drivers
./scripts/fetch_chromedriver.sh

# Download to custom directory
./scripts/fetch_chromedriver.sh --output /path/to/drivers

The --chromedriver flag can also be passed during installation to fetch ChromeDriver automatically:

curl -fsSL https://raw.githubusercontent.com/qltyautomation/QLTYFramework/main/install.sh | bash -s -- --repo https://bitbucket.org/your-org/your-tests.git --chromedriver