Usage

pytest-fluent-logging forwards meta data from pytest to Fluentd for further processing. The meta data are

  • unique session ID

  • unique test ID

  • status of the session respectively test case

  • test case name

  • test results

  • record_property entries

  • custom testcase information

  • custom session information

  • timestamps

Furthermore, the Python logging instance can be extended in order to forward test case runtime logging:

from logging import getLogger

def test_my_runtime_log():
    value = 1
    getLogger().info("Setting value to %s", value)
    assert value == 1

or:

from logging import getLogger

def test_my_runtime_log():
    value = 1
    getLogger('fluent').info("Setting value to %s", value)
    assert value == 1

Fixtures

In order to create your own logger, request the fixture get_logger as follows:

def test_my_runtime_log(get_logger):
    logger = get_logger('my.Logger')
    value = 1
    logger.info("Setting value to %s", value)
    assert value == 1

If you want to get the current UIDs, use the session_uid and test_uid fixtures as follows:

def test_unique_identifier(get_logger, session_uid, test_uid):
    logger = get_logger('fluent')
    logger.info("Session ID: %s", session_uid)
    logger.info("Test ID: %s", test_uid)
    value = 1
    assert value == 1

Callbacks

If you want to add custom data to the datasets of the pytest_sessionstart and pytest_runtest_logstart stages, decorate your callback functions with the following decorators:

from pytest_fluent import (
    additional_session_information_callback,
    additional_test_information_callback
)

@additional_session_information_callback
def provide_more_session_information() -> dict:
    return {
        "more": "session information"
    }

@additional_test_information_callback
def provide_more_test_information() -> dict:
    return {
        "more": "test information"
    }

pytest CLI extensions

The pytest CLI application can be called with the following arguments in order to configure fluent-logging.

argument

description

default

–session-uuid

Use a custom externally created UUID, e.g. link a CI job with the pytest session.

–fluentd-host

Fluentd host address. If not provided, a local Fluentd instance will be called.

–fluentd-port

Fluent host port

24224

–fluentd-tag

Set a custom Fluentd tag

‘test’

–fluentd-label

Set a custom Fluentd label

‘pytest’

–fluentd-timestamp

Specify a Fluentd timestamp

None

–extend-logging

Extend the Python logging with a Fluent handler

False

–add-docstrings

Add test docstrings to testcase call messages

–stage-settings

Use custom stage settings file. See documentation

Ini Configuration Support

Default values of the CLI arguments for a project could also be defined in one of the following ini configuration files:

  1. pytest.ini: Arguments are defined under the pytest section in the file. This file takes precedence over all other configuration files even if it’s empty.

[pytest]
addopts= --session-uuid="ac2f7600-a079-46cf-a7e0-6408b166364c" --fluentd-port=24224  --fluentd-host=localhost --fluentd-tag='dummytest' --fluentd-label='pytest' --extend-logging

  1. pyproject.toml: are considered for configuration when they contain a tool.pytest.ini_options section is available

[tool.pytest.ini_options]
addopts="--fluentd-port=24224 --fluentd-host=localhost --fluentd-tag='test' --fluentd-label='pytest' --extend-logging"

  1. tox.ini: can also be used to hold pytest configuration if they have a [pytest] section.

[pytest]
addopts= --fluentd-port=24224 --fluentd-host=localhost --fluentd-tag='test' --fluentd-label='pytest'

If the same option is specified in both CLI and ini file, then CLI option would have higher priority and override the ini file values.

What data are sent?

pytest-fluent sends any information, e.g. stage information or logging from a test case, as a single chunk. For instance, the data collection from test_addoptions.py test looks as following

[
    {
        "status": "start",
        "stage": "session",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0"
    },
    {
        "status": "start",
        "stage": "testcase",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "testId": "6b444275-4450-4eff-b5d9-8355f0f99ab0",
        "name": "test_fluentd_logged_parameters.py::test_base"
    },
    {
        "type": "logging",
        "host": "myComputer",
        "where": "test_fluentd_logged_parameters.test_base",
        "level": "INFO",
        "stack_trace": "None",
        "message": "Logged from test_base",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "testId": "6b444275-4450-4eff-b5d9-8355f0f99ab0",
        "stage": "testcase"
    },
    {
        "type": "logging",
        "host": "myComputer",
        "where": "test_fluentd_logged_parameters.test_base",
        "level": "INFO",
        "stack_trace": "None",
        "message": "Logged from test_base",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "testId": "6b444275-4450-4eff-b5d9-8355f0f99ab0",
        "stage": "testcase"
    },
    {
        "name": "test_fluentd_logged_parameters.py::test_base",
        "outcome": "passed",
        "duration": 0.0013457999999999526,
        "markers": {
            "test_base": 1,
            "test_fluentd_logged_parameters.py": 1,
            "test_fluentd_logged_parameters0": 1
        },
        "stage": "testcase",
        "when": "call",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "testId": "6b444275-4450-4eff-b5d9-8355f0f99ab0"
    },
    {
        "status": "finish",
        "stage": "testcase",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "testId": "6b444275-4450-4eff-b5d9-8355f0f99ab0",
        "name": "test_fluentd_logged_parameters.py::test_base"
    },
    {
        "status": "finish",
        "duration": 0.00297708511352539,
        "stage": "session",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0"
    }
]

whereat each object in the array is sent independently via Fluentd.

Specifying a timestamp

Timestamps are added to the information if the --fluentd-timestamp option is enabled:

[pytest]
addopts= --session-uuid="ac2f7600-a079-46cf-a7e0-6408b166364c" --fluentd-port=24224  --fluentd-host=localhost --fluentd-tag='dummytest' --fluentd-label='pytest' --fluentd-timestamp='@timestamp' --extend-logging

The timestamp is added to each message. The value is in ISO 8601 format. A sample of the data collection from test_addoptions.py (as above) would look as below:

[
    {
        "status": "start",
        "stage": "session",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "@timestamp": "2022-12-25T03:00:00.000000Z"
    },
    {
        "status": "start",
        "stage": "testcase",
        "sessionId": "d8f01de3-8416-4801-9406-0ea3d5cfe3c0",
        "testId": "6b444275-4450-4eff-b5d9-8355f0f99ab0",
        "name": "test_fluentd_logged_parameters.py::test_base",
        "@timestamp": "2022-12-25T03:00:00.100000Z"
    }
]

Custom stage settings

Sometimes, the default settings are not enough in order to forward the test information as needed. Therefore, you can set custom stage settings in order to fit your needs.

You can set specific values for all stages or specific values for any used stage. In order to do so, call your test run with the --stage-settings=YourFileName.json parameter. The following example stage settings JSON file content:

{
    "all": {
        "tag": "run",
        "label": "pytest",
        "replace": {"keys": {"status": "state", "sessionId": "id"}},
    },
    "pytest_sessionstart": {
        "tag": "run",
        "label": "test",
        "add": {"start_info": "Pytest started"},
    },
    "pytest_sessionfinish": {
        "tag": "result",
        "label": "test",
        "add": {"stop_info": "Pytest finished"},
    },
    "pytest_runtest_logstart": {
        "tag": "run",
        "label": "testcase",
        "add": {"start_info": "Testcase started"},
    },
    "pytest_runtest_logreport": {
        "tag": "result",
        "label": "testcase",
        "replace": {
            "values": {"passed": "pass", "failed": "fail"},
        },
        "add": {"stop_info": "Testcase finished"},
    },
    "logging": {
        "replace": {"keys": {"message": "msg", "sessionId": "id"}},
    },
}

will result in the following output:

[
  {
    "stage": "session",
    "tag": "test",
    "label": "pytest",
    "state": "start",
    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
  },
  {
    "stage": "testcase",
    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
    "name": "test_data_reporter_with_patched_values.py::test_base",
    "tag": "test",
    "label": "pytest",
    "state": "start",
    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
  },
  {
    "type": "logging",
    "host": "hostname",
    "where": "test_data_reporter_with_patched_values.test_base",
    "level": "INFO",
    "stack_trace": "None",
    "message": "Test running",
    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
    "stage": "testcase",
    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
  },
  {
    "name": "test_data_reporter_with_patched_values.py::test_base",
    "outcome": "pass",
    "duration": 0.0034263000000001043,
    "markers": {
      "test_base": 1,
      "test_data_reporter_with_patched_values.py": 1,
      "test_data_reporter_with_patched_values0": 1
    },
    "stage": "testcase",
    "when": "call",
    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
    "tag": "test",
    "label": "pytest",
    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8",
    "stop_info": "Testcase finished"
  },
  {
    "stage": "testcase",
    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
    "name": "test_data_reporter_with_patched_values.py::test_base",
    "tag": "test",
    "label": "pytest",
    "state": "finish",
    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
  },
  {
    "duration": 1.3674933910369873,
    "stage": "session",
    "tag": "test",
    "label": "pytest",
    "state": "finish",
    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
  }
]

for this test case:

import logging

def test_base():
    logger = logging.getLogger()
    logger.info("Test running")
    assert True

Stage setting file

Custom settings for each supported stage can be easily setup. You have to create a file with a .json or .yaml extension and call pytest with this additional parameter --stage-settings.

The file will be validated against a schema of supported values and in case of an error, a jsonschema.ValidationError will be thrown.

Stage settings

Number of supported stage

The following stages can be modified.

  • pytest_sessionstart

  • pytest_runtest_logstart

  • pytest_runtest_logreport

  • pytest_runtest_logfinish

  • pytest_sessionfinish

  • logging

These values are the keys for the dictionary object. Additionally, you can set also a all key for convenience reasons to patch all keys at once.

Patch events

Probably, your stage setting would look like

{
    "pytest_sessionstart": {
        "tag": "run",
        "label": "pytest",
        "replace": {
            "keys": {
                "status": "state",
                "sessionId": "id"
            },
            "values": {
                "passed": "pass"
            }
        },
        "add": {
          "start_info": "Pytest started"
        },
    }
}

The following values are supported:

Key name

action

type

tag

Set a specifc Fluent tag for this stage

str

label

Set a specifc Fluent label for this stage

str

replace

Replace key values from a dictionary and also replace some preset pytest result values

Dict[str, str]

add

Add new values to the result dictionary

Dict[str, str]

drop

Drop specific values from the result dictionary

List[str]

Suppressing stage forwarding

If you want that forwarding of a specific stage is suppressed, just set an empty string as tag.

For instance if you want just a single stage being forwarded, see the following example

{
    "all": {
        "tag": ""
    },
    "pytest_runtest_logreport": {
        "tag": "run",
        "label": "pytest"
    }
}

Replace dictionary

The replace patching action has two keys keys and values in order to replace either a key value or a result value. See the following default values in order to get an idea about the content.

At the moment, the following values can be changed

  • passed

  • failed

  • skipped

  • error

  • start

  • finish

  • session

  • testcase

Use values from ARGV and ENV

If you want to use data provided by the command line arguments or directly from environment variables, use the following syntax for value strings.

Type

Syntax

ARGV

"<fluentd-tag>"

ENV

"${USE_ENV}" or "$USE_ENV"

Here is a simple example using both variants

{
    "pytest_sessionstart": {
        "tag": "run",
        "label": "pytest",
        "replace": {
            "keys": {
                "tag": "<fluentd-tag>",
                "sessionId": "${ID}"
            },
            "values": {
                "passed": "$OUTCOME_PASSED"
            }
        }
    }
}

The data will be mapped after starting the pytest session.

Default values

stage

value

pytest_sessionstart

 

{
    “status”: “start”,
    “stage”: “session”,
    “sessionId”: “8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf”,
    “tag”: “test”,
    “label”: “pytest”
}

pytest_runtest_logstart

 

{
    “status”: “start”,
    “stage”: “testcase”,
    “sessionId”: “8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf”,
    “testId”: “9f0363fa-ef99-49c7-8a2d-6261e90acb00”,
    “name”: “test_data_reporter_with_patched_values.py::test_base”,
    “tag”: “test”,
    “label”: “pytest”
}

pytest_runtest_logreport

 

{
    “name”: “test_data_reporter_with_patched_values.py::test_base”,
    “outcome”: “passed”,
    “duration”: 0.0035069000000005346,
    “markers”: {
      “test_base”: 1,
      “test_data_reporter_with_patched_values.py”: 1,
      “test_data_reporter_with_patched_values0”: 1
    },
    “stage”: “testcase”,
    “when”: “call”,
    “sessionId”: “8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf”,
    “testId”: “9f0363fa-ef99-49c7-8a2d-6261e90acb00”,
    “tag”: “test”,
    “label”: “pytest”
}

pytest_runtest_logfinish

 

{
    “status”: “finish”,
    “stage”: “testcase”,
    “sessionId”: “8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf”,
    “testId”: “9f0363fa-ef99-49c7-8a2d-6261e90acb00”,
    “name”: “test_data_reporter_with_patched_values.py::test_base”,
    “tag”: “test”,
    “label”: “pytest”
}

pytest_sessionfinish

 

{
    “status”: “finish”,
    “duration”: 1.5651893615722656,
    “stage”: “session”,
    “sessionId”: “8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf”,
    “tag”: “test”,
    “label”: “pytest”
}

logging

 

{
    “type”: “logging”,
    “host”: “hostname”,
    “where”: “test_data_reporter_with_patched_values.test_base”,
    “level”: “INFO”,
    “stack_trace”: “None”,
    “message”: “Test running”,
    “sessionId”: “8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf”,
    “testId”: “9f0363fa-ef99-49c7-8a2d-6261e90acb00”,
    “stage”: “testcase”
}