Pytest Markers
idf-ci registers pytest markers on top of the ones provided by pytest-embedded.
disable_test
disable_test disables a test case by an esp-bool-parser expression, by targets, by sdkconfig names, or by any combination of them.
@pytest.mark.disable_test(
'IDF_TARGET == "esp32" and IDF_VERSION_MAJOR < 6',
configs=['iram'],
reason='IDF-1234, will fix it ASAP',
)
@pytest.mark.parametrize('target', ['esp32', 'esp32c3'], indirect=True)
def test_foo(dut):
pass
The positional argument is the boolean expression. It is evaluated by esp-bool-parser with the target and the sdkconfig name of the test case.
targetsandconfigsaccept a string or a list of strings.reasonis optional, but strongly recommended. It is printed when the test case is deselected and shown in the reports generated byidf-ci build collect.
At least one of the expression, targets or configs must be given, otherwise the collection fails. All the given conditions must match for the test case to be disabled.
Marker scope
Only the closest disable_test marker is taken into account. The lookup order is test function, then class, then module-level pytestmark, then pytestmark declared in conftest.py. The closest marker replaces the outer ones, the conditions are never merged:
import pytest
# applies to every test case in this file
pytestmark = [
pytest.mark.disable_test(targets=['esp32'], reason='IDF-1234'),
]
# disabled on esp32, inherited from `pytestmark`
def test_foo(dut):
pass
# the function-level marker replaces the file-level one entirely:
# disabled on esp32c3, and *not* disabled on esp32 anymore
@pytest.mark.disable_test(targets=['esp32c3'], reason='IDF-5678')
def test_bar(dut):
pass
# to keep the file-level condition as well, repeat it explicitly
@pytest.mark.disable_test(targets=['esp32', 'esp32c3'], reason='IDF-5678')
def test_baz(dut):
pass
Matching multi-DUT test cases
A test case is disabled as a whole: a multi-DUT test case cannot run partially.
A plain value matches when any DUT uses it, and the expression matches when it evaluates to true for any DUT:
# apps: esp32/default and esp32c3/default
@pytest.mark.parametrize('count', [2], indirect=True)
@pytest.mark.parametrize('target', ['esp32|esp32c3'], indirect=True)
#
# disabled, the second DUT is esp32c3
@pytest.mark.disable_test(targets=['esp32c3'], reason='IDF-1234')
#
# disabled as well, the expression is true for the second DUT
@pytest.mark.disable_test('IDF_TARGET == "esp32c3"', reason='IDF-1234')
#
# not disabled, no DUT uses esp32s2
@pytest.mark.disable_test(targets=['esp32s2'], reason='IDF-1234')
def test_foo(dut):
pass
A value that contains a comma is a selector, matched against the whole test case. * matches any single DUT. A selector only matches a test case with the same number of DUTs, which is what makes it possible to disable one DUT combination while keeping the others:
@pytest.mark.parametrize('count', [2], indirect=True)
@pytest.mark.parametrize('target', ['esp32|esp32c3', 'esp32|esp32'], indirect=True)
# disables the esp32+esp32c3 pair only, the esp32+esp32 pair still runs
@pytest.mark.disable_test(targets=['esp32,esp32c3'], reason='IDF-1234')
def test_foo(dut):
pass
# disables every two-DUT case whose second DUT is esp32c3
@pytest.mark.disable_test(targets=['*,esp32c3'], reason='IDF-1234')
def test_bar(dut):
pass
# disables every two-DUT case whose first DUT is esp32
@pytest.mark.disable_test(targets=['esp32,*'], reason='IDF-1234')
def test_baz(dut):
pass
configs supports the same syntax, so configs=['*,iram'] matches a two-DUT case whose second app is built with the iram sdkconfig.
Disabling only in CI
esp-bool-parser can resolve environment variables, so any CI variable can be used in the expression. An unset variable evaluates to the integer 0, so always compare against the value you expect:
# disabled in GitLab CI only
@pytest.mark.disable_test('CI_JOB_ID != 0', reason='IDF-1234')
def test_gitlab(dut):
pass
# disabled in GitHub Actions only
@pytest.mark.disable_test('CI == "true"', reason='IDF-1234')
def test_github(dut):
pass
Migrating from temp_skip and temp_skip_ci
The ESP-IDF used custom markers temp_skip and temp_skip_ci to temporarily skip tests. Now, the recommended approach is to use disable_test instead.
Deprecated |
Replacement |
|---|---|
|
|
|
|
|
|
On top of the old behavior, disable_test also supports filtering by sdkconfig name and by boolean expressions.
Multiple disable_test markers
ESP-IDF allowed several temp_skip / temp_skip_ci markers on the same test, and all of them were taken into account. disable_test does not work this way: only the closest marker is used, so the others are silently ignored. Combine them into a single marker instead.
# does NOT work, only the last marker (the closest one) is applied
@pytest.mark.disable_test(targets=['esp32'], reason='IDF-1234')
@pytest.mark.disable_test(targets=['esp32c3'], reason='IDF-5678')
def test_foo(dut):
pass
# works, targets are merged into one marker
@pytest.mark.disable_test(targets=['esp32', 'esp32c3'], reason='IDF-1234, IDF-5678')
def test_foo(dut):
pass
If the markers you want to merge have different conditions, join them with the boolean operators supported by esp-bool-parser:
# was: temp_skip(targets=['esp32']) + temp_skip_ci(targets=['esp32c3'])
@pytest.mark.disable_test(
'IDF_TARGET == "esp32" or (IDF_TARGET == "esp32c3" and CI_JOB_ID != 0)',
reason='IDF-1234, IDF-5678',
)
def test_foo(dut):
pass