################ Pytest Markers ################ idf-ci registers pytest markers on top of the ones provided by pytest-embedded_. ****************** ``disable_test`` ****************** ``disable_test`` disables a test case by an esp-bool-parser_ expression, by targets, by sdkconfig names, or by any combination of them. .. code-block:: python @pytest.mark.disable_test( 'IDF_TARGET == "esp32" and IDF_VERSION_MAJOR < 6', configs=['iram'], reason='IDF-1234, will fix it ASAP', ) @pytest.mark.parametrize('target', ['esp32', 'esp32c3'], indirect=True) def test_foo(dut): pass - The positional argument is the boolean expression. It is evaluated by esp-bool-parser_ with the target and the sdkconfig name of the test case. - ``targets`` and ``configs`` accept a string or a list of strings. - ``reason`` is optional, but strongly recommended. It is printed when the test case is deselected and shown in the reports generated by ``idf-ci build collect``. At least one of the expression, ``targets`` or ``configs`` must be given, otherwise the collection fails. All the given conditions must match for the test case to be disabled. Marker scope ============ Only the **closest** ``disable_test`` marker is taken into account. The lookup order is test function, then class, then module-level ``pytestmark``, then ``pytestmark`` declared in ``conftest.py``. The closest marker **replaces** the outer ones, the conditions are never merged: .. code-block:: python import pytest # applies to every test case in this file pytestmark = [ pytest.mark.disable_test(targets=['esp32'], reason='IDF-1234'), ] # disabled on esp32, inherited from `pytestmark` def test_foo(dut): pass # the function-level marker replaces the file-level one entirely: # disabled on esp32c3, and *not* disabled on esp32 anymore @pytest.mark.disable_test(targets=['esp32c3'], reason='IDF-5678') def test_bar(dut): pass # to keep the file-level condition as well, repeat it explicitly @pytest.mark.disable_test(targets=['esp32', 'esp32c3'], reason='IDF-5678') def test_baz(dut): pass Matching multi-DUT test cases ============================= A test case is disabled as a whole: a multi-DUT test case cannot run partially. A plain value matches when **any** DUT uses it, and the expression matches when it evaluates to true for **any** DUT: .. code-block:: python # apps: esp32/default and esp32c3/default @pytest.mark.parametrize('count', [2], indirect=True) @pytest.mark.parametrize('target', ['esp32|esp32c3'], indirect=True) # # disabled, the second DUT is esp32c3 @pytest.mark.disable_test(targets=['esp32c3'], reason='IDF-1234') # # disabled as well, the expression is true for the second DUT @pytest.mark.disable_test('IDF_TARGET == "esp32c3"', reason='IDF-1234') # # not disabled, no DUT uses esp32s2 @pytest.mark.disable_test(targets=['esp32s2'], reason='IDF-1234') def test_foo(dut): pass A value that contains a comma is a **selector**, matched against the whole test case. ``*`` matches any single DUT. A selector only matches a test case with the same number of DUTs, which is what makes it possible to disable one DUT combination while keeping the others: .. code-block:: python @pytest.mark.parametrize('count', [2], indirect=True) @pytest.mark.parametrize('target', ['esp32|esp32c3', 'esp32|esp32'], indirect=True) # disables the esp32+esp32c3 pair only, the esp32+esp32 pair still runs @pytest.mark.disable_test(targets=['esp32,esp32c3'], reason='IDF-1234') def test_foo(dut): pass # disables every two-DUT case whose second DUT is esp32c3 @pytest.mark.disable_test(targets=['*,esp32c3'], reason='IDF-1234') def test_bar(dut): pass # disables every two-DUT case whose first DUT is esp32 @pytest.mark.disable_test(targets=['esp32,*'], reason='IDF-1234') def test_baz(dut): pass ``configs`` supports the same syntax, so ``configs=['*,iram']`` matches a two-DUT case whose second app is built with the ``iram`` sdkconfig. Disabling only in CI ==================== esp-bool-parser_ can resolve environment variables, so any CI variable can be used in the expression. An **unset** variable evaluates to the integer ``0``, so always compare against the value you expect: .. code-block:: python # disabled in GitLab CI only @pytest.mark.disable_test('CI_JOB_ID != 0', reason='IDF-1234') def test_gitlab(dut): pass # disabled in GitHub Actions only @pytest.mark.disable_test('CI == "true"', reason='IDF-1234') def test_github(dut): pass Migrating from ``temp_skip`` and ``temp_skip_ci`` ================================================= The ESP-IDF used custom markers ``temp_skip`` and ``temp_skip_ci`` to temporarily skip tests. Now, the recommended approach is to use ``disable_test`` instead. .. list-table:: :header-rows: 1 - - Deprecated - Replacement - - ``@pytest.mark.temp_skip(targets=['esp32'], reason='IDF-1234')`` - ``@pytest.mark.disable_test(targets=['esp32'], reason='IDF-1234')`` - - ``@pytest.mark.temp_skip_ci(targets=['esp32'], reason='IDF-1234')`` - ``@pytest.mark.disable_test('CI_JOB_ID != 0', targets=['esp32'], reason='IDF-1234')`` - - ``@pytest.mark.temp_skip(targets=['esp32,*'], reason='IDF-1234')`` - ``@pytest.mark.disable_test(targets=['esp32,*'], reason='IDF-1234')`` On top of the old behavior, ``disable_test`` also supports filtering by sdkconfig name and by boolean expressions. Multiple ``disable_test`` markers --------------------------------- ESP-IDF allowed several ``temp_skip`` / ``temp_skip_ci`` markers on the same test, and all of them were taken into account. ``disable_test`` does not work this way: only the closest marker is used, so the others are silently ignored. Combine them into a single marker instead. .. code-block:: python # does NOT work, only the last marker (the closest one) is applied @pytest.mark.disable_test(targets=['esp32'], reason='IDF-1234') @pytest.mark.disable_test(targets=['esp32c3'], reason='IDF-5678') def test_foo(dut): pass # works, targets are merged into one marker @pytest.mark.disable_test(targets=['esp32', 'esp32c3'], reason='IDF-1234, IDF-5678') def test_foo(dut): pass If the markers you want to merge have different conditions, join them with the boolean operators supported by esp-bool-parser_: .. code-block:: python # was: temp_skip(targets=['esp32']) + temp_skip_ci(targets=['esp32c3']) @pytest.mark.disable_test( 'IDF_TARGET == "esp32" or (IDF_TARGET == "esp32c3" and CI_JOB_ID != 0)', reason='IDF-1234, IDF-5678', ) def test_foo(dut): pass .. _esp-bool-parser: https://github.com/espressif/esp-bool-parser .. _pytest-embedded: https://github.com/espressif/pytest-embedded