idf_ci package

Subpackages

Submodules

class idf_ci.scripts.ProcessedArgs(modified_files: List[str] | None, modified_components: List[str] | None, filter_expr: str | None, default_build_targets: List[str], test_related_apps: List[App] | None, non_test_related_apps: List[App] | None)

Bases: object

Container for processed arguments with meaningful field names.

default_build_targets: List[str]

filter_expr: str | None

modified_components: List[str] | None

modified_files: List[str] | None

non_test_related_apps: List[App] | None

test_related_apps: List[App] | None

idf_ci.scripts.build(*, paths: List[str] | None = None, target: str = 'all', parallel_count: int = 1, parallel_index: int = 1, modified_files: List[str] | None = None, modified_components: List[str] | None = None, only_test_related: bool | None = None, only_non_test_related: bool | None = None, dry_run: bool = False, build_system: UndefinedType | str = __UNDEF__, marker_expr: UndefinedType | str = __UNDEF__, filter_expr: str | None = None) → Tuple[List[App], int]

Build applications based on specified parameters.

Parameters:

paths – List of paths to search for applications
target – Target device(s) separated by commas
parallel_count – Total number of parallel jobs
parallel_index – Index of current parallel job (1-based)
modified_files – List of modified files
modified_components – List of modified components
only_test_related – Only build test-related applications
only_non_test_related – Only build non-test-related applications
dry_run – Do not actually build, just simulate
build_system – Filter the apps by build system. Can be “cmake”, “make” or a custom App class path
marker_expr – Pytest marker expression
filter_expr – Filter expression

Returns:

Tuple of (built apps, build return code)

Get test-related and non-test-related applications.

Parameters:

paths – List of paths to search for applications
target – Target device(s) separated by commas
modified_files – List of modified files
modified_components – List of modified components
filter_expr – Pytest filter expression -k
default_build_targets – Default build targets to use
marker_expr – Pytest marker expression -m
compare_manifest_sha_filepath – Path to the manifest SHA file generated by idf-build-apps dump-manifest-sha
build_system – Filter the apps by build system. Can be “cmake”, “make” or a custom App class path

Returns:

Tuple of (test_related_apps, non_test_related_apps)

idf_ci.scripts.preprocess_args(modified_files: List[str] | None = None, modified_components: List[str] | None = None, filter_expr: str | None = None, default_build_targets: List[str] | None = None) → ProcessedArgs

Set values according to the environment variables, .toml settings, and defaults.

Parameters:

modified_files – List of modified files
modified_components – List of modified components
filter_expr – Pytest filter expression
default_build_targets – Default build targets to use

Returns:

Processed arguments as a ProcessedArgs object

idf_ci.utils.get_current_branch() → str

Get the current Git branch by running git command.

Returns:: The current Git ref (branch name or commit SHA)
Raises:: RuntimeError – If not in a Git repository or git command fails

idf_ci.utils.remove_subfolders(paths: List[str]) → List[str]

Remove paths that are subfolders of other paths in the list.

Parameters:: paths – List of directory paths as strings
Returns:: Filtered list of paths with no subfolder/parent folder relationships

idf_ci.utils.setup_logging(level: int | None = 20) → None

Setup logging

Parameters:: level – logging level

idf_ci.utils.to_list(s: None) → None

idf_ci.utils.to_list(s: Iterable[_T]) → List[_T]

idf_ci.utils.to_list(s: _T) → List[_T]

Turn all objects to lists

Parameters:

s – anything

Returns:

None, if s is None
itself, if s is a list
list(s), if s is a tuple or a set
[s], if s is other type

Module contents

pydantic settings idf_ci.CiSettings

Bases: BaseSettings

Show JSON schema

{
   "title": "CiSettings",
   "type": "object",
   "properties": {
      "component_mapping_regexes": {
         "default": [
            "/components/(.+)/",
            "/common_components/(.+)/"
         ],
         "items": {
            "type": "string"
         },
         "title": "Component Mapping Regexes",
         "type": "array"
      },
      "extend_component_mapping_regexes": {
         "default": [],
         "items": {
            "type": "string"
         },
         "title": "Extend Component Mapping Regexes",
         "type": "array"
      },
      "component_ignored_file_extensions": {
         "default": [
            ".md",
            ".rst",
            ".yaml",
            ".yml",
            ".py"
         ],
         "items": {
            "type": "string"
         },
         "title": "Component Ignored File Extensions",
         "type": "array"
      },
      "extend_component_ignored_file_extensions": {
         "default": [],
         "items": {
            "type": "string"
         },
         "title": "Extend Component Ignored File Extensions",
         "type": "array"
      },
      "built_app_list_filepatterns": {
         "default": [
            "app_info_*.txt"
         ],
         "items": {
            "type": "string"
         },
         "title": "Built App List Filepatterns",
         "type": "array"
      },
      "collected_test_related_apps_filepath": {
         "default": "test_related_apps.txt",
         "title": "Collected Test Related Apps Filepath",
         "type": "string"
      },
      "collected_non_test_related_apps_filepath": {
         "default": "non_test_related_apps.txt",
         "title": "Collected Non Test Related Apps Filepath",
         "type": "string"
      },
      "preserve_test_related_apps": {
         "default": true,
         "title": "Preserve Test Related Apps",
         "type": "boolean"
      },
      "preserve_non_test_related_apps": {
         "default": true,
         "title": "Preserve Non Test Related Apps",
         "type": "boolean"
      },
      "extra_default_build_targets": {
         "default": [],
         "items": {
            "type": "string"
         },
         "title": "Extra Default Build Targets",
         "type": "array"
      },
      "ci_detection_envs": {
         "default": [
            "CI",
            "GITHUB_ACTIONS",
            "CIRCLECI",
            "TRAVIS",
            "JENKINS_URL",
            "DRONE",
            "APPVEYOR",
            "BITBUCKET_COMMIT",
            "SEMAPHORE",
            "TEAMCITY_VERSION"
         ],
         "items": {
            "type": "string"
         },
         "title": "Ci Detection Envs",
         "type": "array"
      },
      "local_runtime_envs": {
         "additionalProperties": true,
         "default": {},
         "title": "Local Runtime Envs",
         "type": "object"
      },
      "ci_runtime_envs": {
         "additionalProperties": true,
         "default": {},
         "title": "Ci Runtime Envs",
         "type": "object"
      },
      "gitlab": {
         "$ref": "#/$defs/GitlabSettings",
         "default": {
            "project": "espressif/esp-idf",
            "known_failure_cases_bucket_name": "ignore-test-result-files",
            "artifacts": {
               "build_job_filepatterns": [
                  "app_info_*.txt",
                  "build_summary_*.xml"
               ],
               "s3": {
                  "debug": {
                     "bucket": "idf-artifacts",
                     "patterns": [
                        "**/build*/bootloader/*.map",
                        "**/build*/bootloader/*.elf",
                        "**/build*/*.map",
                        "**/build*/*.elf",
                        "**/build*/build.log"
                     ]
                  },
                  "flash": {
                     "bucket": "idf-artifacts",
                     "patterns": [
                        "**/build*/bootloader/*.bin",
                        "**/build*/*.bin",
                        "**/build*/partition_table/*.bin",
                        "**/build*/flasher_args.json",
                        "**/build*/flash_project_args",
                        "**/build*/config/sdkconfig.json",
                        "**/build*/sdkconfig",
                        "**/build*/project_description.json"
                     ]
                  }
               },
               "test_job_filepatterns": [
                  "pytest-embedded/",
                  "XUNIT_RESULT*.xml"
               ]
            },
            "build_pipeline": {
               "job_tags": [
                  "build"
               ],
               "job_template_jinja": "{{ settings.gitlab.build_pipeline.job_template_name }}:\n  stage: build\n  tags: {{ settings.gitlab.build_pipeline.job_tags }}\n  timeout: 1h\n  artifacts:\n    paths:\n    {%- for path in settings.gitlab.artifacts.build_job_filepatterns %}\n      - {{ path }}\n    {%- endfor %}\n    expire_in: 1 week\n    when: always\n  script:\n    - idf-ci build run\n      --parallel-count ${CI_NODE_TOTAL:-1}\n      --parallel-index ${CI_NODE_INDEX:-1}",
               "job_template_name": ".default_build_settings",
               "jobs_jinja": "{%- if test_related_apps_count > 0 %}\nbuild_test_related_apps:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if test_related_parallel_count > 1 %}\n  parallel: {{ test_related_parallel_count }}\n{%- endif %}\n  needs:\n    - pipeline: $PARENT_PIPELINE_ID\n      job: generate_build_child_pipeline\n    - pipeline: $PARENT_PIPELINE_ID\n      job: pipeline_variables\n  variables:\n    IDF_CI_BUILD_ONLY_TEST_RELATED_APPS: \"1\"\n{%- endif %}\n\n{%- if non_test_related_apps_count > 0 %}\nbuild_non_test_related_apps:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if non_test_related_parallel_count > 1 %}\n  parallel: {{ non_test_related_parallel_count }}\n{%- endif %}\n  needs:\n    - pipeline: $PARENT_PIPELINE_ID\n      job: generate_build_child_pipeline\n    - pipeline: $PARENT_PIPELINE_ID\n      job: pipeline_variables\n  variables:\n    IDF_CI_BUILD_ONLY_NON_TEST_RELATED_APPS: \"1\"\n{%- endif %}",
               "pre_yaml_jinja": "",
               "presigned_json_job_name": "generate_presigned_json",
               "runs_per_job": 60,
               "workflow_name": "Build Child Pipeline",
               "yaml_filename": "build_child_pipeline.yml",
               "yaml_jinja": "{{ settings.gitlab.build_pipeline.pre_yaml_jinja }}\n\nworkflow:\n  name: {{ settings.gitlab.build_pipeline.workflow_name }}\n  rules:\n    - when: always\n\n{%- if settings.gitlab.build_pipeline.job_template_jinja %}\n{{ job_template }}\n{%- endif %}\n\n{{ jobs }}\n\n{%- if test_related_apps_count > 0 %}\ngenerate_test_child_pipeline:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n  needs:\n    - build_test_related_apps\n  artifacts:\n    paths:\n      - {{ settings.gitlab.test_pipeline.yaml_filename }}\n  script:\n    - idf-ci gitlab test-child-pipeline\n\ntest-child-pipeline:\n  stage: .post\n  needs:\n    - generate_test_child_pipeline\n  variables:\n    PARENT_PIPELINE_ID: $PARENT_PIPELINE_ID\n  trigger:\n    include:\n      - artifact: {{ settings.gitlab.test_pipeline.yaml_filename }}\n        job: generate_test_child_pipeline\n    strategy: depend\n{%- endif %}"
            },
            "test_pipeline": {
               "job_tags": [],
               "job_template_jinja": "{{ settings.gitlab.test_pipeline.job_template_name }}:\n  stage: test\n  timeout: 1h\n  artifacts:\n    paths:\n    {%- for path in settings.gitlab.artifacts.test_job_filepatterns %}\n      - {{ path }}\n    {%- endfor %}\n    expire_in: 1 week\n    when: always\n  variables:\n    PYTEST_EXTRA_FLAGS: \"\"\n  script:\n    - pytest ${nodes}\n      --parallel-count ${CI_NODE_TOTAL:-1}\n      --parallel-index ${CI_NODE_INDEX:-1}\n      --junitxml XUNIT_RESULT_${CI_JOB_NAME_SLUG}.xml\n      ${PYTEST_EXTRA_FLAGS}",
               "job_template_name": ".default_test_settings",
               "jobs_jinja": "{% for job in jobs %}\n{{ job['name'] }}:\n  extends:\n    - {{ settings.gitlab.test_pipeline.job_template_name }}\n    {%- for extra_extend in job.get('extra_extends', []) %}\n    - {{ extra_extend }}\n    {%- endfor %}\n  tags: {{ job['tags'] }}\n{%- if job['parallel_count'] > 1 %}\n  parallel: {{ job['parallel_count'] }}\n{%- endif %}\n  variables:\n    nodes: {{ job['nodes'] }}\n{% endfor %}",
               "pre_yaml_jinja": "",
               "presigned_json_job_name": "generate_presigned_json",
               "runs_per_job": 30,
               "workflow_name": "Test Child Pipeline",
               "yaml_filename": "test_child_pipeline.yml",
               "yaml_jinja": "{{ settings.gitlab.test_pipeline.pre_yaml_jinja }}\n\nworkflow:\n  name: {{ settings.gitlab.test_pipeline.workflow_name }}\n  rules:\n    - when: always\n\n{%- if settings.gitlab.test_pipeline.job_template_jinja %}\n{{ default_template }}\n{%- endif %}\n\n{{ jobs }}"
            }
         }
      }
   },
   "$defs": {
      "ArtifactSettings": {
         "additionalProperties": false,
         "properties": {
            "s3": {
               "additionalProperties": {
                  "$ref": "#/$defs/S3FilePatternConfig"
               },
               "default": {
                  "debug": {
                     "bucket": "idf-artifacts",
                     "patterns": [
                        "**/build*/bootloader/*.map",
                        "**/build*/bootloader/*.elf",
                        "**/build*/*.map",
                        "**/build*/*.elf",
                        "**/build*/build.log"
                     ]
                  },
                  "flash": {
                     "bucket": "idf-artifacts",
                     "patterns": [
                        "**/build*/bootloader/*.bin",
                        "**/build*/*.bin",
                        "**/build*/partition_table/*.bin",
                        "**/build*/flasher_args.json",
                        "**/build*/flash_project_args",
                        "**/build*/config/sdkconfig.json",
                        "**/build*/sdkconfig",
                        "**/build*/project_description.json"
                     ]
                  }
               },
               "title": "S3",
               "type": "object"
            },
            "build_job_filepatterns": {
               "default": [
                  "app_info_*.txt",
                  "build_summary_*.xml"
               ],
               "items": {
                  "type": "string"
               },
               "title": "Build Job Filepatterns",
               "type": "array"
            },
            "test_job_filepatterns": {
               "default": [
                  "pytest-embedded/",
                  "XUNIT_RESULT*.xml"
               ],
               "items": {
                  "type": "string"
               },
               "title": "Test Job Filepatterns",
               "type": "array"
            }
         },
         "title": "ArtifactSettings",
         "type": "object"
      },
      "BuildPipelineSettings": {
         "additionalProperties": false,
         "properties": {
            "workflow_name": {
               "default": "Build Child Pipeline",
               "title": "Workflow Name",
               "type": "string"
            },
            "presigned_json_job_name": {
               "default": "generate_presigned_json",
               "title": "Presigned Json Job Name",
               "type": "string"
            },
            "job_template_name": {
               "default": ".default_build_settings",
               "title": "Job Template Name",
               "type": "string"
            },
            "job_template_jinja": {
               "default": "{{ settings.gitlab.build_pipeline.job_template_name }}:\n  stage: build\n  tags: {{ settings.gitlab.build_pipeline.job_tags }}\n  timeout: 1h\n  artifacts:\n    paths:\n    {%- for path in settings.gitlab.artifacts.build_job_filepatterns %}\n      - {{ path }}\n    {%- endfor %}\n    expire_in: 1 week\n    when: always\n  script:\n    - idf-ci build run\n      --parallel-count ${CI_NODE_TOTAL:-1}\n      --parallel-index ${CI_NODE_INDEX:-1}",
               "title": "Job Template Jinja",
               "type": "string"
            },
            "job_tags": {
               "default": [
                  "build"
               ],
               "items": {
                  "type": "string"
               },
               "title": "Job Tags",
               "type": "array"
            },
            "runs_per_job": {
               "default": 60,
               "title": "Runs Per Job",
               "type": "integer"
            },
            "jobs_jinja": {
               "default": "{%- if test_related_apps_count > 0 %}\nbuild_test_related_apps:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if test_related_parallel_count > 1 %}\n  parallel: {{ test_related_parallel_count }}\n{%- endif %}\n  needs:\n    - pipeline: $PARENT_PIPELINE_ID\n      job: generate_build_child_pipeline\n    - pipeline: $PARENT_PIPELINE_ID\n      job: pipeline_variables\n  variables:\n    IDF_CI_BUILD_ONLY_TEST_RELATED_APPS: \"1\"\n{%- endif %}\n\n{%- if non_test_related_apps_count > 0 %}\nbuild_non_test_related_apps:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if non_test_related_parallel_count > 1 %}\n  parallel: {{ non_test_related_parallel_count }}\n{%- endif %}\n  needs:\n    - pipeline: $PARENT_PIPELINE_ID\n      job: generate_build_child_pipeline\n    - pipeline: $PARENT_PIPELINE_ID\n      job: pipeline_variables\n  variables:\n    IDF_CI_BUILD_ONLY_NON_TEST_RELATED_APPS: \"1\"\n{%- endif %}",
               "title": "Jobs Jinja",
               "type": "string"
            },
            "pre_yaml_jinja": {
               "default": "",
               "title": "Pre Yaml Jinja",
               "type": "string"
            },
            "yaml_jinja": {
               "default": "{{ settings.gitlab.build_pipeline.pre_yaml_jinja }}\n\nworkflow:\n  name: {{ settings.gitlab.build_pipeline.workflow_name }}\n  rules:\n    - when: always\n\n{%- if settings.gitlab.build_pipeline.job_template_jinja %}\n{{ job_template }}\n{%- endif %}\n\n{{ jobs }}\n\n{%- if test_related_apps_count > 0 %}\ngenerate_test_child_pipeline:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n  needs:\n    - build_test_related_apps\n  artifacts:\n    paths:\n      - {{ settings.gitlab.test_pipeline.yaml_filename }}\n  script:\n    - idf-ci gitlab test-child-pipeline\n\ntest-child-pipeline:\n  stage: .post\n  needs:\n    - generate_test_child_pipeline\n  variables:\n    PARENT_PIPELINE_ID: $PARENT_PIPELINE_ID\n  trigger:\n    include:\n      - artifact: {{ settings.gitlab.test_pipeline.yaml_filename }}\n        job: generate_test_child_pipeline\n    strategy: depend\n{%- endif %}",
               "title": "Yaml Jinja",
               "type": "string"
            },
            "yaml_filename": {
               "default": "build_child_pipeline.yml",
               "title": "Yaml Filename",
               "type": "string"
            }
         },
         "title": "BuildPipelineSettings",
         "type": "object"
      },
      "GitlabSettings": {
         "additionalProperties": false,
         "properties": {
            "project": {
               "default": "espressif/esp-idf",
               "title": "Project",
               "type": "string"
            },
            "known_failure_cases_bucket_name": {
               "default": "ignore-test-result-files",
               "title": "Known Failure Cases Bucket Name",
               "type": "string"
            },
            "artifacts": {
               "$ref": "#/$defs/ArtifactSettings",
               "default": {
                  "s3": {
                     "debug": {
                        "bucket": "idf-artifacts",
                        "patterns": [
                           "**/build*/bootloader/*.map",
                           "**/build*/bootloader/*.elf",
                           "**/build*/*.map",
                           "**/build*/*.elf",
                           "**/build*/build.log"
                        ]
                     },
                     "flash": {
                        "bucket": "idf-artifacts",
                        "patterns": [
                           "**/build*/bootloader/*.bin",
                           "**/build*/*.bin",
                           "**/build*/partition_table/*.bin",
                           "**/build*/flasher_args.json",
                           "**/build*/flash_project_args",
                           "**/build*/config/sdkconfig.json",
                           "**/build*/sdkconfig",
                           "**/build*/project_description.json"
                        ]
                     }
                  },
                  "build_job_filepatterns": [
                     "app_info_*.txt",
                     "build_summary_*.xml"
                  ],
                  "test_job_filepatterns": [
                     "pytest-embedded/",
                     "XUNIT_RESULT*.xml"
                  ]
               }
            },
            "build_pipeline": {
               "$ref": "#/$defs/BuildPipelineSettings",
               "default": {
                  "workflow_name": "Build Child Pipeline",
                  "presigned_json_job_name": "generate_presigned_json",
                  "job_template_name": ".default_build_settings",
                  "job_template_jinja": "{{ settings.gitlab.build_pipeline.job_template_name }}:\n  stage: build\n  tags: {{ settings.gitlab.build_pipeline.job_tags }}\n  timeout: 1h\n  artifacts:\n    paths:\n    {%- for path in settings.gitlab.artifacts.build_job_filepatterns %}\n      - {{ path }}\n    {%- endfor %}\n    expire_in: 1 week\n    when: always\n  script:\n    - idf-ci build run\n      --parallel-count ${CI_NODE_TOTAL:-1}\n      --parallel-index ${CI_NODE_INDEX:-1}",
                  "job_tags": [
                     "build"
                  ],
                  "runs_per_job": 60,
                  "jobs_jinja": "{%- if test_related_apps_count > 0 %}\nbuild_test_related_apps:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if test_related_parallel_count > 1 %}\n  parallel: {{ test_related_parallel_count }}\n{%- endif %}\n  needs:\n    - pipeline: $PARENT_PIPELINE_ID\n      job: generate_build_child_pipeline\n    - pipeline: $PARENT_PIPELINE_ID\n      job: pipeline_variables\n  variables:\n    IDF_CI_BUILD_ONLY_TEST_RELATED_APPS: \"1\"\n{%- endif %}\n\n{%- if non_test_related_apps_count > 0 %}\nbuild_non_test_related_apps:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if non_test_related_parallel_count > 1 %}\n  parallel: {{ non_test_related_parallel_count }}\n{%- endif %}\n  needs:\n    - pipeline: $PARENT_PIPELINE_ID\n      job: generate_build_child_pipeline\n    - pipeline: $PARENT_PIPELINE_ID\n      job: pipeline_variables\n  variables:\n    IDF_CI_BUILD_ONLY_NON_TEST_RELATED_APPS: \"1\"\n{%- endif %}",
                  "pre_yaml_jinja": "",
                  "yaml_jinja": "{{ settings.gitlab.build_pipeline.pre_yaml_jinja }}\n\nworkflow:\n  name: {{ settings.gitlab.build_pipeline.workflow_name }}\n  rules:\n    - when: always\n\n{%- if settings.gitlab.build_pipeline.job_template_jinja %}\n{{ job_template }}\n{%- endif %}\n\n{{ jobs }}\n\n{%- if test_related_apps_count > 0 %}\ngenerate_test_child_pipeline:\n  extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n  needs:\n    - build_test_related_apps\n  artifacts:\n    paths:\n      - {{ settings.gitlab.test_pipeline.yaml_filename }}\n  script:\n    - idf-ci gitlab test-child-pipeline\n\ntest-child-pipeline:\n  stage: .post\n  needs:\n    - generate_test_child_pipeline\n  variables:\n    PARENT_PIPELINE_ID: $PARENT_PIPELINE_ID\n  trigger:\n    include:\n      - artifact: {{ settings.gitlab.test_pipeline.yaml_filename }}\n        job: generate_test_child_pipeline\n    strategy: depend\n{%- endif %}",
                  "yaml_filename": "build_child_pipeline.yml"
               }
            },
            "test_pipeline": {
               "$ref": "#/$defs/TestPipelineSettings",
               "default": {
                  "workflow_name": "Test Child Pipeline",
                  "presigned_json_job_name": "generate_presigned_json",
                  "job_template_name": ".default_test_settings",
                  "job_template_jinja": "{{ settings.gitlab.test_pipeline.job_template_name }}:\n  stage: test\n  timeout: 1h\n  artifacts:\n    paths:\n    {%- for path in settings.gitlab.artifacts.test_job_filepatterns %}\n      - {{ path }}\n    {%- endfor %}\n    expire_in: 1 week\n    when: always\n  variables:\n    PYTEST_EXTRA_FLAGS: \"\"\n  script:\n    - pytest ${nodes}\n      --parallel-count ${CI_NODE_TOTAL:-1}\n      --parallel-index ${CI_NODE_INDEX:-1}\n      --junitxml XUNIT_RESULT_${CI_JOB_NAME_SLUG}.xml\n      ${PYTEST_EXTRA_FLAGS}",
                  "job_tags": [],
                  "runs_per_job": 30,
                  "jobs_jinja": "{% for job in jobs %}\n{{ job['name'] }}:\n  extends:\n    - {{ settings.gitlab.test_pipeline.job_template_name }}\n    {%- for extra_extend in job.get('extra_extends', []) %}\n    - {{ extra_extend }}\n    {%- endfor %}\n  tags: {{ job['tags'] }}\n{%- if job['parallel_count'] > 1 %}\n  parallel: {{ job['parallel_count'] }}\n{%- endif %}\n  variables:\n    nodes: {{ job['nodes'] }}\n{% endfor %}",
                  "pre_yaml_jinja": "",
                  "yaml_jinja": "{{ settings.gitlab.test_pipeline.pre_yaml_jinja }}\n\nworkflow:\n  name: {{ settings.gitlab.test_pipeline.workflow_name }}\n  rules:\n    - when: always\n\n{%- if settings.gitlab.test_pipeline.job_template_jinja %}\n{{ default_template }}\n{%- endif %}\n\n{{ jobs }}",
                  "yaml_filename": "test_child_pipeline.yml"
               }
            }
         },
         "title": "GitlabSettings",
         "type": "object"
      },
      "S3FilePatternConfig": {
         "properties": {
            "bucket": {
               "title": "Bucket",
               "type": "string"
            },
            "patterns": {
               "items": {
                  "type": "string"
               },
               "title": "Patterns",
               "type": "array"
            },
            "if_clause": {
               "title": "If Clause",
               "type": "string"
            }
         },
         "required": [
            "bucket",
            "patterns"
         ],
         "title": "S3FilePatternConfig",
         "type": "object"
      },
      "TestPipelineSettings": {
         "additionalProperties": false,
         "properties": {
            "workflow_name": {
               "default": "Test Child Pipeline",
               "title": "Workflow Name",
               "type": "string"
            },
            "presigned_json_job_name": {
               "default": "generate_presigned_json",
               "title": "Presigned Json Job Name",
               "type": "string"
            },
            "job_template_name": {
               "default": ".default_test_settings",
               "title": "Job Template Name",
               "type": "string"
            },
            "job_template_jinja": {
               "default": "{{ settings.gitlab.test_pipeline.job_template_name }}:\n  stage: test\n  timeout: 1h\n  artifacts:\n    paths:\n    {%- for path in settings.gitlab.artifacts.test_job_filepatterns %}\n      - {{ path }}\n    {%- endfor %}\n    expire_in: 1 week\n    when: always\n  variables:\n    PYTEST_EXTRA_FLAGS: \"\"\n  script:\n    - pytest ${nodes}\n      --parallel-count ${CI_NODE_TOTAL:-1}\n      --parallel-index ${CI_NODE_INDEX:-1}\n      --junitxml XUNIT_RESULT_${CI_JOB_NAME_SLUG}.xml\n      ${PYTEST_EXTRA_FLAGS}",
               "title": "Job Template Jinja",
               "type": "string"
            },
            "job_tags": {
               "default": [],
               "items": {
                  "type": "string"
               },
               "title": "Job Tags",
               "type": "array"
            },
            "runs_per_job": {
               "default": 30,
               "title": "Runs Per Job",
               "type": "integer"
            },
            "jobs_jinja": {
               "default": "{% for job in jobs %}\n{{ job['name'] }}:\n  extends:\n    - {{ settings.gitlab.test_pipeline.job_template_name }}\n    {%- for extra_extend in job.get('extra_extends', []) %}\n    - {{ extra_extend }}\n    {%- endfor %}\n  tags: {{ job['tags'] }}\n{%- if job['parallel_count'] > 1 %}\n  parallel: {{ job['parallel_count'] }}\n{%- endif %}\n  variables:\n    nodes: {{ job['nodes'] }}\n{% endfor %}",
               "title": "Jobs Jinja",
               "type": "string"
            },
            "pre_yaml_jinja": {
               "default": "",
               "title": "Pre Yaml Jinja",
               "type": "string"
            },
            "yaml_jinja": {
               "default": "{{ settings.gitlab.test_pipeline.pre_yaml_jinja }}\n\nworkflow:\n  name: {{ settings.gitlab.test_pipeline.workflow_name }}\n  rules:\n    - when: always\n\n{%- if settings.gitlab.test_pipeline.job_template_jinja %}\n{{ default_template }}\n{%- endif %}\n\n{{ jobs }}",
               "title": "Yaml Jinja",
               "type": "string"
            },
            "yaml_filename": {
               "default": "test_child_pipeline.yml",
               "title": "Yaml Filename",
               "type": "string"
            }
         },
         "title": "TestPipelineSettings",
         "type": "object"
      }
   },
   "additionalProperties": false
}

Fields:

field built_app_list_filepatterns: List[str] = ['app_info_*.txt']: Glob patterns for files containing built app information.

field ci_detection_envs: List[str] = ['CI', 'GITHUB_ACTIONS', 'CIRCLECI', 'TRAVIS', 'JENKINS_URL', 'DRONE', 'APPVEYOR', 'BITBUCKET_COMMIT', 'SEMAPHORE', 'TEAMCITY_VERSION']: Environment variables used to detect if running in CI.

field ci_runtime_envs: Dict[str, Any] = {}: Environment variables to set in CI environment.

field collected_non_test_related_apps_filepath: str = 'non_test_related_apps.txt': Path to file containing non-test-related apps.

field collected_test_related_apps_filepath: str = 'test_related_apps.txt': Path to file containing test-related apps.

field component_ignored_file_extensions: List[str] = ['.md', '.rst', '.yaml', '.yml', '.py']: File extensions to ignore when mapping files to components.

field component_mapping_regexes: List[str] = ['/components/(.+)/', '/common_components/(.+)/']: List of regex patterns to extract component names from file paths.

field extend_component_ignored_file_extensions: List[str] = []: Additional file extensions to ignore.

field extend_component_mapping_regexes: List[str] = []: Additional component mapping regex patterns to extend the default list.

field extra_default_build_targets: List[str] = []: Additional build targets to include by default.

field gitlab: GitlabSettings = GitlabSettings(project='espressif/esp-idf', known_failure_cases_bucket_name='ignore-test-result-files', artifacts=ArtifactSettings(s3={'debug': {'bucket': 'idf-artifacts', 'patterns': ['**/build*/bootloader/*.map', '**/build*/bootloader/*.elf', '**/build*/*.map', '**/build*/*.elf', '**/build*/build.log']}, 'flash': {'bucket': 'idf-artifacts', 'patterns': ['**/build*/bootloader/*.bin', '**/build*/*.bin', '**/build*/partition_table/*.bin', '**/build*/flasher_args.json', '**/build*/flash_project_args', '**/build*/config/sdkconfig.json', '**/build*/sdkconfig', '**/build*/project_description.json']}}, build_job_filepatterns=['app_info_*.txt', 'build_summary_*.xml'], test_job_filepatterns=['pytest-embedded/', 'XUNIT_RESULT*.xml']), build_pipeline=BuildPipelineSettings(workflow_name='Build Child Pipeline', presigned_json_job_name='generate_presigned_json', job_template_name='.default_build_settings', job_template_jinja='{{ settings.gitlab.build_pipeline.job_template_name }}:\n stage: build\n tags: {{ settings.gitlab.build_pipeline.job_tags }}\n timeout: 1h\n artifacts:\n paths:\n {%- for path in settings.gitlab.artifacts.build_job_filepatterns %}\n - {{ path }}\n {%- endfor %}\n expire_in: 1 week\n when: always\n script:\n - idf-ci build run\n --parallel-count ${CI_NODE_TOTAL:-1}\n --parallel-index ${CI_NODE_INDEX:-1}', job_tags=['build'], runs_per_job=60, jobs_jinja='{%- if test_related_apps_count > 0 %}\nbuild_test_related_apps:\n extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if test_related_parallel_count > 1 %}\n parallel: {{ test_related_parallel_count }}\n{%- endif %}\n needs:\n - pipeline: $PARENT_PIPELINE_ID\n job: generate_build_child_pipeline\n - pipeline: $PARENT_PIPELINE_ID\n job: pipeline_variables\n variables:\n IDF_CI_BUILD_ONLY_TEST_RELATED_APPS: "1"\n{%- endif %}\n\n{%- if non_test_related_apps_count > 0 %}\nbuild_non_test_related_apps:\n extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n{%- if non_test_related_parallel_count > 1 %}\n parallel: {{ non_test_related_parallel_count }}\n{%- endif %}\n needs:\n - pipeline: $PARENT_PIPELINE_ID\n job: generate_build_child_pipeline\n - pipeline: $PARENT_PIPELINE_ID\n job: pipeline_variables\n variables:\n IDF_CI_BUILD_ONLY_NON_TEST_RELATED_APPS: "1"\n{%- endif %}', pre_yaml_jinja='', yaml_jinja='{{ settings.gitlab.build_pipeline.pre_yaml_jinja }}\n\nworkflow:\n name: {{ settings.gitlab.build_pipeline.workflow_name }}\n rules:\n - when: always\n\n{%- if settings.gitlab.build_pipeline.job_template_jinja %}\n{{ job_template }}\n{%- endif %}\n\n{{ jobs }}\n\n{%- if test_related_apps_count > 0 %}\ngenerate_test_child_pipeline:\n extends: {{ settings.gitlab.build_pipeline.job_template_name }}\n needs:\n - build_test_related_apps\n artifacts:\n paths:\n - {{ settings.gitlab.test_pipeline.yaml_filename }}\n script:\n - idf-ci gitlab test-child-pipeline\n\ntest-child-pipeline:\n stage: .post\n needs:\n - generate_test_child_pipeline\n variables:\n PARENT_PIPELINE_ID: $PARENT_PIPELINE_ID\n trigger:\n include:\n - artifact: {{ settings.gitlab.test_pipeline.yaml_filename }}\n job: generate_test_child_pipeline\n strategy: depend\n{%- endif %}', yaml_filename='build_child_pipeline.yml'), test_pipeline=TestPipelineSettings(workflow_name='Test Child Pipeline', presigned_json_job_name='generate_presigned_json', job_template_name='.default_test_settings', job_template_jinja='{{ settings.gitlab.test_pipeline.job_template_name }}:\n stage: test\n timeout: 1h\n artifacts:\n paths:\n {%- for path in settings.gitlab.artifacts.test_job_filepatterns %}\n - {{ path }}\n {%- endfor %}\n expire_in: 1 week\n when: always\n variables:\n PYTEST_EXTRA_FLAGS: ""\n script:\n - pytest ${nodes}\n --parallel-count ${CI_NODE_TOTAL:-1}\n --parallel-index ${CI_NODE_INDEX:-1}\n --junitxml XUNIT_RESULT_${CI_JOB_NAME_SLUG}.xml\n ${PYTEST_EXTRA_FLAGS}', job_tags=[], runs_per_job=30, jobs_jinja="{% for job in jobs %}\n{{ job['name'] }}:\n extends:\n - {{ settings.gitlab.test_pipeline.job_template_name }}\n {%- for extra_extend in job.get('extra_extends', []) %}\n - {{ extra_extend }}\n {%- endfor %}\n tags: {{ job['tags'] }}\n{%- if job['parallel_count'] > 1 %}\n parallel: {{ job['parallel_count'] }}\n{%- endif %}\n variables:\n nodes: {{ job['nodes'] }}\n{% endfor %}", pre_yaml_jinja='', yaml_jinja='{{ settings.gitlab.test_pipeline.pre_yaml_jinja }}\n\nworkflow:\n name: {{ settings.gitlab.test_pipeline.workflow_name }}\n rules:\n - when: always\n\n{%- if settings.gitlab.test_pipeline.job_template_jinja %}\n{{ default_template }}\n{%- endif %}\n\n{{ jobs }}', yaml_filename='test_child_pipeline.yml')): GitLab-specific settings.

field local_runtime_envs: Dict[str, Any] = {}: Environment variables to set in local development.

field preserve_non_test_related_apps: bool = True: Whether to preserve non-test-related apps.

field preserve_test_related_apps: bool = True: Whether to preserve test-related apps.

classmethod read_apps_from_filepatterns(patterns: List[str]) → List[App] | None

Helper method to read apps from files matching given patterns.

Parameters:: patterns – List of file patterns to search for
Returns:: List of App objects read from the files, or None if no files found

classmethod read_apps_from_files(filepaths: Sequence[str | PathLike]) → List[App] | None

Helper method to read apps from files.

Parameters:: filepaths – List of file paths to read
Returns:: List of App objects read from the files, or None if no files found

get_built_apps_list() → List[App] | None

Get the list of successfully built applications from the app info files.

Returns:: List of App objects representing successfully built applications, or None if no files found

get_modified_components(modified_files: Iterable[str]) → Set[str]

Get the set of components that have been modified based on the provided files.

Parameters:: modified_files – Iterable of file paths that have been modified
Returns:: Set of component names that have been modified

model_post_init(_CiSettings__context: Any) → None: Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

CONFIG_FILE_PATH: ClassVar[Path | None] = None: Path to the configuration file to be used (class variable).

property all_component_mapping_regexes: Set[Pattern]

Get all component mapping regexes as compiled pattern objects.

Returns:: Set of compiled regex patterns

property is_in_ci: bool

Check if the code is running in a CI environment.

Returns:: True if in CI environment, False otherwise

pydantic settings idf_ci.GitlabEnvVars

Bases: BaseSettings

Show JSON schema

{
   "title": "GitlabEnvVars",
   "type": "object",
   "properties": {
      "CHANGED_FILES_SEMICOLON_SEPARATED": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Changed Files Semicolon Separated"
      },
      "GITLAB_HTTPS_SERVER": {
         "default": "https://gitlab.com",
         "title": "Gitlab Https Server",
         "type": "string"
      },
      "GITLAB_ACCESS_TOKEN": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Gitlab Access Token"
      },
      "IDF_S3_SERVER": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf S3 Server"
      },
      "IDF_S3_ACCESS_KEY": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf S3 Access Key"
      },
      "IDF_S3_SECRET_KEY": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf S3 Secret Key"
      },
      "IDF_PATH": {
         "default": "",
         "title": "Idf Path",
         "type": "string"
      },
      "IDF_CI_IS_DEBUG_PIPELINE": {
         "anyOf": [
            {
               "type": "boolean"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf Ci Is Debug Pipeline"
      },
      "IDF_CI_SELECT_BY_FILTER_EXPR": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf Ci Select By Filter Expr"
      },
      "IDF_CI_SELECT_ALL_PYTEST_CASES": {
         "anyOf": [
            {
               "type": "boolean"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf Ci Select All Pytest Cases"
      },
      "IDF_CI_BUILD_ONLY_TEST_RELATED_APPS": {
         "anyOf": [
            {
               "type": "boolean"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf Ci Build Only Test Related Apps"
      },
      "IDF_CI_BUILD_ONLY_NON_TEST_RELATED_APPS": {
         "anyOf": [
            {
               "type": "boolean"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Idf Ci Build Only Non Test Related Apps"
      }
   },
   "additionalProperties": false
}

Fields:

field CHANGED_FILES_SEMICOLON_SEPARATED: str | None = None: Semicolon-separated list of changed files in the pipeline.

field GITLAB_ACCESS_TOKEN: str | None = None: Access token for GitLab API authentication.

field GITLAB_HTTPS_SERVER: str = 'https://gitlab.com': GitLab server URL for API calls.

field IDF_CI_BUILD_ONLY_NON_TEST_RELATED_APPS: bool | None = None: Flag indicating whether to build only non-test-related apps.

field IDF_CI_BUILD_ONLY_TEST_RELATED_APPS: bool | None = None: Flag indicating whether to build only test-related apps.

field IDF_CI_IS_DEBUG_PIPELINE: bool | None = None: Flag indicating whether this is a debug pipeline.

field IDF_CI_SELECT_ALL_PYTEST_CASES: bool | None = None: Flag indicating whether to select all pytest cases.

field IDF_CI_SELECT_BY_FILTER_EXPR: str | None = None: Filter expression for selecting pytest cases.

field IDF_PATH: str = '': Path to the ESP-IDF directory.

field IDF_S3_ACCESS_KEY: str | None = None: S3 access key for authentication.

field IDF_S3_SECRET_KEY: str | None = None: S3 secret key for authentication.

field IDF_S3_SERVER: str | None = None: S3 server endpoint URL.

model_post_init(_GitlabEnvVars__context: Any) → None: Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

property select_all_pytest_cases: bool

Determine if all pytest cases should be selected.

Returns:: True if this is a full pipeline run, False otherwise

property select_by_filter_expr: str | None

Get the filter expression for pytest cases.

Returns:: The filter expression if set, None otherwise

class idf_ci.IdfPytestPlugin(*, cli_target: str, sdkconfig_name: str | None = None)

Bases: object

build_dir(request: FixtureRequest, app_path: str, target: str | None, config: str | None) → str

Find a valid build directory based on priority rules.

Checks local build directories in the following order:

build_<target>_<config>
build_<target>
build_<config>
build

Parameters:

request – Pytest fixture request
app_path – Path to the application
target – Target being used
config – Configuration being used

Returns:

Valid build directory name

Raises:

ValueError – If no valid build directory is found

config(request: FixtureRequest) → str

Fixture that provides the configuration for tests.

Parameters:: request – Pytest fixture request
Returns:: Configuration string, defaults to ‘default’ if not specified

static get_case_by_item(item: Item) → PytestCase | None

Get the test case associated with a pytest item.

Parameters:: item – The pytest test item
Returns:: PytestCase object or None if not found

pytest_collection_modifyitems(config: Config, items: List[Function])

Filter test cases based on target, sdkconfig, and available apps.

Parameters:

config – Pytest configuration
items – Collected test items

pytest_pycollect_makemodule(module_path: Path)

Handle module collection for pytest, mocking any missing modules.

This hook runs before module collection to prevent errors from missing dependencies by automatically mocking them.

Parameters:: module_path – Path to the module being collected

pytest_report_collectionfinish(items: List[Function]) → None

target(request: FixtureRequest) → str

Fixture that provides the target for tests.

Parameters:: request – Pytest fixture request
Returns:: Target string
Raises:: ValueError – If target parameter is not defined

class idf_ci.PytestApp(path: str, target: str, config: str)

Bases: object

Represents a pytest app.

property build_dir: str

Returns the build directory for the app.

Note

Matches the build_dir (by default build_@t_@w) in the idf-build-apps config file.

Returns:: The build directory for the app.

class idf_ci.PytestCase(apps: List[PytestApp], item: Function)

Bases: object

Represents a pytest test case.

KNOWN_ENV_MARKERS: ClassVar[Set[str]] = {}

property all_markers: Set[str]

property caseid: str

property configs: List[str]

property env_markers: Set[str]

property env_selector: str

classmethod from_item(item: Function) → PytestCase | None

Create a PytestCase from a pytest item.

Parameters:: item – Pytest function item
Returns:: PytestCase instance or None if targets not defined

classmethod get_param(item: Function, key: str, default: Any = None) → Any

Get parameter value from pytest item.

Parameters:

item – Pytest function item
key – Parameter key to retrieve
default – Default value if key not found

Returns:

Parameter value or default

get_skip_reason_if_not_built(app_dirs: List[str] | None = None) → str | None

Check if all binaries of the test case are built in the app lists.

Parameters:: app_dirs – App folder paths to check
Returns:: Skip reason string if not all binaries are built, None otherwise

property is_host_test: bool

property is_single_dut: bool

property name: str

property path: str

property runner_tags: Tuple[str, ...]

property target_selector: str

property targets: List[str]

idf_ci.build(*, paths: List[str] | None = None, target: str = 'all', parallel_count: int = 1, parallel_index: int = 1, modified_files: List[str] | None = None, modified_components: List[str] | None = None, only_test_related: bool | None = None, only_non_test_related: bool | None = None, dry_run: bool = False, build_system: UndefinedType | str = __UNDEF__, marker_expr: UndefinedType | str = __UNDEF__, filter_expr: str | None = None) → Tuple[List[App], int]

Build applications based on specified parameters.

Parameters:

paths – List of paths to search for applications
target – Target device(s) separated by commas
parallel_count – Total number of parallel jobs
parallel_index – Index of current parallel job (1-based)
modified_files – List of modified files
modified_components – List of modified components
only_test_related – Only build test-related applications
only_non_test_related – Only build non-test-related applications
dry_run – Do not actually build, just simulate
build_system – Filter the apps by build system. Can be “cmake”, “make” or a custom App class path
marker_expr – Pytest marker expression
filter_expr – Filter expression

Returns:

Tuple of (built apps, build return code)

Get test-related and non-test-related applications.

Parameters:

paths – List of paths to search for applications
target – Target device(s) separated by commas
modified_files – List of modified files
modified_components – List of modified components
filter_expr – Pytest filter expression -k
default_build_targets – Default build targets to use
marker_expr – Pytest marker expression -m
compare_manifest_sha_filepath – Path to the manifest SHA file generated by idf-build-apps dump-manifest-sha
build_system – Filter the apps by build system. Can be “cmake”, “make” or a custom App class path

Returns:

Tuple of (test_related_apps, non_test_related_apps)

Collect pytest test cases from specified paths.

Parameters:

paths – List of file system paths to collect test cases from
target – Filter by targets
sdkconfig_name – Filter tests whose apps are built with this sdkconfig name
marker_expr – Filter by pytest marker expression -m
filter_expr – Filter by pytest filter expression -k

Returns:

List of collected PytestCase objects

Raises:

RuntimeError – If pytest collection fails