Dynamic GitLab Pipelines

idf-ci generates GitLab CI as a small pipeline hierarchy.

It has three parts at minimum:

the parent pipeline prepares inputs and generates the build child pipeline YAML
the build child pipeline builds apps and, when needed, generates the test child pipeline YAML
the test child pipeline runs grouped pytest jobs

What the generated flow looks like

        flowchart LR
    subgraph P["Parent pipeline"]
        direction TD
        PV["pipeline_variables"]
        GBCP["generate_build_child_pipeline"]
    end

    subgraph B["Build child pipeline"]
        direction TD
        BTRA["build_test_related_apps\n(parallel when needed)"]
        BNTRA["build_non_test_related_apps\n(parallel when needed)"]
        GTCP["generate_test_child_pipeline"]
    end

    subgraph TCP["Test child pipeline"]
        direction TD
        TJ1["[target] - [env]\n(parallel when needed)"]
        TJ2["... more grouped test jobs ..."]
    end

    PV --> BTRA
    GBCP --> BTRA
    PV --> BNTRA
    GBCP --> BNTRA
    PV --> GTCP
    BTRA --> GTCP
    GTCP --> TCP
    GTCP --> TJ1
    GTCP --> TJ2

Build child pipeline

The build child pipeline is generated by idf_ci.idf_gitlab.pipeline.build_child_pipeline.

It classifies discovered apps into two buckets:

test-related apps: apps associated with pytest-driven testing
non-test-related apps: apps that still need building but do not feed the generated test pipeline

From those buckets, the generated pipeline can contain up to four jobs.

build_test_related_apps: Builds only apps that feed later pytest execution.
build_non_test_related_apps: Builds the remaining selected apps.
generate_test_child_pipeline: Runs only when there are test-related apps. It depends on build_test_related_apps and the parent pipeline_variables job, then produces test_child_pipeline.yml. The direct parent dependency makes exported variables available during test discovery; a dependency on the build job alone does not pass them through.
test-child-pipeline: Trigger job that starts the downstream test child pipeline from test_child_pipeline.yml.

Both build jobs also declare cross-pipeline needs on:

generate_build_child_pipeline
pipeline_variables

This gives each build job the generated YAML context and exported pipeline variables from the parent pipeline.

If no apps are selected, idf-ci emits a single fake_pass job.

Building with multiple toolchains

Configure the toolchains in .idf_ci.toml (GCC is the only implicit default):

[gitlab]
build_enabled_toolchains = ["gcc", "clang"]
test_enabled_toolchains = ["gcc"]

One invocation of idf-ci gitlab build-child-pipeline discovers apps separately for each toolchain and writes build_child_pipeline.yml for GCC and build_child_pipeline_clang.yml for Clang. It also writes separate app lists (test_related_apps.txt / test_related_apps_clang.txt and their non_test_related_apps counterparts). Include all these files in the parent generation job’s artifacts; no second discovery job is required.

Create one parent trigger job for each desired build YAML, each with an include:artifact pointing at the corresponding file from the same generation job. For example:

generate_build_child_pipeline:
  script: idf-ci gitlab build-child-pipeline
  artifacts:
    paths:
      - build_child_pipeline.yml
      - build_child_pipeline_clang.yml
      - test_related_apps.txt
      - test_related_apps_clang.txt
      - non_test_related_apps.txt
      - non_test_related_apps_clang.txt

build-gcc:
  needs: [generate_build_child_pipeline]
  variables:
    PARENT_PIPELINE_ID: "$CI_PIPELINE_ID"
  trigger:
    include:
      - artifact: build_child_pipeline.yml
        job: generate_build_child_pipeline
    strategy: depend

build-clang:
  needs: [generate_build_child_pipeline]
  variables:
    PARENT_PIPELINE_ID: "$CI_PIPELINE_ID"
  trigger:
    include:
      - artifact: build_child_pipeline_clang.yml
        job: generate_build_child_pipeline
    strategy: depend

The generated build jobs set IDF_TOOLCHAIN and use separate app lists for each toolchain. IDF_CI_ARTIFACT_NAMESPACE supplied to the build-YAML generator names a build variant; the generator combines it with each non-GCC toolchain to make an effective namespace. App discovery also uses that effective value, so a later toolchain cannot accidentally reuse an earlier toolchain’s freshly written list. The same value is written into that toolchain’s build YAML, used as its app-list filename suffix, and used by S3 commands in its build jobs. No explicit namespace is needed for ordinary GCC/Clang builds:

Effective namespace and generated app lists
Toolchain	Namespace on generator	Namespace in build YAML	Test-related app list
`gcc`	unset	unset (legacy S3 prefix)	`test_related_apps.txt`
`clang`	unset	`clang`	`test_related_apps_clang.txt`
`gcc`	`buildv2`	`buildv2`	`test_related_apps_buildv2.txt`
`clang`	`buildv2`	`buildv2_clang`	`test_related_apps_buildv2_clang.txt`

non_test_related_apps uses the same suffixes. The GCC name stays unchanged when the variant is unset; an explicit value of gcc is not treated as unset and produces test_related_apps_gcc.txt. The two toolchains in one generation have separate S3 prefixes (for example, <gitlab.project>/<commit_sha>_buildv2_clang/). Across different pipeline generations, choose raw variant names that do not collide with another effective namespace: a GCC variant named clang would reuse the default Clang namespace and S3 prefix. The build YAML filename still depends only on the toolchain: build_child_pipeline.yml and build_child_pipeline_clang.yml in this example.

To generate this two-toolchain variant, set the raw variant only on the generation job, and upload all four lists along with the two YAML files:

generate_build_child_pipeline:
  variables:
    IDF_CI_ARTIFACT_NAMESPACE: "buildv2"
  script: idf-ci gitlab build-child-pipeline
  artifacts:
    paths:
      - build_child_pipeline.yml
      - build_child_pipeline_clang.yml
      - test_related_apps_buildv2.txt
      - test_related_apps_buildv2_clang.txt
      - non_test_related_apps_buildv2.txt
      - non_test_related_apps_buildv2_clang.txt

Use the parent trigger jobs shown above for each toolchain, but do not forward the raw buildv2 namespace from the parent into both child pipelines: GitLab trigger/pipeline variables can override the generated YAML’s per-toolchain value. In particular, do not set it as a variable on those triggers or enable forward: pipeline_variables: true for a parent pipeline variable of that name. If your project must explicitly pass the namespace, pass buildv2 on the GCC trigger and buildv2_clang on the Clang trigger instead. The build jobs, test-YAML generator, generated test jobs, and S3 upload/download commands must all see the effective namespace for their toolchain, not the raw variant. A standalone artifact command or idf-ci gitlab test-child-pipeline --toolchain clang likewise needs IDF_CI_ARTIFACT_NAMESPACE=buildv2_clang for this variant; do not append _clang a second time. When the test generator runs within the generated build pipeline, it receives the effective value automatically. Only toolchains in test_enabled_toolchains create a test generation/trigger pair. When Clang tests are enabled, the Clang build pipeline generates and triggers test_child_pipeline_clang.yml.

Use gitlab.build_pipeline.additional_include_yaml_files (also available via CLI --config overrides) to include local GitLab CI files in a generated build pipeline. Invoke the generator separately per toolchain if their includes differ: both need common templates, but only GCC might need extra reporting jobs. Do not also set a top-level include in pre_yaml_jinja, which would duplicate the YAML key. Generated test jobs and their trigger belong to idf-ci; project includes should supply only additional jobs and templates.

The nested trigger normally forwards its build pipeline ID as PARENT_PIPELINE_ID. Projects whose test jobs need the original upstream pipeline instead can set gitlab.test_pipeline.parent_pipeline_id = '$PARENT_PIPELINE_ID'.

Names, suffixes, and artifact scopes

These names solve different problems. A namespace does not rename a GitLab pipeline or job, and a job-name suffix does not change an S3 key or an app-list filename.

What each setting names
Setting or value	What changes	When to change it
`gitlab.build_pipeline.workflow_name` and `gitlab.test_pipeline.workflow_name`	Base for `workflow:name` displayed on each child pipeline. Ordinary GCC keeps the base name; Clang displays `<base> (clang)`, and a namespaced variant displays `<base> (gcc / buildv2)` or `<base> (clang / buildv2_clang)`. When the namespace equals the toolchain (for example, an explicit GCC namespace of `gcc`), the name shows it only once: `<base> (gcc)`. Empty/fake-pass pipelines follow the same rule. The current `download-artifacts --pipeline-id` lookup still uses the build setting to compare against a parent trigger job name (see GitLab Artifacts).	A different display name does not isolate jobs, app lists, or artifacts. Do not rely on renaming it to select one of several build variants via `--pipeline-id`.
`gitlab.build_pipeline.job_name_suffix`	Appends to generated build job names, the test-generation job, and the nested test trigger. Their `needs` references use the same suffix.	When combining job definitions that would otherwise have identical names within one pipeline, or when the consuming project’s naming convention requires it. It is not needed merely because GCC and Clang run as separate child pipelines.
`gitlab.test_pipeline.job_name_suffix`	Appends to grouped target-test job names inside the test child pipeline.	When target-test job names would collide with other jobs in that test pipeline.
`gitlab.build_pipeline.parent_pipeline_job_suffix`	Changes references to the parent jobs `generate_build_child_pipeline` and `pipeline_variables` in the generated build YAML.	When those actual parent job names have a suffix; set it to match both parent jobs. It does not rename the parent jobs itself.
Toolchain filename suffix	Adds `_<toolchain>` before the extension of non-GCC build/test child YAML files (for example, `build_child_pipeline_clang.yml`). GCC keeps the original filename.	Automatic when generating for a non-GCC toolchain; use the corresponding filename in `artifacts:paths` and `trigger:include:artifact`.
Effective `IDF_CI_ARTIFACT_NAMESPACE`	Selects app-list filename suffixes and the S3 prefix (for example, `test_related_apps_buildv2_clang.txt` and `<commit_sha>_buildv2_clang/`).	Use a raw variant on the YAML-generator job when isolating artifacts for builds of the same commit; do not apply the toolchain suffix a second time downstream.

For example, job_name_suffix = ":v2" gives a job named build_test_related_apps:v2 and a matching needs reference. It does not give that job buildv2 artifacts: use IDF_CI_ARTIFACT_NAMESPACE=buildv2 for that. If the parent jobs are actually named generate_build_child_pipeline:v2 and pipeline_variables:v2, set parent_pipeline_job_suffix = ":v2" too. The build job suffix and parent job suffix are independent.

Keep both toolchains in separate child pipelines with distinct parent trigger jobs and distinct included YAML filenames, as build-gcc and build-clang do above. Both triggers read artifacts from the same generate_build_child_pipeline job; their include:artifact filenames select different YAML files. They may each contain a job named build_test_related_apps even though their displayed workflow:name differs: GitLab jobs and their native artifacts are scoped to their pipeline/job, not globally keyed by name. Within one parent generator job, however, filenames share a workspace and one GitLab artifact bundle; include both toolchain YAML files and all distinct app lists in artifacts:paths. If you generate two build variants in the same parent job/workspace, give their YAML output files different base names too, or the second invocation overwrites the first. A job-name suffix cannot prevent a filename collision. A project overriding yaml_jinja must render workflow_name itself to adopt these display names.

The generated build jobs and test-generation job use gitlab.artifacts.native.build_job_filepatterns; target-test jobs use gitlab.artifacts.native.test_job_filepatterns. Those GitLab-native artifact paths do not acquire a namespace suffix automatically. The namespace affects the selected app lists and S3 object prefix, not GitLab’s job-artifact storage. Cross-pipeline needs and trigger:include:artifact must still name the correct parent job and generated file. For S3 uploads/downloads of a variant, use the same effective namespace in both jobs; see GitLab Artifacts.

What the build jobs contain

Generated build jobs share .default_build_settings.

Build work is sharded with GitLab parallel when the selected app count is large enough. The shard size is controlled by gitlab.build_pipeline.runs_per_job.

The result has two independent build branches:

one branch that feeds tests
one branch that exists only to build the rest of the requested app set

Only the test-related branch continues into test pipeline generation.

Test child pipeline

The test child pipeline is generated by idf_ci.idf_gitlab.pipeline.test_child_pipeline.

Automatically collected pytest cases are filtered against the manifest for each DUT app. depends_components and depends_filepatterns instead control incremental app discovery; explicitly supplied cases retain their existing behavior. If IDF_CI_SELECT_ALL_PYTEST_CASES=1 and IDF_CI_SELECT_BY_FILTER_EXPR are both set, the filter expression is ignored with a warning, but manifest eligibility still applies.

Instead of creating one job per test case, it groups pytest cases by:

target selector
environment selector
runner tags

Each generated job is named like <target_selector> - <env_selector> and contains:

job tags derived from the grouped test cases
a nodes variable containing the selected pytest node IDs
optional parallel sharding when a group is large enough

All generated test jobs extend .default_test_settings.

Each test job depends on generate_test_child_pipeline so it can download the build artifacts needed for execution.

If no test cases are selected, the generated test child pipeline also falls back to fake_pass.

Set IDF_CI_SKIP_TARGET_TEST_JOBS=1 on the test-child-pipeline generator to skip target-test collection and produce only the passing fake_pass job. An unset value or 0 leaves normal test generation unchanged. This affects the test child pipeline, not the build child pipeline.

idf-ci gitlab pipeline-variables also re-exports explicitly set IDF_CI_SELECT_BY_FILTER_EXPR, IDF_CI_SELECT_ALL_PYTEST_CASES, IDF_CI_SKIP_TARGET_TEST_JOBS, and IDF_CI_SELECT_BY_TARGETS to its dotenv output. Jobs that fetch the parent pipeline_variables artifact through needs can use these controls even when trigger variable inheritance is disabled. Values inferred from the pipeline type or MR metadata take precedence when they set the same key.

The default toolchain is GCC; a manifest must explicitly declare supported_toolchains: [gcc, clang] (or [clang]) for Clang. An enable condition cannot override toolchain admission; disable rejects both builds and tests, while disable_test rejects only tests. The default GCC-only rule applies even when no manifest is found. IDF_CI_SELECT_ALL_PYTEST_CASES does not bypass manifest eligibility, runner tags, nightly settings, or markers. This changes the output of pipelines that previously did not opt into manifest test filtering, and unlisted apps no longer enter Clang pipelines.