CI Templates

The CI templates are a set of Gitlab CI job templates that can be integrated into your .gitlab-ci.yml file. These templates simplify building and re-using container images for the project’s registry.

To make use of the templates, you need to

include the template and
define a job extending the template

The CI templates provide a set of templates per distribution and you need to include the one for the distribution(s) you want to use. The examples below use the Fedora templates.

Why use the CI templates?

With the CI templates you can easily build a persistent container image. Images are identified by a unique tag that allows for them to be used in the actual CI jobs.

So you can say “I want a Fedora 31 container with packages X, Y, Z installed”, tag this with “2020-02-03.0” for the current date (best practice but not required) and that image can be used from all tests. To build a new container or rebuild the existing container with updated packages, simply change the tag in your .gitlab-ci.yml.

Merge request re-use the same container image for CI. The CI templates will take care of the rest. This gives you reproducible test results as all merge requests will use the same container images.

Where a merge request changes the tag, a new container is built for that merge request and this merge request runs CI with the new container image. All other merge requests are unaffected. Only once the merge request has been merged into the upstream project will future merge requests use the new container image.

All this is available with minimal boilerplate - you specify the distribution, version and the packages to install and the templates do the rest.

Including the CI templates

There are two ways of including a template in your project, depending on whether your project is hosted on gitlab.freedesktop.org or hosted elsewhere. In both cases, you should use a specific git sha of the files you want to include - the CI templates do not use version numbers.

Warning

You can use master as git ref but we do not recommend this. Using a specific git sha protects you from unplanned CI failures caused by changes in the CI templates.

Projects hosted on gitlab.freedesktop.org

If your project is hosted on gitlab.freedesktop.org you can include it as follows:

.templates_sha: &templates_sha 123456deadbeef

include:
  - project: 'freedesktop/ci-templates'     # the project to include from
    ref: *templates_sha                 # git ref of that project
    file: '/templates/fedora.yml'       # the actual file to include
  - project: 'freedesktop/ci-templates'
    ref: *templates_sha
    file: '/templates/alpine.yml'

# rest of your .gitlab-ci.yml goes here

The above snippet first defines the git sha of the CI templates we want to include as a YAML anchor. It then includes the Fedora and Alpine templates from the CI templates repository of that specific sha. Using a YAML anchor is recommended to avoid duplication.

You can specify different shas for different files though it is not something we recommend. You can specify any valid git ref (e.g. master) though we recommend that you use a specific sha.

For more information on the include: statement, see the GitLab documentation

Projects not hosted on gitlab.freedesktop.org

If youre project is not hosted on gitlab.freedesktop.org you can use the CI templates as follows.

include:
  - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/123456deadbeef/templates/fedora.yml'
  - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/123456deadbeef/templates/alpine.yml'

# rest of your .gitlab-ci.yml goes here

The above snippets links to the files directly using the git sha.

You can specify different shas for different files though it is not something we recommend. You can specify any valid git ref (e.g. master) though we recommend that you use a specific sha.

For more information on the include: statement, see the GitLab documentation

Extending the template jobs

All jobs provided by the CI templates use a naming scheme in the form .fdo.<type>@<distribution>. They start with a dot (.) and thus do not get invoked by the GitLab CI runners. To make use of them, use extends:.

# include statements go here

myjob:
  extends: .fdo.container-build@fedora
  stage: somestage
  variables:
    FDO_DISTRIBUTION_VERSION: 31
    FDO_DISTRIBUTION_PACKAGES: 'curl wget valgrind'

In the (incomplete) example above a job called myjob is defined to be invoked in the somestage stage (user’s choice). It extends the .fdo.container-build@fedora template (see Building container images). The variables will be used by the CI template to generate the correct container image.

For more information on the extends: statement, see the GitLab documentation

Building container images

The CI templates provide two ways of building a container image: container-build for a normal container and qemu-build for building an image that can be run through QEMU on a KVM-enabled host. For projects without specific hardware interactions the container-build is sufficient.

As the name implies, this builds a container image if it does not already exist. At runtime, it will check your image repository (i.e. the GitLab username/project) for the container image and then the upstream repository.

If the image does not exist in your repository but it does exist upstream, the image is copied to your repository. Thus, even if upstream changes the image later, you have a copy of that image.

If the image does not exist upstream, it is built in your repository. Once your merge request is merged, the image will be rebuilt for the upstream repository.

Note

If you want to force a container build, set the variable FDO_FORCE_REBUILD.

Below is an example on how to build a container. To avoid repetition of boilerplate code, we define a template for the distribution we want to build on and re-use that template to fill in the variables for us where required.

include:
- project: 'freedesktop/ci-templates'
  ref: 12345deadbeef
  file: '/templates/fedora.yml'


# Let's define some stages to get the correct order of jobs.
stages:
  - prep
  - test

variables:
  # The upstream repository path on gitlab.freedesktop.org to check
  # for existing container images.
  FDO_UPSTREAM_REPO: some/path

# A simple template so we only have one place where we need to
# define the Fedora version and the image tag
# Using a date as tag is best practice but it can be any string
# allowed by the GitLab registry.
.myproject.fedora:30:
  variables:
    FDO_DISTRIBUTION_VERSION: 31
    FDO_DISTRIBUTION_TAG: '2020-03-10.0'


# A job to build a Fedora 31 container with valgrind and gcc
# installed.
#
# You must not define script: in this job, it is used by the
# container-build template.
build-fedora-container:
  extends:
  - .fdo.container-build@fedora     # the CI template
  - .myproject.fedora:30            # our template job above
  stage: prep
  variables:
    # Packages to install on the container
    FDO_DISTRIBUTION_PACKAGES: "valgrind gcc"


# The test job for your project. Extending from the CI Templates
# .fdo.distribution-image makes it use the same image we built above.
mytest:
  extends:
  - .fdo.distribution-image@fedora
  - .myproject.fedora:30
  stage: test
  script:
    # FDO_DISTRIBUTION_NAME is set by the distribution-image job.
    # It should be considered read-only
    - echo "Hello world in $FDO_DISTRIBUTION_NAME"

Warning

The .fdo.qemu-build template uses an arch-specific suffix. See Handling multi-arch images for important details.

Building QEMU-capable container images

Some tests need to run in a virtual machine instead of a container. For those, qemu-capable images can be prepared with the CI templates. The templates are identical to the ones shown above but use the term qemu instead of container. The above example thus becomes:

# A job to build a Fedora 31 qemu image with valgrind and gcc
# installed.
#
# You must not define script: in this job, it is used by the
# qemu-build template.
build-fedora-vm:
  extends:
  - .fdo.qemu-build@fedora@x86_64   # the CI template
  - .myproject.fedora:30            # our template job above
  stage: prep
  variables:
    # Packages to install on the vm
    FDO_DISTRIBUTION_PACKAGES: "valgrind gcc"

Warning

The .fdo.qemu-build template uses an arch-specific suffix. See Handling multi-arch images for important details.

Once built, the container image provides the script /app/vmctl start to start the virtual machine. The VM is configured to accept ssh connection and aliased as host vm. Commands can be run on the virtual machine with the /app/vmctl exec helper.

# The test job for your project. Extending from the CI Templates
# .fdo.distribution-image makes it use the same image we built above.
mytest:
  extends:
  - .fdo.distribution-image@fedora
  - .myproject.fedora:30
  stage: test
  script:
    # start the VM. This also sets up ssh/scp to connect to "vm"
    # correctly.
    - /app/vmctl start
    # copy our workspace to the VM
    # The quotes are required to stop the ':' from parsing as yaml
    - scp -r $PWD "vm:"
    # We don't want any failed commands to exit our script until VM
    # cleanup has been completed.
    - set +e
    # run test-command on the VM and create the .success file if it
    # succeeds
    - /app/vmctl exec "cd $CI_PROJECT_NAME ; test-command" && touch .success
    # copy any test results from the VM to our container so we can
    # save them as artifacts
    - scp -r vm:$CI_PROJECT_NAME/test-results.xml .
    # shut down the VM
    - /app/vmctl stop
    # VM cleanup is complete, any command failures now should result in
    # a CI failed job
    - set -e
    # our CI script exit code should match the test command exit status
    - test -e .success || exit 1

Noteworthy in the above is that the actual test command and subsequent VM cleanup are wrapped by a set +e and set -e call. This ensures that any failed call does not immediately terminate the CI job but allows for the proper cleanup of the VM. The .success file is what determines the actual success status of the CI job.

Image labels

Images built with the CI templates have a number of labels set that can be used by jobs or the GitLab setup itself. The example below shows how to access the labels from a CI job using skopeo and jq:

check label:
  extends:
    - .myproject.fedora:30            # our template job above
    - .fdo.distribution_image@fedora
  # We don't want/need to use the actual image we built earlier, we can use
  # any image that has skopeo and jq, or install those as part of
  # script:
  image: any-image-with-skopeo-and-jq
  script:
    # FDO_DISTRIBUTION_IMAGE still has indirections
    - DISTRO_IMAGE=$(eval echo ${FDO_DISTRIBUTION_IMAGE})
    # retrieve the infos from the registry (once)
    - JSON_IMAGE=$(skopeo inspect docker://$DISTRO_IMAGE)

    # Parse the the pipeline_id label
    - IMAGE_PIPELINE_ID=$(echo $JSON_IMAGE | jq -r '.Labels["fdo.pipeline_id"]')

    # If the image was built as part of this pipeline, the image's pipeline
    # ID is the same as the current pipeline ID.
    # This can be used to poke other projects to rebuild dependent images.
    - if [[ x"$IMAGE_PIPELINE_ID" == x"$CI_PIPELINE_ID" ]]; then
          echo "Image was built in this pipeline"
      fi

Note

Extending from .fdo.distribution_image@fedora provides FDO_DISTRIBUTION_IMAGE. We do not actually use the image itself, any image with skopeo and jq will work here.

The labels currently set by the CI templates are as follows:

Image Label	Value
`fdo.pipeline_id`	`$CI_PIPELINE_ID`
`fdo.job_id`	`$CI_JOB_ID`
`fdo.commit`	`$CI_COMMIT_SHA`
`fdo.project`	`$CI_PROJECT_PATH`
`fdo.upstream-repo`	`$FDO_UPSTREAM_REPO` (optional)
`fdo.expires-after`	`$FDO_EXPIRES_AFTER` (optional)

For the values starting with CI_, see the GitLab environment variables documentation

As in the example above, the image’s fdo.pipeline_id can be used to check if an image was built as part of the current pipeline.

Deleting container images

Unfortunately, deleting container images is nontrivial with templates and it requires extra authentication tokens. Use the ci-fairy tool for this task:

delete-containers:
  extends:
   - .fdo.distribution-image@fedora
   - .myproject.fedora:30
  stage: cleanup
  image: golang:alpine
  before_script:
    - apk add python3 git
    - pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates
  script:
    # Go to your Profile, Settings, Access Tokens
    # Create a personal token with 'api' scope, copy the value.
    # Go to CI/CD, Schedules, schedule a new monthly job (or edit the existing one)
    # Define a variable of type File named AUTHFILE. Content is that token
    # value.
    #
    # This example assumes that you want to delete all but the current tag
    - ci-fairy -v --authfile $AUTHFILE delete-image
            --repository $FDO_DISTRIBUTION_NAME/$FDO_DISTRIBUTION_VERSION
            --exclude-tag $FDO_DISTRIBUTION_TAG
  only:
    - schedules

This is a job to run container cleanup on a schedule job. We get $FDO_DISTRIBUTION_NAME by extending .fdo.distribution-image@fedora but since we only need to run a simple python tool, we can just run off a golang:alpine image. All other variables are courtesy of .myproject.fedora:30 (see Extending the template jobs)

Because of restrictions in GitLab, this can only be run with an API token, CI_JOB_TOKEN does not have permissions to delete images.

The ci-fairy command as run here will delete all images in the fedora/30 image repository, excluding the one with the tag 2020-03-10.0.

Handling multi-arch images

The .fdo.container-build and .fdo.qemu-build templates use an arch-specific suffix (@x86_64 or @aarch64) to build for the appropriate architecture. This arch-specific suffix is not encoded in the image name, and a potential pitfall when building identical images for multiple architectures is that those images overwrite each other. This example illustrates the problem:

# DO NOT USE THIS SNIPPET. This example illustrates a bug

.fedora:
  variables:
    FDO_DISTRIBUTION_VERSION: 32
    FDO_DISTRIBUTION_TAG: '2020-11-12.0'

# uses distribution name, version and tag to store the image in the
# registry
build-x86:
  stage: prep
  extends:
    - .fdo.container-build@fedora
    - .fedora

# uses the same distribution name, version and tag to store the image
# in the registry, potentially overwriting the other image.
# if this job runs after the build-x86 job completed, it does nothing
# because an image with the given tag is already in the registry
build-arm:
  stage: prep
  extends:
    - .fdo.container-build@fedora
    - .fedora

# this job runs on whichever image got stored in the registry
run-image:
  stage: build
  extends:
    - .fdo.distribution-image@fedora
    - .fedora
  script:
    - echo "I don't know which image I'm running on"

Both build- jobs produce the same image tag and which image ends up in the registry depends on the (non-deterministic) order the jobs are run and/or completed.

Where multi-arch jobs are required, the arch must be encoded in the image name using FDO_DISTRIBUTION_TAG (or FDO_REPO_SUFFIX):

.fedora:
  variables:
    FDO_DISTRIBUTION_VERSION: 32
    BASE_TAG: "2020-11-12.0"

.fedora-x86:
  extends:
    - .fedora
  variables:
    FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG"

.fedora-arm:
  extends:
    - .fedora
  variables:
    FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG"

build-x86:
  stage: prep
  extends:
    - .fdo.container-build@fedora
    - .fedora-x86
  variables:
    FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG"

build-arm:
  stage: prep
  extends:
    - .fdo.container-build@fedora
    - .fedora-arm
  variables:
    FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG"

.run-image:
  stage: build
  extends:
    - .fdo.distribution-image@fedora
  script:
    - echo "I'm running on $FDO_DISTRIBUTION_TAG"

run-x86:
  extends:
    - .run-image
    - .fedora-x86
  variables:
    FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG"

run-arm:
  extends:
    - .run-image
    - .fedora-arm
  variables:
    FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG"

Templating the .gitlab-ci.yml can ease the maintenance of such pipelines.