Hacking on the CI Templates

This is an outline of how everything fits together in this repository.

This repository contains both the source for the templates and the generated templates that projects use. In addition, the GitLab CI pipeline of the CI Templates produces several public container images that are used when projects employ the CI Templates to build images.

Explanation of the image build process

When the CI Templates CI pipeline runs, the first stage is bootstrap. This stage builds several base images that are used by the .fdo.container-build and .fdo.qemu-build templates. The other stages are to test the distribution-specific CI templates but do not otherwise alter any global state.

To reduce the bandwidth costs, the base images are pushed to https://quay.io and their names are hardcoded into the .fdo.container-build and .fdo.qemu-build templates.

$digraph ci { compound=true; rankdir="TB"; node [shape="box"]; subgraph cluster_quay { label="quay.io"; style="dashed"; q_run_base [ label="qemu run base image"]; q_build_base [ label="qemu build base image"]; c_build_base [ label="container build base image"]; } bootstrap_ci [ label="bootstrap CI stage", shape="component" ]; bootstrap_ci -> c_build_base; bootstrap_ci -> q_build_base; bootstrap_ci -> q_run_base; }$

The bootstrapping stage builds three images:

a container build base image. This image is used exclusively by the .fdo.container-build templates and is capable of building a normal container image.
a qemu run base image. This image is capable of starting a QEMU-compatible raw image file.
a qemu build base image. This image is used exclusively by the .fdo.qemu-build templates and is capable of building a QEMU-compatible raw image file. The .fdo.qemu-build job then places that raw image file inside the qemu run image and publishes that as the image to be run by the project-specific pipelines.

The actual templates then use those https://quay.io images to compose the project-specific images.

In the graphs below, an arrow to an image indicates that image is built, an error from an image indicates that image is used.

$digraph ci { compound=true; rankdir="TB"; node [shape="box"]; subgraph cluster_quay { label="quay.io"; c_build_base [ label="container build base image"]; q_build_base [ label="qemu build base image"]; q_run_base [ label="qemu run base image"]; } subgraph cluster_registry { label="project registry"; style="dashed"; project_img [ label="project image" ]; subgraph cluster_qemuimg { style="solid"; node [ shape="ellipse"; ] label="project qemu image"; imagefile [label="image.qcow2", shape="folder"]; } } container_build [ label=".fdo.container-build", shape="component" ]; qemu_build [ label=".fdo.qemu-build", shape="component" ]; distribution_img [ label=".fdo.distribution_image", shape="component" ]; distribution_img_q [ label=".fdo.distribution_image (qemu)", shape="component" ]; c_build_base -> container_build [style="dashed", label="runs on"]; container_build -> project_img; project_img -> distribution_img [style="dashed", label="runs on"]; q_build_base -> qemu_build [style="dashed", label="runs on"]; q_run_base -> qemu_build [label="uses"]; q_run_base -> imagefile[ lhead=cluster_qemuimg, arrowhead="none", label="copy of", style="dotted" ]; qemu_build -> imagefile; qemu_build -> imagefile [lhead=cluster_qemuimg]; imagefile -> distribution_img_q[ ltail=cluster_qemuimg, style="dashed", label="runs on" ]; }$

In summary, the .fdo.*-build templates use the https://quay.io images to compose project-specific images and store those in the project’s image registry. The .fdo.distribution-image and .fdo.suffixed-image templates then use those images to run the CI jobs.

Note

The images stored in the project registry are independent of the quay.io images.

CI Templates directory layout

The directory structure is as follows:

bootstrap: Scripts used by the bootstrap images. The primary scripts of interest are cbuild, the script that ends up building the various containers, and vmctl a helper script to control the QEMU virtual machines generated with the .fdo.qemu-build templates.
src: The source template files and per-distribution configuration. Templates ending in -ci.tmpl are those used by our CI run to build the base images and test the templates themselves.
templates: The generated templates. Do not edit.
test: Scripts used by the CI to test images.
tools: The ci-fairy tool.

When templates change, make templates generates both the resulting templates/ and the .gitlab-ci.yml file that generates the consumable images and the various CI pipelines to test these images and the templates.

The consumable images are built in the bootstrap phase of the CI pipeline, see the .gitlab-ci/bootstrap-ci.yml file and Explanation of the image build process for details.

Modifying templates

A typical sequence of steps to modify the distribution templates is:

edit the src/*.tmpl files
run make templates
commit and file MR

Where the bootstrap phase or the cbuild and vmctl scripts are modified, new images for https://quay.io must be built:

edit the scripts
bump the respective tag(s) in src/config/ci-globals.yml
run make templates
commit and file MR

Use a pending MR in a different project

When implementing a new feature, it is best to also provide a usage of the currently opened MR in a separate project.

Thanks to gitlab’s magic, users can actually reference any sha in any opened MR even if they are not merged, yet.

However, the cbuild script will not be pushed to the canonical repo, meaning that hackers need to overwrite the URL to have a working use case.

To achieve that, please have a look at any job that makes use of cbuild in the ci-templates local fork.

At the beginning of the logs (alpine:latest@container-build@x86_64 from the alpine child pipeline for example), you’ll see

$ echo $FDO_CBUILD
https://gitlab.freedesktop.org/api/v4/projects/5764/packages/generic/cbuild/sha256-3606152dfd34af076f6c09d24bb8cdf128fce83d07d53d91745c101ec06c7209/cbuild

This URL is the current cbuild that is used by this version of the MR.

Tip

The format is the following:

https://gitlab.../api/v4/projects/{fork_project_id}/packages/generic/cbuild/sha256-{sha256 of cbuild}/cbuild

Then, in your example of usage, in the job that extends .fdo.container-build@distribution, override the FDO_CBUILD variable with the URL above.

A more complete example is the following:

include:
- project: 'freedesktop/ci-templates'
    ref: 62b11fa3f746b38ae413fc936cc6ae96960dd448
    file:
      - '/templates/freebsd.yml'

stages:
 - prep
 - use

freebsd:13.0@qemu-build@x86_64:
  extends:
  - .fdo.qemu-build@freebsd@x86_64
  stage: prep
  variables:
    FDO_CBUILD: https://gitlab.freedesktop.org/api/v4/projects/5764/packages/generic/cbuild/sha256-3606152dfd34af076f6c09d24bb8cdf128fce83d07d53d91745c101ec06c7209/cbuild
    FDO_DISTRIBUTION_PACKAGES: 'wget curl'
    FDO_DISTRIBUTION_VERSION: '13.0'

freebsd:13.0@qemu-check@x86_64:
  extends:
  - .fdo.distribution-image@freebsd
  stage: use
  script:
    - ...