Zuul3 user documentation

Note

This is a lightweight documentation intended to get users started with setting up CI pipelines and jobs. For more insight on what Zuul3 can do, please refer to its upstream documentation and migration documentation.

Zuul is a commit gating system for continuous integration that ensures that repositories are always in a healthy state. It is up to repositories managers to define what constitutes a healthy state: for example passing all unit tests, success in building software, etc.

Software Factory ships with Zuul3, which is a major rewrite of Zuul with many improvements. Most basic concepts remain valid though, so it is advised to read Software Factory’s general documentation on Zuul before proceeding further.

Some of the major changes reflected in Software Factory are:

  • In Zuul3’s terminology, a project is a gated repository
  • Projects must be added to the main configuration in the config repository’s zuulV3 directory
  • Jobs can inherit from the zuul-jobs repository base jobs
  • Pipelines are managed within repositories
  • Custom jobs can be defined within repositories
  • Github was added as a possible repository source for gating projects, enabling Software Factory to act as a third party CI for repositories hosted on Github.

Adding a project to the zuulV3 service

A project file must be submitted to the config repository, under the zuulV3 directory. For example, config/zuulV3/project-name.yaml can contain the following:

- tenant:
    name: local
    source:
      source-name:
        untrusted-projects:
          - project-name
  • Leave the tenant name to local.
  • Replace source-name by the location of the repository: for example, gerrit for Software Factory’s internal gerrit. Other source names, if available, will depend on Software Factory’s configuration.
  • Replace project-name by the repository’s name.

Tip

Why is my project “untrusted” ?

For each source, Zuul defines two categories of projects:

  • Projects listed under config-projects hold configuration parameters for Zuul, for example pipelines or base jobs definitions, or secrets used by these jobs.
  • Projects listed under untrusted-projects are the actual repositories for the software being tested or deployed. Zuul runs jobs for these projects in a restricted environment.

Everything the “config-projects” do is already taken care of by Software Factory in its config and zuul-jobs repositories, meaning you should not have to add any repository under this category. Therefore repositories should always be declared under the “untrusted-projects” category.

After merging this change, the config-update job will reload the zuul scheduler.

Adding a predefined job to a project

A project’s CI configuration is happening in repositories, a project can define jobs and pipelines contents by having a file named .zuul.yaml at the root of the project’s repository:

---
- project:
    name: project-name
    check:
      jobs:
        - linters
    gate:
      jobs:
        - linters
  • name is the name of the project, same as the one defined in ZuulV3’s configuration file earlier
  • check, gate are pipelines defined in Software Factory.

A default deployment of Software Factory comes with the following base jobs:

Name Description
linters Run the bashate, flake8 and yaml linters on relevant files
tox Run tox
tox-py27 Run tox -epy27

Software Factory can be configured to import the openstack-infra/zuul-jobs jobs library; ask your instance operator if this is the case. A list of the jobs in this library can be found here.

A full list of all the jobs that have been built at least once on Software Factory can be accessed at https://<fqdn>/zuul3/local/jobs.html.

Defining a custom job within a project

It is possible to define jobs specific to a project within its repository. This is done in the .zuul.yaml file at the root of the repository. Jobs are based on Ansible playbooks.

For example, the following .zuul.yaml file will define a job called unit-tests to be run in the check pipeline along the linters:

---
- job:
    name: unit-tests
    parent: base
    description: this is running the unit tests for this project
    run: playbooks/unittests.yaml
    nodeset:
      nodes:
        - name: test-node
          label: dib-centos-7

- project:
    name: project-name
    check:
      jobs:
        - unit-tests
        - linters
  • setting parent: base allows this job to inherit from the default pre and post playbooks which are run before and after the custom job’s playbook. These playbooks prepare the work environment and automatically publish artifacts and logs on Software Factory’s log server, so while not mandatory, it is advised to add this setting to make use of Software Factory’s integrations.
  • nodeset defines the nodes that will be spawned to build the job. Label refers to nodepool label definitions, see the nodepool documentation for further details. Name is the name of the node as it will appear in the job’s playbook inventory.

The previous example expects the Ansible playbook “playbooks/unittests.yaml” (or “playbooks/unittests/run.yaml”) to be present in the project’s repository. Here is an example of what this playbook could contain:

---
- hosts: test-node
  tasks:
    - name: install tox package
      yum:
        name: python-tox
        state: present
      become: yes
    - name: run unit tests
      command: tox
      args:
        chdir: {{ zuul.project.src_dir }}/tests

Further documentation can be found online:

Create a secret to be used in jobs

Zuul provides a public key for every project that needs to be used to encrypt secret data. To fetch a given project’s public key:

curl -O https://<fqdn>/zuul3/keys/gerrit/project-name.pub

The encrypt_secret.py tool, from the Zuul repository (branch feature/zuulv3), can be used to create the YAML tree to be pushed in the project .zuul.d/ directory.

./encrypt_secret.py https://<fqdn>/zuul3/ gerrit project-name --infile secret.data --outfile secret.yaml

Then <name> and <fieldname> fields that are placeholders must be replaced in the generated secret.yaml file.

A secret used in a job must be defined in the same project than the job is defined. The user should read carefully the section about secrets.

Web Interface

Zuul comes with the following web interface:

Status

Zuul’s status can be reached at https://<fqdn>/zuul3/local/status.html

This page shows the current buildsets in Zuul’s pipelines. Filtering options are available.

Each buildset can be expanded to show the advancement of its builds. Clicking on a build will open a stream of its logs in real time.

Jobs

Zuul’s Jobs dashboard can be reached at https://<fqdn>/zuul3/local/jobs.html

This page lists all the jobs that have been built at least once by Zuul. Filtering options are available.

Builds

Zuul’s Builds dashboard can be reached at https://<fqdn>/zuul3/local/builds.html

This page lists all the builds and build sets that have completed. Filtering options are available.

Configure a GitHub App

After a GitHub Application is created and configured in Software Factory (see this guide), to add the application to your projects, follow this documentation:

  • Visit the application page, e.g.: https://github.com/apps/my-org-zuul
  • Click “Install”
  • Select your GitHub organisation
  • Select the repositories to install the application on
  • Click “Install”

Then you’ll be redirected to the Setup URL with the instruction to finish the configuration, checkout the Zuul3 user documentation:

  • Update the config repository to add the projects to the zuul main.yaml file.
  • Create a Pull Request to add a .zuul.yaml to your project and verify it works.

Note

Software Factory pre-configures its default check pipeline to trigger jobs according to the pull request status. Status of jobs running in the check pipeline will be reported in the pull request. You might want to define specific pipelines like a github-gate and a github-post. To do so you should propose them in the config repository zuul.d/. Refer to the upstream manual.