Continous Integration Testing

In this section you will:

  • Understand the benefits Continuous Integration.

  • Configure GitLab CI or Travis-CI, “continuous integration” services, to operate on your repository.

What is CI for?

If “Continuous Integration” (CI) is new to you, we refer you to this excellent Software Carpentry tutorial on the subject. To summarize, CI speeds development by checking out your code on a fresh, clean server, installing your software, running the tests, and reporting the results. This helps you ensure that your code will work on your colleague’s computer—that it doesn’t accidentally depend on some local detail of your machine. It also creates a clear, public record of whether the tests passed or failed, so if things are accidentally broken (say, while you are on vacation) you can trace when the breaking change occurred.

GitLab CI configuration

The UKAEA GitLab has its own CI servers with their own distinct configuration. An example of such a configuration file is included in the cookiecutter template.

image: python:latest

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/reference/pip_install/#caching
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
  paths:
    - .cache/pip
    - venv/

# Install package and dependencies
before_script:
  - apt update && apt upgrade -y
  - python3 -V               # Print out python version for debugging
  - python3 -m venv venv
  - source venv/bin/activate
  - python -m pip install .
  - python -m pip install -r requirements-dev.txt

stages:
  - tests
  - docs

pytest-and-coverage:
  stage: tests
  script:
    - coverage run -m pytest
    - coverage report -m
    - coverage xml
    # Patch against https://gitlab.com/gitlab-org/gitlab/-/issues/217664
    # Remove these two lines once GitLab is upgraded to 13+
    - sed -i "s=<source>.*example</source>=<source>./example</source>=g" coverage.xml
    - sed -i "s;filename=\";filename=\"example/;g" coverage.xml
    - flake8
  coverage: '/TOTAL.+ ([0-9]{1,3}%)/'
  artifacts:
    reports:
      cobertura: coverage.xml
    expire_in: 1 week
  rules:
    - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "develop"'
    - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
    - if: '$CI_COMMIT_BRANCH' 

build-docs:
  stage: docs
  script:
    - make -C docs html

Using GitLab CI

The config file, .gitlab-ci.yml, included in the cookiecutter template will be automatically recognised and used to configure the CI service. Alternatively if a different configuration for the CI is needed then one can be set up by deleting the existing config and using the Set up CI/CD button on your repository homepage. For convenience there are also many templates that can be chosen. If a non-template-based configuration file is needed then a step-by-step guide for writing a .gitlab-ci.yml file is available.

Travis-CI Configuration

If instead you are using GitHub, the cookiecutter template also has generated a configuration file for Travis-CI, which is one of several CI services that are free for public open-source projects.

# .travis.yml

language: python
python:
  - 3.9
cache:
  directories:
    - $HOME/.cache/pip
    - $HOME/.ccache  # https://github.com/travis-ci/travis-ci/issues/5853

install:
  # Install this package and the packages listed in requirements.txt.
  - pip install .
  # Install extra requirements for running tests and building docs.
  - pip install -r requirements-dev.txt

script:
  - coverage run -m pytest  # Run the tests and check for test coverage.
  - coverage report -m  # Generate test coverage report.
  - codecov  # Upload the report to codecov.
  - flake8 --max-line-length=115  # Enforce code style (but relax line length limit a bit).
  - make -C docs html  # Build the documentation.

You can customize this to your liking. For example, if you are migrating a large amount of existing code that is not compliant with PEP8, you may want to remove the line that does flake8 style-checking.

Activate Travis-CI for Your GitHub Repository

  1. Go to https://travis-ci.org and sign in with your GitHub account.

  2. You will be prompted to authorize Travis-CI to access your GitHub account. Authorize it.

  3. You will be redirected to https://travis-ci.org/profile, which shows a list of your GitHub repositories. If necessary, click the “Sync Account” button to refresh that list.

  4. Find your new repository in the list. Click the on/off switch next to its name activate Travis-CI on that repository.

  5. Click the repository name, which will direct you to the list of builds at https://travis-ci.org/YOUR_GITHUB_USERNAME/YOUR_REPO_NAME/builds. The list will currently be empty. You’ll see construction cones.

  6. The next time you open a pull request or push a new commit to the master branch, Travis-CI will kick off a new build, and that list will update.

Note

If this repository belongs to a GitHub organization (e.g. http://github.com/NSLS-II) as opposed to a personal user account (e.g. http://github.com/danielballan) you should follow Steps 3-5 above for the organization’s profile at https://travis-ci.org/profile/YOUR_GITHUB_ORGANIZATION. It does no harm to also activate Travis-CI for your personal fork at https://travis-ci.org/profile, but it’s more important to activate it for the upstream fork associated with the organization.

Code Coverage

The example GitLab CI configuration file, .gitlab-ci.yml runs a number of commands in the tests stage concerning code coverage. The coverage package is one of the packages pre-populated in requirements-dev.txt. This can generate a report of the coverage of the tests that are run, as in what percentage of lines of code are executed when the tests are run. If this is a low percentage then there may be bugs hiding in the code which cannot be identified as the tests do not cover the lines where they may be hiding.

GitLab CI has the ability to parse the reports from coverage and produce an overall test coverage badge (see the “Settings > CI > General” section to find it) as well as test coverage visualisation for merge requests. Both of these features have already been set up in the .gitlab-ci.yml file with the cookie-cutter.

Note

Test coverage visualisation is a bit hacky at the moment. Don’t be surprised if it doesn’t show up or stops working.

Alternatively, GitHub does not come with test coverage reporting out of the box, and you must use the codecov package instead. The Travis-CI file uses a bash script to upload the coverage report, but you will need to sign up to codecov in order to use this. You can use your GitHub account to sign up to codecov.