Continous Integration Testing¶
In this section you will:
Understand the benefits Continuous Integration.
Configure GitLab CI or Travis-CI, “continuous integration” services, to operate on your repository.
What is CI for?¶
If “Continuous Integration” (CI) is new to you, we refer you to this excellent Software Carpentry tutorial on the subject. To summarize, CI speeds development by checking out your code on a fresh, clean server, installing your software, running the tests, and reporting the results. This helps you ensure that your code will work on your colleague’s computer—that it doesn’t accidentally depend on some local detail of your machine. It also creates a clear, public record of whether the tests passed or failed, so if things are accidentally broken (say, while you are on vacation) you can trace when the breaking change occurred.
GitLab CI configuration¶
The UKAEA GitLab has its own CI servers with their own distinct configuration. An example of such a configuration file is included in the cookiecutter template.
image: python:latest
# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/reference/pip_install/#caching
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
paths:
- .cache/pip
- venv/
# Install package and dependencies
before_script:
- apt update && apt upgrade -y
- python3 -V # Print out python version for debugging
- python3 -m venv venv
- source venv/bin/activate
- python -m pip install .
- python -m pip install -r requirements-dev.txt
stages:
- tests
- docs
pytest-and-coverage:
stage: tests
script:
- coverage run -m pytest
- coverage report -m
- coverage xml
# Patch against https://gitlab.com/gitlab-org/gitlab/-/issues/217664
# Remove these two lines once GitLab is upgraded to 13+
- sed -i "s=<source>.*example</source>=<source>./example</source>=g" coverage.xml
- sed -i "s;filename=\";filename=\"example/;g" coverage.xml
- flake8
coverage: '/TOTAL.+ ([0-9]{1,3}%)/'
artifacts:
reports:
cobertura: coverage.xml
expire_in: 1 week
rules:
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "develop"'
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
- if: '$CI_COMMIT_BRANCH'
build-docs:
stage: docs
script:
- make -C docs html
Using GitLab CI¶
The config file, .gitlab-ci.yml
, included in the cookiecutter template will be
automatically recognised and used to configure the CI service.
Alternatively if a different configuration for the CI is needed then one can be set
up by deleting the existing config and using the Set up CI/CD
button on your
repository homepage. For convenience there are also many templates that can be chosen.
If a non-template-based configuration file is needed then a step-by-step guide for
writing a .gitlab-ci.yml file
is available.
Travis-CI Configuration¶
If instead you are using GitHub, the cookiecutter template also has generated a configuration file for Travis-CI, which is one of several CI services that are free for public open-source projects.
# .travis.yml
language: python
python:
- 3.9
cache:
directories:
- $HOME/.cache/pip
- $HOME/.ccache # https://github.com/travis-ci/travis-ci/issues/5853
install:
# Install this package and the packages listed in requirements.txt.
- pip install .
# Install extra requirements for running tests and building docs.
- pip install -r requirements-dev.txt
script:
- coverage run -m pytest # Run the tests and check for test coverage.
- coverage report -m # Generate test coverage report.
- codecov # Upload the report to codecov.
- flake8 --max-line-length=115 # Enforce code style (but relax line length limit a bit).
- make -C docs html # Build the documentation.
You can customize this to your liking. For example, if you are migrating a
large amount of existing code that is not compliant with PEP8, you may want to
remove the line that does flake8
style-checking.
Activate Travis-CI for Your GitHub Repository¶
Go to https://travis-ci.org and sign in with your GitHub account.
You will be prompted to authorize Travis-CI to access your GitHub account. Authorize it.
You will be redirected to https://travis-ci.org/profile, which shows a list of your GitHub repositories. If necessary, click the “Sync Account” button to refresh that list.
Find your new repository in the list. Click the on/off switch next to its name activate Travis-CI on that repository.
Click the repository name, which will direct you to the list of builds at
https://travis-ci.org/YOUR_GITHUB_USERNAME/YOUR_REPO_NAME/builds
. The list will currently be empty. You’ll see construction cones.The next time you open a pull request or push a new commit to the master branch, Travis-CI will kick off a new build, and that list will update.
Note
If this repository belongs to a GitHub organization (e.g.
http://github.com/NSLS-II) as opposed to a personal user account
(e.g. http://github.com/danielballan) you should follow Steps 3-5
above for the organization’s profile at
https://travis-ci.org/profile/YOUR_GITHUB_ORGANIZATION
. It does no
harm to also activate Travis-CI for your personal fork at
https://travis-ci.org/profile
, but it’s more important to activate it for
the upstream fork associated with the organization.
Code Coverage¶
The example GitLab CI configuration file, .gitlab-ci.yml
runs a number
of commands in the tests
stage concerning code coverage. The
coverage
package is one of the packages pre-populated in requirements-dev.txt
. This
can generate a report of the coverage of the tests that are run, as in what
percentage of lines of code are executed when the tests are run. If this is a
low percentage then there may be bugs hiding in the code which cannot be
identified as the tests do not cover the lines where they may be hiding.
GitLab CI has the ability to parse the reports from coverage
and
produce an overall test coverage badge (see the “Settings > CI > General”
section to find it) as well as test coverage visualisation for merge requests.
Both of these features have already been set up in the .gitlab-ci.yml
file
with the cookie-cutter.
Note
Test coverage visualisation is a bit hacky at the moment. Don’t be surprised if it doesn’t show up or stops working.
Alternatively, GitHub does not come with test coverage reporting out of the box, and you must use the codecov package instead. The Travis-CI file uses a bash script to upload the coverage report, but you will need to sign up to codecov in order to use this. You can use your GitHub account to sign up to codecov.