Changes

Page history
Update gitlab_cicd authored by Jens Henrik Goebbert's avatar Jens Henrik Goebbert
Show whitespace changes
Inline Side-by-side
Continuous-Integration-and-Delivery.md 0 → 100644
View page @ 084fafbf
# GitLab Continuous Integration and Delivery (GitLab CI/CD)
### Table of Content:
[[_TOC_]]
## Introduction
The benefits of Continuous Integration (CI) and Continuous Delivery (CD) are huge when automation plays an integral part of your workflow.
Each Git commit or push can trigger a CI/CD pipeline, if you add a [.gitlab-ci.yml](https://docs.gitlab.com/ee/ci/yaml/README.html) file to the root directory of your repository and configure your GitLab project to use a [GitLab Runner](https://docs.gitlab.com/runner/).
This wiki page is about GitLab CI/CD on the concrete example of this project showing all scripts and details you need, but requires a reasonable understanding of version control with [Git](https://git-scm.com/doc).
|**Note:** This wiki is **not** intended to be a general discussion on GitLab CI/CD, **but** explains the use of GitLLab CI/CD on a concrete example, which is this project.|
|:---------------------------------------------------------------------------:|
If you are looking for general documentation on GitLab CI/CD it is recommended to start with the following:
* [GitLab Continuous Integration (GitLab CI/CD)](https://docs.gitlab.com/ee/ci/)
* [GitLab Continuous Integration (GitLab CI/CD) > Getting started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/README.html)
* [Demo: CI/CD with GitLab in action](https://about.gitlab.com/2017/03/13/ci-cd-demo/)
* ....
Anyway, we will introduce a few keywords and concepts in the beginning, which are required to understand the intention of the scripts and settings of our example.
## Git Workflow
![cicd_pipeline_infograph](/uploads/c04edf837829d758f94b4cf870ae407c/cicd_pipeline_infograph.png)
During the development of software, there can be many stages until it's ready for public consumption.
You want to test your code first, then deploy it in a testing or staging environment before you release it to the public.
That way you can prevent bugs not only in your software, but in the deployment process as well.
The process from code development to shipping the software can be defined in a workflow with the major parts of
* **Continuous Integration (CI) pipeline**
* **Continuous Delivery (CD) pipeline**.
How to use Git to accomplish this work in a consistent and productive manner can be described by a [Git Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows). It encourage users to leverage Git effectively and consistently.
For larger groups and complex projects complex workflows like the [Gitflow Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow) might fit the projects needs. Small groups should better stick to simple workflows like the most known [Git Feature Branch Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow).
## Git Feature Branch Workflow
| **Note:** Here we are following for now the simple [Git Feature Branch Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow). |
|:---------------------------------------------------------------------------:|
"The core idea behind the Feature Branch Workflow is that all feature development should take place in a dedicated branch instead of the master branch. This encapsulation makes it easy for multiple developers to work on a particular feature without disturbing the main codebase. It also means the master branch will never contain broken code, which is a huge advantage for continuous integration environments.
Encapsulating feature development also makes it possible to leverage pull requests, which are a way to initiate discussions around a branch. They give other developers the opportunity to sign off on a feature before it gets integrated into the official project. Or, if you get stuck in the middle of a feature, you can open a pull request asking for suggestions from your colleagues. The point is, pull requests make it incredibly easy for your team to comment on each other’s work."
![Workflow_FeatureBranch](/uploads/3b870d7925654b8f88bfbb83bd10f8b5/Workflow_FeatureBranch.png)
[Git Feature Branch Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow) boils down to:
* **Feature Branches**
All the development happens in feature branches.
For each commit into a feature branch the branch is deployed to a ''testing environment''.
* **Master**
The master branch is a constantly shippable product.
Whenever a feature branch is merged to master an automated deployment is triggered to ''staging environment''(s).
It ensures that the master branch is always deployable and gives a stage to test things out before shipping them into production.
* **Tags**
Tags are releases to make production deployments.
Whenever a tag is set an automated deployment is triggered to ''production environment''(s).
The software is packaged and shipped.
| **Note:** Ensure that the master branch is constantly shippable |
|:---------------------------------------------------------------------------:|
How can we ensure that? We need to analyse, build and test the code as best as possible.
And we need to do these checks **before** we merge a feature branch into master.
| **Note:** Only merge a feature branch into master if it passes all checks without any error. |
|:---------------------------------------------------------------------------:|
**But ...**
You do not want to do these checks for every [commit](https://git-scm.com/docs/git-commit) or [merge request](https://docs.gitlab.com/ee/user/project/merge_requests/) manually.
Actually, you do not want to think about it at all and get these checks done fully automatically.
## GitLab Runner
[GitLab Runners](https://docs.gitlab.com/runner/) are used to run all your checks and even more. It will do what you tell it to do in the [projects .gitlab-ci.yml](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/.gitlab-ci.yml) file.
A GitLab runner is a program, which runs on a certain machine waiting for a CI/CD Pipeline to execute. A project can connect to multiple GitLab Runners on different machines to run the CI/CD Pipeline. It only has to know the runner´s URL and its token to connect and authorize.
You can also setup a Docker image, which is started by your GitLab Runner for each run of your CI/CD Pipeline.
In this project we make use of this Docker support inside the GitLab Runner. This way we ensure to have an always working environment to run the CI/CD Pipeline of the project with all the needed tools installed.
| **Note:** Check the Dockerfile of this project [HERE](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/env-testing/ci_docker_centos7/Dockerfile).
|:---------------------------------------------------------------------------:|
Independent of the project, the GitLab Runner must be setup to support this Docker image. In our case, we are using a shared GitLab Runner, which is setup by the GitLab administrator. To use the Docker image the administrator of this shared GitLab Runner has added the Docker image to the runner´s setup for us. The name the administrator gave to this Docker image is used in the projects .gitlab-ci.yml [HERE](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/.gitlab-ci.yml#L2).
# Continuous Integration (CI) Pipeline
| **Note:** GitLab's Continuous Integration (CI) Pipeline can do any automatic checks for you defined in [.gitlab-ci.yml.](https://docs.gitlab.com/ee/ci/yaml/README.html).
|:---------------------------------------------------------------------------:|
A [GitLab CI Pipeline](https://docs.gitlab.com/ee/ci/pipelines.html) is a group of jobs that get executed in stages. All of the jobs in a stage are executed in parallel, and if they all succeed, the pipeline moves on to the next stage. You can access the pipelines page in the project's Pipelines tab.
| **Note:** Check the .gitlab-ci.yml file of this project [HERE](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/.gitlab-ci.yml).
|:---------------------------------------------------------------------------:|
Our pipeline in this project looks like this. Check it out yourself [HERE](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/pipelines).
![GitLab-Pipeline](/uploads/d6711cf1ec4cce45aacf58cbde65f090/GitLab-Pipeline.png)
## CI Jobs
The first thing you need to do is defining these checks (or 'jobs' how GitLab calls them).
All of these CI Jobs are run through CI Scripts written for bash. They can be directly embedded in the .gitlab-ci.yml file, which can make the file quite large und difficult to read. Therefore, in this project they are written as separated shell-scripts with one file for each CI Job. These CI Scripts are then called from within .gitlab-ci.yml. All these CI Scripts are located in this project for better readability in a [projects CI folder](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/tree/master/ci).
### CI Scripts
As an example for CI Scripts you can have a look at the [build-serial.sh](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/build-serial.sh) script in the following:
```sh
#!/bin/bash
source $(cd "$(dirname "$0")"; pwd -P)/ci_funcs.sh
# create badge
create_badge "${BADGE_FILENAME}" build-serial unknown "--color=#808080"
[[ ${BADGE_ONLY} ]] && pushbadge_exit "${BADGE_FILENAME}" 0
###### build ######
mkdir -p build && cd build
if [[ $? == 0 ]]; then
# configure & build
${CMAKE} -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_MPI_PARALLEL=OFF .. && \
make -j ${THREADS}
# create badge
if [[ $? == 0 ]]; then
create_badge "${BADGE_FILENAME}" build-serial passed --color=green
pushbadge_exit "${BADGE_FILENAME}" 0
fi
fi
create_badge "${BADGE_FILENAME}" build-serial failed --color=red
pushbadge_exit "${BADGE_FILENAME}" 1
```
As you can see, the structure of all CI scripts in this project are almost the same to support badges:
1. a "unknown" badge is created
2. the CI job is executed
3. the CI script exits with 0 on success and 1 on error and the badge is updated to that effect.
What checks could we do automatically and how could we group these checks?
Here is my list (of course with no guarantee of completeness).
#### Static Analysis
* code style
* with [clang-format](https://clang.llvm.org/docs/ClangFormat.html)
and options from [.clang-format file](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/.clang-format)
started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/analysis-clangformat.sh)
* code quality
* with [cppcheck](http://cppcheck.sourceforge.net/) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/analysis-cppcheck.sh)
* with [clang-check](https://clang.llvm.org/docs/ClangCheck.html) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/analysis-clangcheck.sh)
* with [clang-tidy](http://clang.llvm.org/extra/clang-tidy/) - **most important** -
and options from [.clang-tidy file](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/.clang-tidy) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/analysis-clangtidy.sh)
* check-out the [clang-tidy checks](https://clang.llvm.org/extra/clang-tidy/checks/list.html) to get impressed
#### Build Tests
* different compiler, architectures, build options through [CMake](https://cmake.org/)
* with single threaded started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/build-serial.sh)
* with MPI with [OpenMPI](https://www.open-mpi.org/) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/build-openmpi.sh)
#### Dynamic Analysis
* unit tests
* with single threaded [Boost.Test](https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/index.html) and integrated into [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/test/CMakeLists.txt#L9) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/test-serial.sh)
* with MPI-enabled [Boost.Test](https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/index.html) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/test-openmpi.sh) and integrated into [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/test/CMakeLists.txt#L9)
* integration tests
* memory errors
* with [Valgrind/Memcheck](http://valgrind.org/docs/manual/mc-manual.html) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/test-memcheck.sh) and integrated into [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/test/CMakeLists.txt#L119)
* thread synchronisation errors
* with [Valgrind/Helgrind](http://valgrind.org/docs/manual/hg-manual.html) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/test-helgrind.sh) and integrated into the [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/test/CMakeLists.txt#L119)
* test coverage
* with gcc + [gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html) + [gcovr](https://gcovr.com/) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/coverage-test.sh) and integrated into [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/test/CMakeLists.txt#L91)
#### Documentation
* create test coverage documentation
* with gcc + [gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html) + [gcovr](https://gcovr.com/) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/coverage-test-html.sh) and integrated into [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/test/CMakeLists.txt#L91)
* create code documentation from source
* with [doxygen](http://www.doxygen.nl/) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/deploy-doxygenhtml.sh) and integrated into [THIS CMake script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/doc/CMakeLists.txt#L10)
* code documentation coverage
* with [doxy-coverage](https://github.com/alobbs/doxy-coverage) started by [THIS project CI script](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/coverage-doc.sh)
### CI Feedback
A CI Job can return its results in different ways to the user.
1. Each CI Job shows its **Status** depending on the exit code of its CI Script.
2. Each CI Job stores its **Console Output**.
3. Each CI Job can save files created by the CI Script as so called **Artifacts**.
4. For this project, each CI Job can generate a so called **Badge**, which is an icon to be shown on the project page and Readme.md.
In this project we are using all of these possibilities.
#### Status
The status of the CI pipeline with all its CI Jobs is shown through the CI/CD->Pipeline tab of the project.
![GitLab-Pipeline__Feedback](/uploads/bbf3e9b14c3ce351fcb5e7d07b7fa69b/GitLab-Pipeline__Feedback.png)
#### Console Output
If one clicks on one of the CI Jobs on the CI/CD->Pipeline tab, one can see the console output of its run.
![GitLab-Pipeline__Feedback2](/uploads/9bb0cdf6f84ba22b1f621dadfa09ad78/GitLab-Pipeline__Feedback2.png)
#### Artifacts
In the same view, which shows the console output one can also download/browse files created by a CI Job, if available.
![GitLab-Pipeline__Feedback3](/uploads/ab5ca9b69d8430b4b683cad88a646113/GitLab-Pipeline__Feedback3.png)
#### Badges
On the projects front-page badges show the status and more of the most important CI Jobs of the laste pipeline run on branch master.
![GitLab-Pipeline__Feedback4](/uploads/3f681c5f7e0603c35fa2d657f2f7cdb0/GitLab-Pipeline__Feedback4.png)
A good support for automatic generaded badges came out to be more complicated than expected (you can read about the reasons in detail [HERE](https://gitlab.com/gitlab-org/gitlab-ce/issues/50603)), but the final solution seems to be nice and good working. This solution requires an Access Token with api access to the projects repository. This Access Token can be created by any user with permissions to create/read/write/delete new branches of the projects repository.
Access Tokens can be created in the GitLab´s profile settings of that user: Settings->Access Tokens->Scopes "api".
To ensure, that the Access Tokens are not readable by everyone who has access to the GitLab repository,
the Access Tokens need to be protected and not stored in any repository file. Instead of Git, GitLab can store the Access Token at a protected place, which is only readable to users with maintainer-permissions. Therefore, create a new Access Token and store it in Project->Settings->CI/CD->Variables as **Protected Variable**. For this project the variable name of the token is CI_BADGE_TOKEN.
| **Note:** Inside the .gitlab-ci.yml file the protected variables are available for the CI Scripts ([EXAMPLE](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/.gitlab-ci.yml#L80)).|
|:---------------------------------------------------------------------------:|
Badges are generated with [anybadge](https://pypi.org/project/anybadge/) by the [THIS bash function](https://gitlab.version.fz-juelich.de/vis/jusense-cicd/blob/master/ci/ci_funcs.sh#L3). It is stored in a temporary branch with the name tmp_<PIPELINE_NAME>, which is deleted when the execution of the CI pipeline has finished. Before, the generated badges of all CI Jobs are copied by the last CI Job "pages" to the artifact "pages".
The artifact "pages" is published online by [GitLab Pages](https://docs.gitlab.com/ee/user/project/pages/). Therefore, for all CI Jobs a fixed URL is available to the latest generated badge.
An example URL is: http://vis.pages.jsc.fz-juelich.de/jusense-cicd/badges/badge_build-openmpi_%{default_branch}.svg
To include the badges to your project front-page, they can be set in Project->Settings->Badges like this:
![GitLab-Pipeline__SetBadges](/uploads/630320add68c17e5c1c3a2335af1e8af/GitLab-Pipeline__SetBadges.png)
# Continuous Delivery (CD) Pipeline
... to be continued ...