Skip to content

Set Up Gitlab

GitLab serves as our development backbone. It contains the GitLab CI/CD pipeline for automated testing, extensive code quality checks, infrastructure provisioning, application deployment, the terraform remote state to manage the state of the infrastructure provisioning, the docker registry , and the helm package registry.

Create an organization in GitLab and use it to fork code repository and infrastructure as code repository . If you set the project visibility to public, ensure that CI/CD rights are limited for non-members. Enable Operations in your repository. Otherwise, terraform will not be able to save state files.

Configure the CI/CD Pipeline

Before getting started, make sure that only owners and maintainers have the rights to start pipelines.

Add Required CI/CD Variables

You can add the variables at Settings -> CI/CD -> Variables.

Common Deployment CI/CD Variables

Variable Required Purpose Where to find
SONAR_TOKEN Y Token offered by the SonarCloud interface SonarCloud UI
IMG_REGISTRY_USERNAME Y Username to log in to the registry GitLab Access Token name
IMG_REGISTRY_SECRET Y Password to log in to the registry GitLab Access Token value
NIGHTLY_CLEANUP_ACCESS_TOKEN Y GitLab Token to delete outdated nightly artifacts GitLab Access Token value
IAC_REPO_TOKEN Y eurodat-trustee-iac GitLab token for Terraform and git push GitLab Access Token value
NVD_API_KEY N NVD api key to avoid running into rate limits with the OWASP job GitLab Access Token value
UNIQUE_VARIABLES Y List of developer specific values for unique cluster naming Manually created
SMTP_API_KEY Y SMTP server's credentials Manually created
SMTP_API_SECRET_KEY Y SMTP server's credentials Manually created

Additional GCP CI/CD Variables

Variable Required Purpose Where to find
GCP_PROJECT_ID Y The ID of the project on GCP Google Console > Dashboard > Project info --> Project name
GCP_SERVICE_ACCOUNT_NAME Y User or Service Account name - needed for provisioning the k8s cluster Google Console > IAM and admin > Service accounts --> find the name in the list
GOOGLE_CREDENTIALS Y User or Service Account credentials - needed for provisioning the k8s cluster The credentials you've downloaded when creating the service account
STATIC_CLUSTER_IP Y One of the reserved external static IP addresses Google console > VPC network > IP addresses
STATIC_CLUSTER_IP_EXT_CONSUMER Y One of the reserved external static IP addresses Google console > VPC network > IP addresses
STATIC_CLUSTER_IP_EXT_PROVIDER Y One of the reserved external static IP addresses Google console > VPC network > IP addresses
CMEK_ENCRYPTION_KEY_ID Y The id of the customer managed encryption key Google console > Security > Key Management > Find your key > Copy resource name
CLOUD_PROVIDER_TARGET=gcp Y Choose the cloud provider to deploy to Static value

Set up SonarCloud

  1. Before getting started, ensure that you are the owner of the GitLab repository and that you are also added to the Owners group in SonarCloud.

  2. Add every project manually to SonarCloud. (create a project -> Click on "create a project manually") with keys as defined in the pipeline files:

    • $SONARCLOUD_TEAM_IDENTIFIER-app-service
    • $SONARCLOUD_TEAM_IDENTIFIER-backmessaging-service
    • $SONARCLOUD_TEAM_IDENTIFIER-client-controller
    • $SONARCLOUD_TEAM_IDENTIFIER-controller
    • $SONARCLOUD_TEAM_IDENTIFIER-credential-service
    • $SONARCLOUD_TEAM_IDENTIFIER-transaction-service
    • $SONARCLOUD_TEAM_IDENTIFIER-credential-service-client
    • $SONARCLOUD_TEAM_IDENTIFIER-transaction-service-client
    • $SONARCLOUD_TEAM_IDENTIFIER-database-service
    • $SONARCLOUD_TEAM_IDENTIFIER-user-service-client
    • $SONARCLOUD_TEAM_IDENTIFIER-backmessaging-service-client

  3. Every project needs a new code definition which can be set in your organisation.

    • To set a default for your organization, go to Administration -> New Code and pick one (e.g., previous version).
    • If a project has no new code definition, the quality gate will fail for the main pipeline.
    • For projects that were created without a new code definition, a new one can be set similarly using the project administration.
    • If your main branch is not called master, then you need to rename the long-running master branch to your default long-running branch name (e.g., main) for every project.
    • Set up a new default quality gate with lower coverage:
      • Open your Organization page -> Quality Gates
      • While Sonar way quality gate is selected:
        1. Click on copy and set a unique name, click on copy again
        2. Use the Edit button to set the coverage to 70%
        3. Click on Update Condition and then on Set as Default
    • You can look up and change your SONAR_ORGANISATION from Administration -> Organization Settings -> Edit Organization Key

First Pipeline Run

Click on Run Pipeline (top right) in Gitlab-> CI/CD -> Pipelines and select the main branch. Click Run Pipeline.

The first run of the pipeline will provision the kubernetes cluster from scratch and therefore takes longer. The pipeline sometimes also require two or three restarts when ran for the first time.

Additionally, build jobs with SonarCloud will likely throw a QUALITY GATE STATUS: FAILED on the first run. This is due to how SonarCloud requires an initial run for code quality comparisons. These jobs need to simply be restarted to pass.

To observe warnings, use the command kubectl get events --field-selector type=Warning.

An Illustration of the Pipeline Architecture

diagram.

A User-Guide for the EuroDaT Platform Pipeline

On this project, every developer can work on their own cluster if needed. To ensure the resources of the developers don't share names, unique variables were introduced. They are fetched from the UNIQUE_VARIABLES GitLab CI variable. Before the first pipeline run this variable needs to be extended with the developers unique values.

We implemented some options to configure what parts of the pipeline to run. These can be set by adding a keyword between square brackets: [keyword]. The available options are:

  • [full]: Runs the whole pipeline. Basically the same configuration as the main pipeline.
  • [build]: Builds images and deploys to the cluster. Skips the build jobs if the kotlin code hasn't been changed compared to main.
  • [test]: Basically only runs the e2e tests. Often combined with [build] to have a full pipeline but skips the image build if not necessary.
  • [sast]: Runs the sast jobs and the jobs necessary for them.

The pipeline clones the main branch of the EuroDaT IaC repository to use the control-cluster.sh script and terraform files. However, sometimes it is necessary to change something in the IaC repo and then test it in the platform pipeline. To change the branch to clone simply define the variable IAC_BRANCH. This can be done similarly to the options above using curly brackets instead. For example: {IAC_BRANCH=my-branch}. The inject-ci-variables job then injects the variable into the pipeline. Some other things can be configured the same way:

  • CLOUD_PROVIDER_TARGET: Default is gcp.
  • GCP_CLUSTER: Default is dev1. Set it to dev2, dev3, dev4, or dev5 to choose from the predefined FQDN and Static IP sets.
  • DEPLOY_USE_CASE: Can either be "UC2" or "UC3". Changes the environment to provision clusters for the use-cases.

Common Issues

  1. Failed jobs during a pipeline run
    Some jobs can fail due to race conditions (e.g., code quality checks). It helps to just restart them a couple of times then. Please also check logs of failed jobs to see possible errors.