Build and Deploy Locally

For local development, we recommend using Tilt and KinD . Tilt is a tool that watches the local repository for changes, and when changes are detected, executes builds and updates Kubernetes resources. KinD (Kubernetes-in-Docker) allows the creation of multi-node Kubernetes clusters on a local machine by using Docker containers as nodes.

The repository contains Tiltfiles for a EuroDaT Trustee-platform cluster together with a client cluster containing simulated clients which can be used fort testing purposes. Our provided Makefile simplifies the setup and maintenance.

Our architecture allows for rapid iteration of builds, image pushes and re-deployment of resources as we use jib in combination with local Docker registries. Image builds and pushes should be a matter of seconds (depending on your local machine).

A caveat of this setup is that builds often trigger base image pulls from Docker Hub. You may run into rate caps if you frequently re-build images. (At the time of writing, Docker Hub limited daily image pulls to 100 per day for anonymous users.)

System requirements

For optimal performance during local building and deployment, it is highly recommended to use a system with a minimum of 32GB RAM.

Local repository setup

Clone the Git repository:

git clone https://gitlab.com/eurodat/trustee-platform.git

Environment setup

On Windows 10

Prerequisites:
- Install Git-Bash . Some scripts may not work otherwise.
- Install Chocolatey and Scoop.
- Install make (using Chocolatey).
- Install Docker Desktop .
- Install kubectl and Helm.
- Install Temurin Java 17
- Install Maven
- Install python3
- Install WSL2 Ubuntu 22.04 LTS and install GO language in it using sudo apt install golang
- Install in the wsl shell jq with sudo apt install jq
Optional
- Docker Access Token

Change the WSL distribution

It might happen that the WSL default distribution is set to one docker installs. In this case Windows won't be able to find the WSL command which leads to issues with Tilt. To fix this, change the default distribution with this command: wsl -s <distro>. If you have Ubuntu-22.04 installed the command would be wsl -s Ubuntu-22.04. You can check the available distributions with wsl -l.

Install Tilt and Tilt dependencies (run with admin privileges)

make prep

Optional: Create access token and log in to Docker

It is possible that you'll run into pull-rate limits if you're restarting Tilt resources frequently within a short amount of time. You can increase this limit by creating an access token for the Docker Hub. For this, open Docker Desktop and click on your user and Account Settings. This will guide you to the Docker Hub page with your profile. Go to Security and click on new access token. Then copy the generated password and open a powershell. Run the command but replace your own Docker name and enter your copied password.

docker login registry-1.docker.io -u <yourdockername>

On Ubuntu Linux

All the necessary installations can be made using the Makefile from the repository. For this follow the steps below, else jump to the next step (make prep).

Install "make":

sudo apt install make

Install prerequisites (some helpers (Maven, jq, curl, ca-certificates, gnupg) and OpenAPI Generator, Docker, kubectl, Helm, Temurin-17-jdk on Ubuntu:

make prerequisites-Ubuntu
newgrp docker

If you’re running Linux in a virtual machine, it may be necessary to restart the virtual machine for changes to take effect (see Docker for Linux )

Install Tilt and the necessary dependencies (KinD, kubectx and ctlptl) and setup local Maven repository with:

make prep

To avoid networking problems, add the following entry to /etc/hosts:

172.17.0.1 <FQDN>

Set up the local Maven repository

On Windows, you need to run this script using Git Bash:

./cluster/scripts/publish_argo_client_to_maven_local.sh

Additional, useful tools

Use k9s

We recommend k9s for Kubernetes debugging. Among its numerous features, note:
- painless access to logs, descriptions, and manifests.
- rapid switching between Kubernetes contexts,
- decryption of secrets,
- easily executing a shell in containers,
- easily managing port forwards.

Sysctl

In case you retrieve the warning "too many open files failed to create fsnotify watcher..." change sysctl limits to:

sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify

You need to adapt this command for Windows, for example so:

wsl -d docker-desktop sh -c "sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify"

Install the pre-commit hook

We use three pre-commit hooks to check our code:

ktlint : kotlin files
markdownlint-cli2 : markdown files
Talisman : scans for possibly sensitive data

The script cluster/scripts/install-pre-commit-hook.sh installs Talisman and ktlint only for the repository the file was executed from and creates the .git/hooks/pre-commit file. markdownlint-cli2 needs to be installed manually (it requires node.js ). Make sure to run the script from inside the repository you want the hooks to be installed in. If you run Git from a different OS than you ran the installation script from, please rerun the script in that OS.

You must do commits through a command line to see Talisman warnings. However, Talisman errors are shown even by other tools, e.g., IntelliJ interface.

You must correct confidential information caught by Talisman right after it was shown to you by Talisman. It is needed because Talisman checks only git diff, and it will not catch your secrets or passwords again later if you will not do any changes in them. If you push then, that sensitive information will be caught by the Talisman job in a pipeline, but all the history with this information will leak.

For more information refer to this instruction.

Configure the SMTP Server

For Keycloak to be able to send an email to the user for verification, appropriate settings on the SMTP server must be configured. These settings are defined within the values.yaml file of the eurodat-chart, including smtpApiKey, smtpApiSecretKey, and smtpMailFrom.

smtpMailFrom specifies the email address from which the verification email is sent. smtpApiKey and smtpApiSecretKey, your SMTP server's credentials, are locally overwritten by the get_smtp_keys.py script. To execute this, you have to first complete the env_template and then relocate it to the main directory of the trustee-platform. In addition, the connection settings such as Host, SSL, and the associated Port are configured within the kc_do script. This completes the setting up of your SMTP server for Keycloak to perform email verification of users.

Tilt environment setup

The two clusters in the local setup are defined in ./Tiltfile and ./Tiltfile_ext. Most components are rolled out by installing the Helm Charts defined in our repository, with values customised for the local settings, thereby acquired as much dev-prod parity as possible.

The startup and teardown of a local setup is automated with a Makefile. To see all the make targets, you may invoke make or make help.

Create KinD clusters and local Docker registries

make setup

Internal cluster

Start EuroDaT Tilt session (internal cluster):

make start-int

Start without updating the Helm Charts:

make nou=1 start-int

It is possible to exclude (xcl) services or pre-specified groups of services from the Tilt session to speed-up builds and decrease resource consumption:

make xcl="group1 group2 service1" start-int

The groups of service currently available to disable in the internal cluster are:

Group	Services
argo-workflows	argo-workflows-controller, argo-workflows-server, argo-workflows-config
broker	eurodat-controller, eurodat-postgresql, broker-config, broker-ingress, controller-argo
config	cert-manager, eurodat-cluster-reflector, eurodat-cluster-reloader, restart-reflector, cluster-config, cert-management, restart-reflector-config, eurodat-certificates, eurodat-network-policies, trust-manager
credential-service	eurodat-credential-service, credential-service-config, credential-service-ingress
database-service	eurodat-database-service, database-service-config, database-service-ingress
data-plane	eurodat-data-plane
ingress	ingress-nginx
keycloak	keycloak, keycloak-config-job
kubernetes-secret-generator	eurodat-cluster-kubernetes-secret-generator
monitoring	grafana, grafana-config, opentelemetry-collector, opentelemetry-collector-config, tempo, tempo-config, loki, loki-config
postgres	eurodat-postgresql, postgresql-config
rabbitmq	eurodat-rabbitmq, eurodat-rabbitmq-config
transaction-service	eurodat-transaction-service, transaction-service-config
transaction-vault	eurodat-transaction-vault, eurodat-transaction-vault-config

Note that a service can not be disabled, if an enabled service depends on it. In this case, Tilt overrules the exclusion. For an overview of all internal services and their dependencies refer to the following image (click to enlarge):

In case Tilt does not display the option to start the web GUI, open localhost:10350 in your browser.

External cluster

Once the EuroDat Tilt session has fully started up, start the EuroDaT client Tilt session (external cluster) from a separate shell

cd /path/to/local/repo
make start-ext

Start without updating the Helm Charts:

make nou=1 start-ext

The exclusion of services from the Tilt session is also available for the external cluster:

make xcl="group1 group2 service1" start-ext

The groups of service currently available to disable in the external cluster are:

Group	Services
config	cert-manager, eurodat-ext-cluster-reflector, eurodat-ext-cluster-reloader, restart-reflector, cluster-config, cert-management, eurodat-ext-certificates,eurodat-ext-network-policies, restart-reflector-config
consumer	eurodat-ext-consumer, consumer-config
consumer-ingress	consumer-ingress-nginx, eurodat-ext-cluster-consumer-nginx-admission-create, eurodat-ext-cluster-consumer-nginx-admission-patch
provider	eurodat-ext-provider, provider-config
provider-ingress	provider-ingress-nginx, eurodat-ext-cluster-provider-nginx-admission-create, eurodat-ext-cluster-provider-nginx-admission-patch

Note that a service can not be disabled, if an enabled service depends on it. In this case, Tilt overrules the exclusion. For an overview of all external services and their dependencies refer to the following image (click to enlarge):

The web GUI of the client Tilt session can be accessed by opening localhost:10351 in your browser.

To update all the Helm dependencies, use make update.
To delete the local clusters, execute make teardown. Note: This should not be necessary between Tilt sessions.
The internal and external clusters can be setup and torndown separately with make setup-int/make teardown-int and make setup-ext/make teardown-ext. The teardown commands will only remove the KinD cluster but not the registry.
make update-controller updates the controller Helm Chart and the charts that depend on it. The command is automatically executed by the internal Tiltfile if any file in the controller Helm Chart changes.

Running the trustee locally

Scope of the local setup

The local setup described above is relatively complete. It contains the following resources:

In both clusters:
- cert-manager for certificate management
- ingress-nginx as ingress controller/load balancer. The local Kubernetes clusters are reachable under the DNS name <machine name>.local and certificates are issued to this DNS name. The ingress controller is mapped to port 443 for the trustee cluster and 9443/9444 for the provider/consumer in the external cluster, respectively. The Kubernetes API instances are mapped to the ports 50000 and 50001, respectively.
In the trustee cluster:
- Keycloak for client auth management
- The following Trustee platform components:
  - control plane, consisting of the controller
- data-plane
- flash vaults:
  - ice-cream-app
In the client (or external) cluster:
- One controller each for consumer and provider:

A limitation of the current setup is that it is based on the KinD CNI which does not enforce NetworkPolicy. Thus, ingress and egress rules as defined by these are not respected.

Swagger UI

Once all necessary components are up and running you can use Swagger-UI (available at https://<machine name>.local/control-plane/api/v1/controller/swagger-ui/) to test/try out the system.

Common Issues

When running Tilt, some resources do not start correctly.

This can happen due to race conditions. Please restart all failed resources starting from ingresses (you can do several attempts).

Otherwise, it can happen due to errors. Please check them using Tilt logs for failed jobs and reviewing Kubernetes resources (e.g., in k9s).

In case you retrieve the warning "too many open files failed to create fsnotify watcher...":

Change sysctl limits for Linux systems:

sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify

You need to adapt this command for Windows with wsl2, for example so:

wsl -d docker-desktop sh -c "sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify"