Skip to content

Build and Deploy Locally

For local development, we recommend using Tilt and KinD . Tilt is a tool that watches the local repository for changes, and when changes are detected, executes builds and updates Kubernetes resources. KinD (Kubernetes-in-Docker) allows the creation of multi-node Kubernetes clusters on a local machine by using Docker containers as nodes.

The repository contains Tiltfiles for a EuroDaT Trustee-platform cluster together with a client cluster containing simulated clients which can be used fort testing purposes. Our provided Makefile simplifies the setup and maintenance.

Our architecture allows for rapid iteration of builds, image pushes and re-deployment of resources as we use jib in combination with local Docker registries. Image builds and pushes should be a matter of seconds (depending on your local machine).

A caveat of this setup is that builds often trigger base image pulls from Docker Hub. You may run into rate caps if you frequently re-build images. (At the time of writing, Docker Hub limited daily image pulls to 100 per day for anonymous users.)

System requirements

For optimal performance during local building and deployment, it is highly recommended to use a system with a minimum of 32GB RAM.

Local repository setup

Clone the Git repository:

git clone https://gitlab.com/eurodat/trustee-platform.git

Environment setup

On Windows 10

Change the WSL distribution

It might happen that the WSL default distribution is set to one docker installs. In this case Windows won't be able to find the WSL command which leads to issues with Tilt. To fix this, change the default distribution with this command: wsl -s <distro>. If you have Ubuntu-22.04 installed the command would be wsl -s Ubuntu-22.04. You can check the available distributions with wsl -l.

Install Tilt and Tilt dependencies (run with admin privileges)

make prep

Optional: Create access token and log in to Docker

It is possible that you'll run into pull-rate limits if you're restarting Tilt resources frequently within a short amount of time. You can increase this limit by creating an access token for the Docker Hub. For this, open Docker Desktop and click on your user and Account Settings. This will guide you to the Docker Hub page with your profile. Go to Security and click on new access token. Then copy the generated password and open a powershell. Run the command but replace your own Docker name and enter your copied password.

docker login registry-1.docker.io -u <yourdockername>

On Ubuntu Linux

All the necessary installations can be made using the Makefile from the repository. For this follow the steps below, else jump to the next step (make prep).

  • Install "make":
sudo apt install make
  • Install prerequisites (some helpers (Maven, jq, curl, ca-certificates, gnupg) and OpenAPI Generator, Docker, kubectl, Helm, Temurin-17-jdk on Ubuntu:
make prerequisites-Ubuntu
newgrp docker

If you’re running Linux in a virtual machine, it may be necessary to restart the virtual machine for changes to take effect (see Docker for Linux )

  • Install Tilt and the necessary dependencies (KinD, kubectx and ctlptl) and setup local Maven repository with:
make prep
  • To avoid networking problems, add the following entry to /etc/hosts:

172.17.0.1        <FQDN>

Set up the local Maven repository

On Windows, you need to run this script using Git Bash:

./cluster/scripts/publish_argo_client_to_maven_local.sh

Additional, useful tools

Use k9s

  • We recommend k9s for Kubernetes debugging. Among its numerous features, note:
    • painless access to logs, descriptions, and manifests.
    • rapid switching between Kubernetes contexts,
    • decryption of secrets,
    • easily executing a shell in containers,
    • easily managing port forwards.

Sysctl

In case you retrieve the warning "too many open files failed to create fsnotify watcher..." change sysctl limits to:

sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify

You need to adapt this command for Windows, for example so:

wsl -d docker-desktop sh -c "sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify"

Install the pre-commit hook

We use three pre-commit hooks to check our code:

The script cluster/scripts/install-pre-commit-hook.sh installs Talisman and ktlint only for the repository the file was executed from and creates the .git/hooks/pre-commit file. markdownlint-cli2 needs to be installed manually (it requires node.js ). Make sure to run the script from inside the repository you want the hooks to be installed in. If you run Git from a different OS than you ran the installation script from, please rerun the script in that OS.

You must do commits through a command line to see Talisman warnings. However, Talisman errors are shown even by other tools, e.g., IntelliJ interface.

You must correct confidential information caught by Talisman right after it was shown to you by Talisman. It is needed because Talisman checks only git diff, and it will not catch your secrets or passwords again later if you will not do any changes in them. If you push then, that sensitive information will be caught by the Talisman job in a pipeline, but all the history with this information will leak.

For more information refer to this instruction.

Tilt environment setup

The two clusters in the local setup are defined in ./Tiltfile and ./Tiltfile_ext. Most components are rolled out by installing the Helm Charts defined in our repository, with values customised for the local settings, thereby acquired as much dev-prod parity as possible.

The startup and teardown of a local setup is automated with a Makefile. To see all the make targets, you may invoke make or make help.

Create KinD clusters and local Docker registries

make setup

Internal cluster

Start EuroDaT Tilt session (internal cluster):

make start-int

Start without updating the Helm Charts:

make nou=1 start-int

It is possible to exclude (xcl) services or pre-specified groups of services from the Tilt session to speed-up builds and decrease resource consumption:

make xcl="group1 group2 service1" start-int

The groups of service currently available to disable in the internal cluster are:

Group Services
argo-workflows argo-workflows-controller, argo-workflows-server, argo-workflows-config
broker eurodat-controller, eurodat-postgresql, broker-config, broker-ingress, controller-argo
config cert-manager, eurodat-cluster-reflector, eurodat-cluster-reloader, restart-reflector, cluster-config, cert-management, restart-reflector-config, eurodat-certificates, eurodat-network-policies, trust-manager
credential-service eurodat-credential-service, credential-service-config, credential-service-ingress
database-service eurodat-database-service, database-service-config, database-service-ingress
data-plane eurodat-data-plane
ingress ingress-nginx
keycloak keycloak, config-job
kubernetes-secret-generator eurodat-cluster-kubernetes-secret-generator
monitoring grafana, grafana-config, opentelemetry-collector, opentelemetry-collector-config, tempo, tempo-config, loki, loki-config
postgres eurodat-postgresql, postgresql-config
rabbitmq eurodat-rabbitmq, eurodat-rabbitmq-config
transaction-service eurodat-transaction-service, transaction-service-config
transaction-vault eurodat-transaction-vault, eurodat-transaction-vault-config

Note that a service can not be disabled, if an enabled service depends on it. In this case, Tilt overrules the exclusion. For an overview of all internal services and their dependencies refer to the following image (click to enlarge):

Tilt Service Dependencies

In case Tilt does not display the option to start the web GUI, open localhost:10350 in your browser.

External cluster

Once the EuroDat Tilt session has fully started up, start the EuroDaT client Tilt session (external cluster) from a separate shell

cd /path/to/local/repo
make start-ext

Start without updating the Helm Charts:

make nou=1 start-ext

The exclusion of services from the Tilt session is also available for the external cluster:

make xcl="group1 group2 service1" start-ext

The groups of service currently available to disable in the external cluster are:

Group Services
config cert-manager, eurodat-ext-cluster-reflector, eurodat-ext-cluster-reloader, restart-reflector, cluster-config, cert-management, eurodat-ext-certificates,eurodat-ext-network-policies, restart-reflector-config
consumer eurodat-ext-consumer, consumer-config
consumer-ingress consumer-ingress-nginx, eurodat-ext-cluster-consumer-nginx-admission-create, eurodat-ext-cluster-consumer-nginx-admission-patch
provider eurodat-ext-provider, provider-config
provider-ingress provider-ingress-nginx, eurodat-ext-cluster-provider-nginx-admission-create, eurodat-ext-cluster-provider-nginx-admission-patch

Note that a service can not be disabled, if an enabled service depends on it. In this case, Tilt overrules the exclusion. For an overview of all external services and their dependencies refer to the following image (click to enlarge):

Tilt Service Dependencies

The web GUI of the client Tilt session can be accessed by opening localhost:10351 in your browser.

  • To update all the Helm dependencies, use make update.
  • To delete the local clusters, execute make teardown. Note: This should not be necessary between Tilt sessions.
  • The internal and external clusters can be setup and torndown separately with make setup-int/make teardown-int and make setup-ext/make teardown-ext. The teardown commands will only remove the KinD cluster but not the registry.
  • make update-controller updates the controller Helm Chart and the charts that depend on it. The command is automatically executed by the internal Tiltfile if any file in the controller Helm Chart changes.

Running the trustee locally

Scope of the local setup

The local setup described above is relatively complete. It contains the following resources:

  • In both clusters:
    • cert-manager for certificate management
    • ingress-nginx as ingress controller/load balancer. The local Kubernetes clusters are reachable under the DNS name <machine name>.local and certificates are issued to this DNS name. The ingress controller is mapped to port 443 for the trustee cluster and 9443/9444 for the provider/consumer in the external cluster, respectively. The Kubernetes API instances are mapped to the ports 50000 and 50001, respectively.
  • In the trustee cluster:
    • Keycloak for client auth management
    • The following Trustee platform components:
      • control plane, consisting of the controller
    • data-plane
    • flash vaults:
      • ice-cream-app
  • In the client (or external) cluster:
    • One controller each for consumer and provider:

A limitation of the current setup is that it is based on the KinD CNI which does not enforce NetworkPolicy. Thus, ingress and egress rules as defined by these are not respected.

Swagger UI

Once all necessary components are up and running you can use Swagger-UI (available at https://<machine name>.local/control-plane/api/v1/controller/swagger-ui/) to test/try out the system.

Common Issues

  1. When running Tilt, some resources do not start correctly.

This can happen due to race conditions. Please restart all failed resources starting from ingresses (you can do several attempts).

Otherwise, it can happen due to errors. Please check them using Tilt logs for failed jobs and reviewing Kubernetes resources (e.g., in k9s).

  1. In case you retrieve the warning "too many open files failed to create fsnotify watcher...":

    Change sysctl limits for Linux systems:

    sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify
    

    You need to adapt this command for Windows with wsl2, for example so:

    wsl -d docker-desktop sh -c "sysctl -n -w fs.inotify.max_user_watches=10485677 && sysctl -n -w fs.inotify.max_user_instances=512 && sysctl fs.inotify"