EuroDat Frontend

Architecture

Design patterns

The EuroDaT platform frontend is based on Angular and follows the Angular style guide. The application is structured in modules, components, services, and models. Angular is a hierarchical component-based framework, and we make heavy use of that to better organize the frontend code for development, testing and deployment purposes.

The EuroDaT platform frontend represents a private subset of the whole EuroDaT's organisational website and is only accessible to authenticated EuroDaT participants. EuroDaT's organisational website (https://eurodat.org) is managed and provided by the TYPO3 CMS containing a lot of static sites. The EuroDaT platform frontend is completely independent of TYPO3 and is accessible on the subdomain (https://app.eurodat.org).

The following chart provides an overview of the EuroDaT platform frontend architecture:

• We use a single Angular workspace (frontend/eurodat-app) to manage the EuroDaT frontend development
• The Angular workspace consists of two Angular projects and an Angular library folder, i.e.
  • EuroDaT host project (projects/host)
  • EuroDaT web component projects (embedded/<project name>)
  • Angular libraries (libraries/<library name>)

The EuroDaT host project provides headers including menu items and a footer. By bootstrapping the EuroDaT frontend components it enables the development and testing of the web components with ng serve host and Cypress, respectively. Both in tilt and pipeline, the host project will be deployed to be end-to-end tested with Cypress.

The EuroDaT web component projects contain the actual EuroDaT frontend parts. As those projects consists of Custom Angular HTML Elements created with createApplication they are not meant to be developed or tested with ng serve or Cypress directly. Instead, they are embedded in the host project. A new Angular Element which is to be embedded in host project requires a new Angular component in the host project. Though being Angular Elements aka. web components, you can fully make use of all Angular features including hierarchical component-based architecture, services, and models (see admin dashboard project as an example).

Both projects can share code like components and services using Angular libraries.

Operations

Deployment

In Kubernetes, the mock host frontend is deployed in its own namespace frontend. The page is served using the nginx web server.

Which paths are routed to the web page?

All traffic whose path is not explicitly defined in another ingress will be routed to this page. This is achieved by using the path / in the ingress. If any other ingress defines a different path, that path will be prioritized over / since it then has the same prefix and is longer.

Local development environment

Prerequisites

If you don't have Angular CLI installed, please install it with npm install -g @angular/cli. Once done, please install required packages with npm install. Make sure that you use Angular version 17 or above!

tilt

For local development we recommend a more streamlined set of tilt resources which ignores most of the backend services and third party components:

make start dev-profile=frontend

This includes only a subset of tilt resources in contrast to the fully fledged setup.

If the frontend development evolves, and you need more backend services (e.g. contract service), you can always switch to the full setup with make start or adjust the list of excluded services.

Local Development with ng serve

Please create the environment.standalone.ts file in the environments folder like:

export const environment = {
    clusterUrl: 'https://YOUR_FQDN/', // This URL must be ended with an slash
    keycloakUrl: 'https://YOUR_FQDN/auth',
    keycloakRealm: 'eurodat-frontend',
    keycloakClient: 'eurodat-app',
    useKeycloak: '', // set to noKeycloak in order to disable keycloak entirely
    eurodatAppCssSource: 'https://eurodat.org/fileadmin/templates/css/styles.css',
}

or use the environment.standalone.template.ts as a template.

Run ng serve host to test the entire EuroDaT app. Navigate to http://localhost:4200/. The application will automatically reload if you change any of the source files (both host project and web component projects).

NPM tool window in IntelliJ IDE

For frontend development, some pre-defined npm/npx scripts are available in package.json for testing, launching cypress or building. IntelliJ offers a nice one-click-solution to run these scripts: NPM tool window.

Here is a table of the scripts currently available in the package.json file:

Script	Command	Description
npm install	npm install	Installs all dependencies listed in the package.json file. This is usually required if somebody has updated the dependencies or you start from a clean working copy.
start-eurodat-app	ng serve host	Starts the Angular development server for the entire EuroDaT app.
build-prod-eurodat-app	ng build host --configuration production --output-hashing none --base-href ./	Builds the host configuration of the Angular application for production with output hashing disabled and base href set to ./.
build-prod-eurodat-app-local	ng build host --configuration production_local --output-hashing none --base-href ./ && mkdir -p dist/host/keycloak && touch dist/host/keycloak/env.js	Similar to build-prod-eurodat-app, but it also creates a env.js file in the dist/host/keycloak directory. This allows local production build testing in a Apache2/nginx local web server.
test-component-test	cypress run --component	Runs the component tests using Cypress.
test-coverage-summary	nyc report --reporter=text-summary	Generates a summary of the test coverage using nyc.
test-check-coverage	nyc check-coverage	Checks if the test coverage meets the thresholds specified in the nyc configuration.
test-component-run-all	npm run test-component-test && npm run test-check-coverage && npm run test-coverage-summary	Runs all component tests and coverage checks.
cypress	cypress open	Opens the Cypress test runner.
eslint	eslint . --fix	Runs ESLint on the project and automatically fixes any fixable issues.
eslint-ci	eslint .	Runs ESLint on the project without automatically fixing issues. This is typically used in a continuous integration environment.
prettier	prettier . --write	Runs Prettier on the project and automatically formats any files that don't adhere to the Prettier configuration.
prettier-ci	prettier . --check	Runs Prettier on the project without automatically formatting files. This is typically used in a continuous integration environment.
stylelint	stylelint "**/*.scss" "!**/dist/**" --fix	Runs Stylelint on all SCSS files in the project (excluding those in the dist directory) and automatically fix any fixable issues.
stylelint-ci	stylelint "**/*.scss" "!**/dist/**"	Runs Stylelint on all SCSS files in the project (excluding those in the dist directory) without automatically fixing issues. This is typically used in a continuous integration environment.
htmlhint	npx htmlhint . --ignore="**/coverage-component/**, **/cypress/**, **/dist/**"	Runs HTMLHint on the project, ignoring certain directories.
firefox-e2e	cypress run --browser firefox --headless --reporter junit	Runs end-to-end tests using Cypress with Firefox in headless mode and the JUnit reporter.
firefox-component-test	cypress run --component --browser firefox --headless --reporter junit	Runs component tests using Cypress with Firefox in headless mode and the JUnit reporter.
chrome-e2e	cypress run --browser chrome --headless --reporter junit	Runs end-to-end tests using Cypress with Chrome in headless mode and the JUnit reporter.
chrome-component-test	cypress run --component --browser chrome --headless --reporter junit	Runs component tests using Cypress with Chrome in headless mode and the JUnit reporter.
edge-e2e	cypress run --browser edge --headless --reporter junit	Runs end-to-end tests using Cypress with Edge in headless mode and the JUnit reporter.
edge-component-test	cypress run --component --browser edge --headless --reporter junit	Runs component tests using Cypress with Edge in headless mode and the JUnit reporter.

Note: Make sure to use git bash as default runner for npm scripts on Windows: How to set GitBash as the default NPM script runner on Windows

Code scaffolding

Generate a new Angular project to be embedded in the host project

As all Angular projects share the same workspace and thus, NPM packages, there is no need to install the Angular Elements package for the new added Angular projects (use npm i @angular/elements in case it is not installed, yet). The Angular Elements package is required to generate web components from Angular components.

Instead, we can start right away to generate a new Angular project with the following command:

ng generate application test --standalone=true --ssr=false --inline-style=false --inline-template=false --prefix=<prefix-name> --project-root=projects/embedded/<project-name>> --style=scss --routing=false --skipTests=true

We will use the --standalone=true flag to generate a standalone Angular project that can embedded in the host project. The --ssr=false flag is used to disable server-side rendering as it is not needed for the component. The --inline-style=false and --inline-template=false flags are used to generate separate style and template files. The --prefix=<prefix-name> flag is used to set the prefix for the generated components. The --project-root=projects/embedded/<project-name> flag is used to set the project root directory. The --style=scss flag is used to set the style file extension to SCSS. The --routing=false flag is used to disable routing as it is not needed for the component. The --skipTests=true flag is used to skip generating test files as we do not use Jest, Karma and Jasmine but Cypress for testing.

However, Angular CLI does not provide any flags to generate the project as web component out-of-the-box. Thus, we have to take care of that after the project has been created.

• Clear all demo content auto-generated from the app.component.html file.
• Remove all auto-generated code from the main.ts file:

import { bootstrapApplication } from '@angular/platform-browser';
import { appConfig } from './app/app.config';
import { AppComponent } from './app/app.component';

bootstrapApplication(AppComponent, appConfig)
.catch((err) => console.error(err));

• Add the following code to the main.ts file:

import { createCustomElement } from '@angular/elements';
import { createApplication } from '@angular/platform-browser';
import { AppComponent } from './app/app.component';

(async () => {

  const app = await createApplication({
    providers: [
      /* your global providers here */
    ],
  });

  const <web component name> = createCustomElement(AppComponent, {
    injector: app.injector,
  });

  customElements.define('<custom HTML tag to be used for the web component>', <web component name>);

})();

Though we have enabled client-side rendering, we still have to take care of some settings in the angular.json file. Search for the newly added <project-name> and set the builder key to browser-esbuild which also requires to rename the browser key to main:

"<project-name>": {
      "projectType": "application",
      "schematics": {
        "@schematics/angular:component": {
          "style": "scss"
        }
      },
      "root": "projects/embedded/<project-name>",
      "sourceRoot": "projects/embedded/<project-name>/src",
      "prefix": "<prefix-name>",
      "architect": {
        "build": {
          "builder": "@angular-devkit/build-angular:browser-esbuild",
          "options": {
            "outputPath": "dist/<project-name>n",
            "index": "projects/embedded/<project-name>/src/index.html",
            "main": "projects/embedded/<project-name>/src/main.ts",
            "polyfills": [
              "zone.js"
            ]

After that ng serve <project name> renders a blank page as due to the custom element the Angular application is not bootstrapped anymore. For that reason, we provide a developer only host Angular project which is a simple Angular application that embeds the custom element and allows us to develop the frontend components properly using ng serve host. Another way to render the components is by using a local apache/ngnix server where the component can be displayed independently of any integration, allowing developers to test them in isolation from the main application.

//localhost:orhttp(s)://127.0.0.1:. Some tools also supports virtual hosts which can be used to access the components by a custom domain likehttp(s)://eurodat.local:`

Generate a new Angular component to host the frontend components

Every new web component requires a new Angular component in the Angular host project to be developed and tested properly. Create a new component with the following command:

ng generate component <desired-subfolder>/<component-name> --standalone=true --inline-style=false --inline-template=false --prefix=<prefix-name> --project=host --style=scss

The prefix <desired-subfolder>/ on <component-name> ensures that the component will be placed in the specific subfolder of the Angular project (app/<desired-subfolder>/<component-name>). The --standalone=true flag is used to generate a standalone component to follow the latest Angular design patterns. The --inline-style=false and --inline-template=false flags are used to generate separate style and template files. The --prefix=prefix-name flag is used to set the prefix for the generated component. The --project=host flag is used to set the project where the component should be generated. The --style=scss flag is used to set the style file extension to SCSS. Delete the respective <component-name>.spec.ts after component generation, as it is not used for testing.

To make the new component use the before created web component, we have to add the following code as last line to the <component-name>.component.html

<<prefix-name-web-component>-root></<prefix-name-web-component>>

and import the component within <component-name>.component.ts files, respectively:

import {Component} from '@angular/core';
import {AppComponent as

<component prefix
name > AppComponent
}
from
'../../../../../embedded/<project-name>/src/app/app.component';

@Component({
    selector: 'app-dashboard',
    standalone: true,
    imports: [<component prefix name > AppComponent],
    templateUrl: './dashboard.component.html',
    styleUrl: './dashboard.component.scss'
})
export class DashboardComponent {

}

As components need to be unique, we have to rename the AppComponent to a unique name using the alias statement.

Generate a new Angular library for shared code

Use the following command to create a new Angular library within the EuroDaT context:

ng generate library ngx-<library-name> --prefix=<library-prefix> --project-root libraries/ngx-<library-name> --standalone

The --prefix=<library-prefix> flag is used to set the prefix for the generated library. The --project-root libraries/ngx-<library-name> flag is used to set the project root directory, as Angular libraries are treated differently than Angular projects. The --standalone flag is used to generate a standalone library that can be used in other Angular projects like EuroDaTs web components or the host mock.

As we do not provide an EuroDaT internal NPM registry, we have to link the library to the web component projects and the host project directly. To do so, first, we have to adjust code within tsconfig.json file. Change the element compilerOptions.paths.ngx-<library-name> from

"compilerOptions": {
  "outDir": "./dist/out-tsc",
  ...
  "paths": {
    "ngx-<library-name>": [
      "./dist/ngx-<library-name>"
    ]
  }

to

"compilerOptions": {
  "outDir": "./dist/out-tsc",
  ...
  "paths": {
    "ngx-<library-name>": [
      "./libraries/ngx-<library-name>/src/public-api"
    ]
  }

This will enable both the web component projects and the host project to directly import the library from the source folder of the library. Consequently, the steps of building and publishing to an NPM registry become unnecessary. Further, changes in source files are automatically processed with ng serve host.

Next, we have to import the library in the <app/component name>.component.ts files of the web component projects and the host project, respectively where want to make use of the component:

import {Ngx<LibraryName>Component} from "ngx-<Library-Name>";

Within the component's HTML file, we can reference the library's components like:

<<library-prefix>-<library-name>></<library-prefix>-<library-name>>

As a final housekeeping step remove the component's and service's spec.ts files as well as the reference inside the angular.json file:

"test": {
  "builder": "@angular-devkit/build-angular:karma",
  "options": {
    "tsConfig": "libraries/ngx-<librar-name>/tsconfig.spec.json",
    "polyfills": [
    "zone.js",
    "zone.js/testing"
    ]
  }
}

For better maintenance, have the template and style definitions in seperate files. Replace the inline template and style strings in the library's component typescript file

template: ``,
styles:``

with the following code

templateUrl: './ngx-<library-name>.component.html',
styleUrl:'./ngx-<library-name>.component.scss'

and create the corresponding files in the library's component folder.

Build

Development build

To develop, live preview and test the mocked host, its web components and shared libraries always use ng serve host. Direct access to the web components and libraries are not possible as they are not bootstrapped.

Production build

The build artefacts for the docker image have to be created by calling the command

ng build host --configuration production

The build artifacts will be stored in the dist/ directory.

Library build

No extra build step is required for the shared libraries as they are built together with the web components and the host project.

Testing

Our test framework is cypress for both component/unit and e2e testing. An interactive and user-friendly UI can be opened with the following command:

cd frontend/eurodat-app
npx cypress open

Component Test

In the following, component testing includes also unit tests, since for the frontend setup as we have it, they can be treated equally.

A major advantage of the component tests is that they run completely without cluster and therefore they also run fast. This also means that if there are any dependencies to other services (e.g. API calls, Keycloak, ...) these interaction have to be mocked.

A component test may be only mounting a button and click on it. Or call a helper method and check for correct return (like a classical unit test).

Component tests should be used and written generously and test edge cases of the UI (error handling) and not only the "happy user path" as they are responsible for our code coverage.

npx cypress run --component

e2e Test

E2e tests need the cluster up and running and test the setup as would a user. They are (compared to component tests) slower and therefore precious. When writing e2e tests, take care of artifacts that may stay after running your test (e.g. registering a new user, interaction with external services, emails that are sent).

e2e tests should depict full user journeys / user stories e.g. " As a user, I want to register on EuroDaT, onboard my company and see the progress of the onboarding in my dashboard."

npx cypress run --e2e

Code Coverage Measurement

We measure only the coverage of the component tests. This incentives the usage of component tests over e2e-tests. The code coverage measurement is setup following this guide.

The current thresholds (branch, statements, lines, ... ) of coverage can be configured in package.json. The (total) coverage is also displayed in a MR with successful pipeline in GitLab.

The code coverage is measured with every run of the component tests. The following three scripts are running in the pipeline and can also be executed locally (for a one-click solution in the IDE see this section: NPM tool window in IntelliJ IDE).

npm run test-component-test
npm run test-coverage-summary
npm run test-check-coverage

Visual and interactive results can be found in frontend/eurodat-app/coverage-component/lcov-report/index.html.

Pipeline

In the pipeline, both component and e2e tests run. Any dev pipeline covers only the default browser, whereas the nightly covers multiple different browsers and browser versions.

Component tests and coverage check run in the build and code_quality stages, whereas e2e tests run with the backend e2e tests (after build).

Code coverage results are uploaded to sonarcloud. Sonarcloud computes the coverage on newly added files of the current branch. The global sonar threshold lies at 70%.

Maintenance

We use NPM as Angular package manager to provide the EuroDaT frontend with required external dependencies like Keycloak integration or Bootstrap CSS framework. As Angular does not provide an automated update of external dependencies, we have to update the dependencies manually.

To do so, we have to run the following commands:

Minor updates

To avoid breaking changes in Angular CLI and third party components Angular provides a command to update all dependencies to the latest minor version.

1. npm outdated to get a list of outdated dependencies. Example output would look like:

2. Run ng update @angular/core @angular/cli to update Angular CLI and Angular Core to the latest minor version. You can skip the package names to update all dependencies at once.

3. Run npm outdated again to check if all dependencies are up-to-date. Column "Current" matches "Wanted" for all dependencies. Column "Latest" shows the latest version available, however it could break you code.

4. To make use of the updated dependencies, run npm install to update the package-lock.json file. If you run into any kind of error, try removing both the node_modules folder and package-lock.json file and run npm install again.

Major updates

If you need to migrate to the latest major version of Angular or any other dependency, you have to follow the steps below:

1. Install npm-check-updates using npm i -g npm-check-updates

2. Check for outdated dependencies using ncu. Example output would look like:

3. Run ncu -u to update the package.json file with the latest versions of all dependencies.

4. Run ncu again to check if all versions have been updated in the package.json file.

5. To make use of the updated dependencies, run npm install to update the package-lock.json file. If you run into any kind of error, try removing both the node_modules folder and package-lock.json file and run npm install again.

All above steps are obsolete once we have https://eurodat.atlassian.net/browse/EDTBUC-2049 implemented.

Keep in mind that you still have to refactor your code if any breaking changes have been introduced by the updated dependencies like deprecated function calls.

There might still be some open security issues with the updated dependencies. Run a final check with npm audit to see if there are any security issues left.

Eslint, stylelint and Prettier

As linter, we use eslint for the code quality check of TypeScript code, htmlhint for html and stylelint for scss-files. As formatting tool we use prettier, mostly in their default configuration. You can find the configurations in eslint.config.mjs, .htmlhintrc and .stylelintrc.json.

Eslint, prettier and stylelintcan automatically fix minor issues with those following commands. Consider running these scripts from package.json before pushing code (e.g. as githooks) or activate auto-formatting in your IDE.

npm run eslint
npm run prettier
npm run stylelint
npm run htmlhint

However, in the pipeline we do not want the checked code to be manipulated, so we do just run checks without auto-formatting.