EuroDat Frontend

Architecture

Design patterns

The EuroDaT platform frontend is based on Angular and follows the Angular style guide. The application is structured in modules, components, services, and models. Angular is a hierarchical component-based framework, and we make heavy use of that to better organize the frontend code for development, testing and deployment purposes.

The EuroDaT platform frontend represents a private subset of the whole EuroDaT's organisational website and is only accessible to authenticated EuroDaT participants. EuroDaT's organisational website is managed and provided by the TYPO3 CMS containing a lot of static sites. Though, EuroDaT platform's specific frontends will be provided as widgets to the TYPO3 CMS.

TYPO3 takes care of the login and sign-up process leveraged by EuroDaT's platform Keycloak identity provider. Thus, TYPO3 initializes the user authentication and provides the user with a JWT token. The EuroDaT platform frontend uses this token to authenticate the user against the Keycloak server. This ensures that the widget is only accessible to authenticated EuroDaT participants.

Due to the fact that we are using Angular Elements to provide the EuroDaT frontends as widgets, we have to follow some specific rules to make the widgets work properly. The following chart provides an overview of the EuroDaT frontend architecture:

• We use a single Angular workspace (frontend/eurodat-web) to manage the EuroDaT frontend development
• The Angular workspace consists of two Angular projects and an Angular library folder, i.e.
  • EuroDaT host project (projects/host)
  • EuroDaT web component projects (embedded/<project name>)
  • Angular libraries (libraries/<library name>)

The EuroDaT host project mocks the TYPO3 CMS in Angular by providing headers including menu items and a footer. By bootstrapping the EuroDaT frontend components it enables the development and testing of the widgets with ng serve host and Cypress, respectively. Both in tilt and pipeline, the host project will be deployed to be end-to-end tested with Cypress.

The EuroDaT web component projects contain the actual EuroDaT frontend parts also called "widgets" that are provided as widgets to the TYPO3 CMS. As those projects consists of Custom Angular HTML Elements created with createApplication they are not meant to be developed or tested with ng serve or Cypress directly. Instead, for developing purposes, they are embedded in the host project and later are built with ng build <project name> with the resulting JS files being copied to the TYPO3 CMS hosting filesystem. A new Angular Element which is to be embedded in TYPO3 as a widget requires a new Angular component in the host project. Though being Angular Elements aka. web components, you can fully make use of all Angular features including hierarchical component-based architecture, services, and models (see admin dashboard project as an example).

Both projects can share code like components and services using Angular libraries.

Separation of concerns between TYPO3 and Angular

We want to leverage as much as possible from the already available page design by using TYPO3. This means static elements like header, header pictures and page introductions are defined within TYPO3. The EuroDaT widgets are responsible for the main content of the pages which is shown inside the red box:

TYPO3 Widget integration

Within the TYPO3 backend, the widget definition takes place inside the TYPO3 HTML element whereas image and text definitions are done in the TYPO3 page itself:

TYPO3 Widget definition

The TYPO3 HTML element is used to define the widget and to include the EuroDaT platform frontend JS files which are responsible for rendering the widget content in the custom HTML element. The EuroDaT platform frontend JS files are generated by the Angular CLI.

<div class="fsc-default ce-html py-5 bgLightGrey consortium">

  <admin-dashboard></admin-dashboard>

  <link rel="stylesheet" href="angular/styles.css">
  <f:asset.script identifier="polyfill" src="angular/polyfills.js" defer="true" />
  <f:asset.script identifier="angularmain" src="angular/main.js" type="module" />

</div>

Above example HTML snippet shows the integration of the admin dashboard as widget. Don't forget to copy all output files including the media folder to the filesystem.

• The first line consists of the custom HTML tag. This tag is defined in the EuroDaT web component project's main.ts file.
• The stylesheet which is product of the build process is imported.
• It is crucial to include the polyfills.js file before the main.js file to ensure that the widget works in all browsers.
• As Angular web components use ES6 modules (i.e., it uses export or import), type="module" must be used in the script tag of the main.js file. Loading the file as a module also ensures that the widget is loaded asynchronously and does not block the page rendering.

Operations

Deployment

In Kubernetes, the mock host frontend is deployed in its own namespace frontend. The page is served using the nginx web server.

Which paths are routed to the web page?

All traffic whose path is not explicitly defined in another ingress will be routed to this page. This is achieved by using the path / in the ingress. If any other ingress defines a different path, that path will be prioritized over / since it then has the same prefix and is longer.

Local development environment

Prerequisites

If you don't have Angular CLI installed, please install it with npm install -g @angular/cli. Once done, please install required packages with npm install. Make sure that you use Angular version 17 or above!

tilt

For local development we recommend a more streamlined set of tilt resources which ignores most of the backend services and third party components:

make start-int dev-profile=frontend

This includes only a subset of tilt resources in contrast to the fully fledged setup.

If the frontend development evolves and you need more backend services (e.g. contract service), you can always switch to the full setup with make start-int or adjust the list of excluded services.

Local Development with ng serve

Please create the environment.standalone.ts file in the environments folder like:

export const environment = {
    clusterUrl: 'https://YOUR_FQDN/',
    keycloakUrl: 'https://YOUR_FQDN/auth/',
    keycloakRealm: 'eurodat-frontend',
    keycloakClient: 'host-mock',
    typo3DevHtaccessUsername: 'The credentials can be found in GCP Secret Manager',
    typo3DevHtaccessPassword: 'The credentials can be found in GCP Secret Manager',

}

or use the environment.standalone.template.ts as a template.

Run ng serve host to test the EuroDaT mock with widgets included. Navigate to http://localhost:4200/. The application will automatically reload if you change any of the source files (both host project and web component projects).

Local Development with the non-integrated widget

Please create the widget.environment.standalone.js file in the environments folder like:

window.env = {
    clusterUrl: 'https://YOUR_FQDN/'
}

or use the widget.environment.standalone.template.js file as a template.

NPM tool window in IntelliJ IDE

For frontend development, some pre-defined npm/npx scripts are available in package.json for testing, launching cypress or building. IntelliJ offers a nice one-click-solution to run these scripts: NPM tool window.

Here is a table of the scripts currently available in the package.json file:

Script	Command	Description
npm install	npm install	Installs all dependencies listed in the package.json file. This is usually required if somebody has updated the dependencies or you start from a clean working copy.
start-host	ng serve host	Starts the Angular development server for the host project. This is required to rapid develop the widgets as widgets cannot be tested without further preparations.
build-prod-host-mock	ng build host --configuration production --output-hashing none --base-href ./	Builds the host configuration of the Angular application for production with output hashing disabled and base href set to ./.
build-prod-host-mock-local	ng build host --configuration production_local --output-hashing none --base-href ./ && mkdir -p dist/host/keycloak && touch dist/host/keycloak/env.js	Similar to build-prod-host-mock, but it also creates a env.js file in the dist/host/keycloak directory. This allows local production build testing in a Apache2/nginx local web server.
build-prod-dashboard-admin	ng build dashboard-admin --configuration production --output-hashing none --base-href ./	Builds the dashboard-admin of the Angular application for production with output hashing disabled and base href set to ./. This will later be copied to TYPO3.
build-prod-dashboard-admin-local	ng build dashboard-admin --configuration production_local --output-hashing none --output-path dist/dashboard-admin-local --base-href ./ --deploy-url ./ && cp environments/widget.environment.standalone.js dist/dashboard-admin-local/env.js && cp projects/embedded/dashboard-admin/src/index_widget_local_test.html dist/dashboard-admin-local/index.html	Similar to build-prod-dashboard-admin, but it also replaces the index.html file in the dist/dashboard-admin-local directory with index_widget_local_test.html and provides widget-specific environment variables. This allows local production build testing in a Apache2/nginx local web server.
build-prod-all	npm run build-prod-dashboard-admin && npm run build-prod-host-mock	Builds both the dashboard-admin and host configurations of the Angular application for production.
build-prod-all-local	npm run build-prod-dashboard-admin-local && npm run build-prod-host-mock-local	Builds both the dashboard-admin and host configurations of the Angular application for local testing.
test-component-test	cypress run --component	Runs the component tests using Cypress.
test-coverage-summary	nyc report --reporter=text-summary	Generates a summary of the test coverage using nyc.
test-check-coverage	nyc check-coverage	Checks if the test coverage meets the thresholds specified in the nyc configuration.
test-component-run-all	npm run test-component-test && npm run test-check-coverage && npm run test-coverage-summary	Runs all component tests and coverage checks.
cypress	cypress open	Opens the Cypress test runner.
eslint	eslint . --fix	Runs ESLint on the project and automatically fixes any fixable issues.
eslint-ci	eslint .	Runs ESLint on the project without automatically fixing issues. This is typically used in a continuous integration environment.
prettier	prettier . --write	Runs Prettier on the project and automatically formats any files that don't adhere to the Prettier configuration.
prettier-ci	prettier . --check	Runs Prettier on the project without automatically formatting files. This is typically used in a continuous integration environment.
stylelint	stylelint "**/*.scss" "!**/dist/**" --fix	Runs Stylelint on all SCSS files in the project (excluding those in the dist directory) and automatically fix any fixable issues.
stylelint-ci	stylelint "**/*.scss" "!**/dist/**"	Runs Stylelint on all SCSS files in the project (excluding those in the dist directory) without automatically fixing issues. This is typically used in a continuous integration environment.
htmlhint	npx htmlhint . --ignore="**/coverage-component/**, **/cypress/**, **/dist/**"	Runs HTMLHint on the project, ignoring certain directories.
firefox-e2e	cypress run --browser firefox --headless --reporter junit	Runs end-to-end tests using Cypress with Firefox in headless mode and the JUnit reporter.
firefox-component-test	cypress run --component --browser firefox --headless --reporter junit	Runs component tests using Cypress with Firefox in headless mode and the JUnit reporter.
chrome-e2e	cypress run --browser chrome --headless --reporter junit	Runs end-to-end tests using Cypress with Chrome in headless mode and the JUnit reporter.
chrome-component-test	cypress run --component --browser chrome --headless --reporter junit	Runs component tests using Cypress with Chrome in headless mode and the JUnit reporter.
edge-e2e	cypress run --browser edge --headless --reporter junit	Runs end-to-end tests using Cypress with Edge in headless mode and the JUnit reporter.
edge-component-test	cypress run --component --browser edge --headless --reporter junit	Runs component tests using Cypress with Edge in headless mode and the JUnit reporter.

Note: Make sure to use git bash as default runner for npm scripts on Windows: How to set GitBash as the default NPM script runner on Windows

Code scaffolding

Generate a new Angular project to be used as widget

As all Angular projects share the same workspace and thus, NPM packages, there is no need to install the Angular Elements package for the new added Angular projects (use npm i @angular/elements in case it is not installed, yet). The Angular Elements package is required to generate web components from Angular components.

Instead, we can start right away to generate a new Angular project with the following command:

ng generate application test --standalone=true --ssr=false --inline-style=false --inline-template=false --prefix=<prefix-name> --project-root=projects/embedded/<project-name>> --style=scss --routing=false --skipTests=true

We will use the --standalone=true flag to generate a standalone Angular project that can be used as a widget. The --ssr=false flag is used to disable server-side rendering as it is not needed for the widget. The --inline-style=false and --inline-template=false flags are used to generate separate style and template files. The --prefix=<prefix-name> flag is used to set the prefix for the generated components. The --project-root=projects/embedded/<project-name> flag is used to set the project root directory. The --style=scss flag is used to set the style file extension to SCSS. The --routing=false flag is used to disable routing as it is not needed for the widget. The --skipTests=true flag is used to skip generating test files as we do not use Jest, Karma and Jasmine but Cypress for testing.

However, Angular CLI does not provide any flags to generate the project as web component out-of-the-box. Thus, we have to take care of that after the project has been created.

• Clear all demo content auto-generated from the app.component.html file.
• Remove all auto-generated code from the main.ts file:

import { bootstrapApplication } from '@angular/platform-browser';
import { appConfig } from './app/app.config';
import { AppComponent } from './app/app.component';

bootstrapApplication(AppComponent, appConfig)
.catch((err) => console.error(err));

• Add the following code to the main.ts file:

import { createCustomElement } from '@angular/elements';
import { createApplication } from '@angular/platform-browser';
import { AppComponent } from './app/app.component';

(async () => {

  const app = await createApplication({
    providers: [
      /* your global providers here */
    ],
  });

  const <web component name> = createCustomElement(AppComponent, {
    injector: app.injector,
  });

  customElements.define('<custom HTML tag to be used for the web component>', <web component name>);

})();

Though we have enabled client-side rendering, we still have to take care of some settings in the angular.json file. Search for the newly added <project-name> and set the builder key to browser-esbuild which also requires to rename the browser key to main:

"<project-name>": {
      "projectType": "application",
      "schematics": {
        "@schematics/angular:component": {
          "style": "scss"
        }
      },
      "root": "projects/embedded/<project-name>",
      "sourceRoot": "projects/embedded/<project-name>/src",
      "prefix": "<prefix-name>",
      "architect": {
        "build": {
          "builder": "@angular-devkit/build-angular:browser-esbuild",
          "options": {
            "outputPath": "dist/<project-name>n",
            "index": "projects/embedded/<project-name>/src/index.html",
            "main": "projects/embedded/<project-name>/src/main.ts",
            "polyfills": [
              "zone.js"
            ]

After that ng serve <project name> renders a blank page as due to the custom element the Angular application is not bootstrapped anymore. For that reason, we provide a developer only host Angular project which is a simple Angular application that embeds the custom element and allows us to develop the widget properly using ng serve host. Another way to render the widget is by using a local apache/ngnix server where the component can be displayed independently of any integration, allowing developers to test them in isolation from the main application.

Setup of a local apache/ngnix server for widget testing

While ng serve host allows rapid widget development, isolated widget testing is also possible by setting up a local apache/ngnix server. This is especially useful for testing the widget without interference from the host mock application before copying it to the TYPO3 hosting solution.

1. Install Apache or Ngnix on your local machine. For Apache, you can use XAMPP, WAMP, Laragon or MAMP. Those tools also come with PHP and MySQL database support if local TYPO3 testing might be relevant in future. MAMP and Laragon also provide Ngnix support.
2. To avoid conflicts with already locally exposed ports from your tilt session, it is highly recommended to use different ports for unsecured (e.g. 8080/8888) and protected (e.g. 4443/4444) connections on the local web server. This can be configured in the Apache/Ngnix configuration file or via GUI depending on the chosen tool.
3. As settings differ between local and TYPO3 build use the matching npm run widget. For example, for the admin dashboard project use npm run build-prod-dashboard-admin-local to build the widget for local testing.
4. Depending on the local web server, you either have to copy the dist/<project-name>-local folder to the Apache/Ngnix server's root directory or you might change it to point directly to the dist/<project-name>-local folder.
5. The build widget can be accessed by http(s)://localhost:<port> or http(s)://127.0.0.1:<port>. Some tools also supports virtual hosts which can be used to access the widget by a custom domain like http(s)://eurodat.local:<port>

Generate a new Angular component to host the widget

Remember that the host project is only for development purposes and should not be used in production!

Every new web component required for TYPO3 integration requires a new Angular component in the Angular host project to be developed and tested properly. You may also require to add new components in the widgets itself. Create a new component with the following command:

ng generate component <desired-subfolder>/<component-name> --standalone=true --inline-style=false --inline-template=false --prefix=<prefix-name> --project=host --style=scss

The prefix <desired-subfolder>/ on <component-name> ensures that the component will be placed in the specific subfolder of the Angular project (app/<desired-subfolder>/<component-name>). The --standalone=true flag is used to generate a standalone component to follow the latest Angular design patterns. The --inline-style=false and --inline-template=false flags are used to generate separate style and template files. The --prefix=prefix-name flag is used to set the prefix for the generated component. The --project=host flag is used to set the project where the component should be generated. The --style=scss flag is used to set the style file extension to SCSS. Delete the respective <component-name>.spec.ts after component generation, as it is not used for testing.

To make the new component use the before created web component, we have to add the following code as last line to the <component-name>.component.html

<<prefix-name-web-component>-root></<prefix-name-web-component>>

and import the component within <component-name>.component.ts files, respectively:

import {Component} from '@angular/core';
import {AppComponent as

<component prefix
name > AppComponent
}
from
'../../../../../embedded/<project-name>/src/app/app.component';

@Component({
    selector: 'app-dashboard',
    standalone: true,
    imports: [<component prefix name > AppComponent],
    templateUrl: './dashboard.component.html',
    styleUrl: './dashboard.component.scss'
})
export class DashboardComponent {

}

As components need to be unique, we have to rename the AppComponent to a unique name using the alias statement.

Generate a new Angular library for shared code

Use the following command to create a new Angular library within the EuroDaT context:

ng generate library ngx-<library-name> --prefix=<library-prefix> --project-root libraries/ngx-<library-name> --standalone

The --prefix=<library-prefix> flag is used to set the prefix for the generated library. The --project-root libraries/ngx-<library-name> flag is used to set the project root directory, as Angular libraries are treated differently than Angular projects. The --standalone flag is used to generate a standalone library that can be used in other Angular projects like EuroDaTs web components or the host mock.

As we do not provide an EuroDaT internal NPM registry, we have to link the library to the web component projects and the host project directly. To do so, first, we have to adjust code within tsconfig.json file. Change the element compilerOptions.paths.ngx-<library-name> from

"compilerOptions": {
  "outDir": "./dist/out-tsc",
  ...
  "paths": {
    "ngx-<library-name>": [
      "./dist/ngx-<library-name>"
    ]
  }

to

"compilerOptions": {
  "outDir": "./dist/out-tsc",
  ...
  "paths": {
    "ngx-<library-name>": [
      "./libraries/ngx-<library-name>/src/public-api"
    ]
  }

This will enable both the web component projects and the host project to directly import the library from the source folder of the library. Consequently, the steps of building and publishing to an NPM registry become unnecessary. Further, changes in source files are automatically processed with ng serve host.

Next, we have to import the library in the <app/component name>.component.ts files of the web component projects and the host project, respectively where want to make use of the component:

import {Ngx<LibraryName>Component} from "ngx-<Library-Name>";

Within the component's HTML file, we can reference the library's components like:

<<library-prefix>-<library-name>></<library-prefix>-<library-name>>

As a final housekeeping step remove the component's and service's spec.ts files as well as the reference inside the angular.json file:

"test": {
  "builder": "@angular-devkit/build-angular:karma",
  "options": {
    "tsConfig": "libraries/ngx-<librar-name>/tsconfig.spec.json",
    "polyfills": [
    "zone.js",
    "zone.js/testing"
    ]
  }
}

For better maintenance, have the template and style definitions in seperate files. Replace the inline template and style strings in the library's component typescript file

template: ``,
styles:``

with the following code

templateUrl: './ngx-<library-name>.component.html',
styleUrl:'./ngx-<library-name>.component.scss'

and create the corresponding files in the library's component folder.

Build

Development build

To develop, live preview and test the mocked host, its web components and shared libraries always use ng serve host. Direct access to the web components and libraries are not possible as they are not bootstrapped.

Production build

Mocked host build

For local tilt setup and cluster pipeline deployments, we use the mocked host project. It is necessary to build the mocked host project as we will not have individual developer specific TYPO3 instances for integration available.

The build artefacts for the docker image have to be created by calling the command

ng build host --configuration production

The build artifacts will be stored in the dist/ directory.

TYPO3 build

TYPO3 uses Javascript files of the embedded isolated Angular projects only. Run the following command to build the desired Angular project as a web component:

ng build <project-name> --output-hashing=none --configuration production`.

We have to set the outputHashing property to none to avoid hash values in the generated JS files. This is crucial as the JS files are referenced in the TYPO3 CMS and the hash values would break the widget functionality in any future update.

During EuroDaT release, all relevant web component projects are automatically built and the JS files from the build artefacts are copied to the webspace of the TYPO3 test instance.

Note that EuroDaT's CI/CD pipeline does not create the necessary requirements in TYPO3 to include the widgets. This has to be done manually by the TYPO3 administrator (e.g. create page, enable page, define build blocks to the page, define header, create introduction text).

Library build

No extra build step is required for the shared libraries as they are built together with the web components and the host project.

Testing

Our test framework is cypress for both component/unit and e2e testing. An interactive and user-friendly UI can be opened with the following command:

cd frontend/eurodat-web
npx cypress open

Component Test

In the following, component testing includes also unit tests, since for the frontend setup as we have it, they can be treated equally.

A major advantage of the component tests is that they run completely without cluster and therefore they also run fast. This also means that if there are any dependencies to other services (e.g. API calls, Keycloak, ...) these interaction have to be mocked.

A component test may be only mounting a button and click on it. Or call a helper method and check for correct return (like a classical unit test).

Component tests should be used and written generously and test edge cases of the UI (error handling) and not only the "happy user path" as they are responsible for our code coverage.

npx cypress run --component

e2e Test

E2e tests need the cluster up and running and test the setup as would a user. They are (compared to component tests) slower and therefore precious. When writing e2e tests, take care of artifacts that may stay after running your test (e.g. registering a new user, interaction with external services, emails that are sent).

e2e tests should depict full user journeys / user stories e.g. " As a user, I want to register on EuroDaT, onboard my company and see the progress of the onboarding in my dashboard."

npx cypress run --e2e

Code Coverage Measurement

We measure only the coverage of the component tests. This incentives the usage of component tests over e2e-tests. The code coverage measurement is setup following this guide.

The current thresholds (branch, statements, lines, ... ) of coverage can be configured in package.json. The (total) coverage is also displayed in a MR with successful pipeline in GitLab.

The code coverage is measured with every run of the component tests. The following three scripts are running in the pipeline and can also be executed locally (for a one-click solution in the IDE see this section: NPM tool window in IntelliJ IDE).

npm run test-component-test
npm run test-coverage-summary
npm run test-check-coverage

Visual and interactive results can be found in frontend/eurodat-web/coverage-component/lcov-report/index.html.

Pipeline

In the pipeline, both component and e2e tests run. Any dev pipeline covers only the default browser, whereas the nightly covers multiple different browsers and browser versions.

Component tests and coverage check run in the build and code_quality stages, whereas e2e tests run with the backend e2e tests (after build).

Code coverage results are uploaded to sonarcloud. Sonarcloud computes the coverage on newly added files of the current branch. The global sonar threshold lies at 70%.

Maintenance

We use NPM as Angular package manager to provide the EuroDaT frontend with required external dependencies like Keycloak integration or Bootstrap CSS framework. As Angular does not provide an automated update of external dependencies, we have to update the dependencies manually.

To do so, we have to run the following commands:

Minor updates

To avoid breaking changes in Angular CLI and third party components Angular provides a command to update all dependencies to the latest minor version.

1. npm outdated to get a list of outdated dependencies. Example output would look like:

2. Run ng update @angular/core @angular/cli to update Angular CLI and Angular Core to the latest minor version. You can skip the package names to update all dependencies at once.

3. Run npm outdated again to check if all dependencies are up-to-date. Column "Current" matches "Wanted" for all dependencies. Column "Latest" shows the latest version available, however it could break you code.

4. To make use of the updated dependencies, run npm install to update the package-lock.json file. If you run into any kind of error, try removing both the node_modules folder and package-lock.json file and run npm install again.

Major updates

If you need to migrate to the latest major version of Angular or any other dependency, you have to follow the steps below:

1. Install npm-check-updates using npm i -g npm-check-updates

2. Check for outdated dependencies using ncu. Example output would look like:

3. Run ncu -u to update the package.json file with the latest versions of all dependencies.

4. Run ncu again to check if all versions have been updated in the package.json file.

5. To make use of the updated dependencies, run npm install to update the package-lock.json file. If you run into any kind of error, try removing both the node_modules folder and package-lock.json file and run npm install again.

All above steps are obsolete once we have https://eurodat.atlassian.net/browse/EDTBUC-2049 implemented.

Keep in mind that you still have to refactor your code if any breaking changes have been introduced by the updated dependencies like deprecated function calls.

There might still be some open security issues with the updated dependencies. Run a final check with npm audit to see if there are any security issues left.

Eslint, stylelint and Prettier

As linter, we use eslint for the code quality check of TypeScript code, htmlhint for html and stylelint for scss-files. As formatting tool we use prettier, mostly in their default configuration. You can find the configurations in eslint.config.mjs, .htmlhintrc and .stylelintrc.json.

Eslint, prettier and stylelintcan automatically fix minor issues with those following commands. Consider running these scripts from package.json before pushing code (e.g. as githooks) or activate auto-formatting in your IDE.

npm run eslint
npm run prettier
npm run stylelint
npm run htmlhint

However, in the pipeline we do not want the checked code to be manipulated, so we do just run checks without auto-formatting.