diff --git a/INSTALL.md b/INSTALL.md index 82312f22d..1c587d65c 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -23,31 +23,7 @@ docker compose -f docker/compose/dependencies/docker-compose.yml up -d ``` This will run a Postgres and Keycloak with an initial realm already configured to be used by the BPDM services. -You will need to pass the necessary client secrets when starting the BPDM services. -Those client secrets are generated by Keycloak after the first time setup. -Therefore, we will log in to the Keycloak server, look-up the secrets and write them into environment variables that are read by the BPDM services. - -1. Log in to Keycloaks admin console with the admin credentials (default in the Docker Compose file): http://localhost:8180/admin/master/console -2. Navigate to realm "Cx-Central" -3. Navigate to the following clients and write the client secret to the environment variable: - -| Client | Environment Variable | -|-----------------------------------|-------------------------------| -| DUMMY-ORCHESTRATOR-TASK_PROCESSOR | BPDM_DUMMY_ORCH_CLIENT_SECRET | -| GATE-ORCHESTRATOR-TASK_CREATOR | BPDM_GATE_ORCH_CLIENT_SECRET | -| GATE-POOL-SHARING_MEMBER | BPDM_GATE_POOL_CLIENT_SECRET | -| POOL-ORCHESTRATOR-TASK-PROCESSOR | BPDM_POOL_ORCH_CLIENT_SECRET | - -This information on what client expects which environment variable is also available in the application properties files of the respective BPDM applications (like for the [gate](bpdm-gate/src/main/resources/application.yml)) -Now, make sure that client secrets are available in the expected environment variables: -```console -export BPDM_DUMMY_ORCH_CLIENT_SECRET=... -export BPDM_GATE_ORCH_CLIENT_SECRET=... -export BPDM_GATE_POOL_CLIENT_SECRET=... -export BPDM_POOL_ORCH_CLIENT_SECRET=... -``` - -After this step, we can finally build, install the BPDM applications. +After this step, we can build and install the BPDM applications. ```console mvn clean install @@ -69,20 +45,50 @@ mvn spring-boot:run ``` After this you have full-fledged running BPDM system. +The Keycloak realm adheres to the configuration of the [Central-IDP](https://github.com/eclipse-tractusx/portal-iam/blob/main/docs/admin/technical-documentation/03.%20Clients.md#initial-clients-and-service-accounts). +For accessing the BPDM APIs you can choose one of the pre-existing clients over client authentication flow, notably sa-cl7-cx5 or sa-cl7-cx7 for access to Gate and Pool. As a next step on how to pass business partner data and create golden records you can have a look at the [api documentation](docs/api/README.md) of this repository. +### Gate Configuration + +#### Owned Gates + +On default, the BPDM Gate supports a simple multi-tenancy feature. +Each company, identified by the BPNL of the accessing user, has a separate tenant of business partner data in one Gate. + +Alternatively, a BPDM Gate can be configured to be single tenant only. +This allows access for users only from a single company identified by its BPNL. + +```yaml +bpdm: + bpn: + owner-bpn-l: BPNLXXXXXXXXXX01 +``` +If it is set with a BPNL only users belonging to that company can access the Gate. +If it is not set or set to null a user from any company can use the Gate, albeit the user can only see its own company shared business partner data. + +#### Manual Sharing + +On default, the BPDM Gate automatically starts the golden record process for uploaded business partner data. +Alternatively, the Gate can be configured to not start the process automatically. +In this case business partner data can be uploaded and changed, before the user has to manually set the data to be ready to be shared to the golden record process. + +```yaml +bpdm: + tasks: + creation: + starts-as-ready: false +``` + ### Insecure Installation -You may want to perform a quick installation in which security is not necessary. -In this case, you don't need to configure any environment variables. -This means you can directly run the BPDM applications after you set up the necessary dependencies. +You may want to perform a quick local installation in which security is not necessary. Make sure to disable authentication requirements by using the provided `no-auth` profile when running the applications: ```console mvn spring-boot:run -Dspring.profiles.active=no-auth ``` - ## Helm Charts Installation of BPDM applications with the Helm Charts has the most software requirements but is the qickest way to set up a running system. @@ -104,74 +110,149 @@ Then install a new release of BPDM on your default namespace via helm: helm install bpdm ./charts/bpdm ``` This will install the BPDM applications with its own Postgres and Keycloak in default values. -Mind that this will also install the applications with default passwords. -### Override Default Secrets +### Override Defaults -It is good practice to overwrite the default secrets and passwords that are used in the BPDM Charts. -For this, you can first define a bunch of environment variables holding new secret values and use them later during deployment: +The easiest way to overrride default configuration of the BPDM Helm Chart is to provide a custom values file while deploying. -```console -helm install bpdm \ - --set-value postgres.auth.password=$BPDM_POSTGRES \ - --set-value keycloak.auth.adminPassword=$BPDM_KEYCLOAK_ADMIN \ - --set-value keycloak.bpdm.realm.clientSecrets.cleaningDummyOrchestrator=$BPDM_DUMMY_ORCH_CLIENT_SECRET \ - --set-value keycloak.bpdm.realm.clientSecrets.poolOrchestrator=$BPDM_POOL_ORCH_CLIENT_SECRET \ - --set-value keycloak.bpdm.realm.clientSecrets.gateOrchestrator=$BPDM_GATE_ORCH_CLIENT_SECRET \ - --set-value keycloak.bpdm.realm.clientSecrets.gatePool=$BPDM_GATE_POOL_CLIENT_SECRET \ - --set-value bpdm-gate.applicationSecrets.bpdm.client.orchestrator.registration=$BPDM_GATE_ORCH_CLIENT_SECRET \ - --set-value bpdm-gate.applicationSecrets.bpdm.client.pool.registration=$BPDM_GATE_POOL_CLIENT_SECRET \ - --set-value bpdm-pool.applicationSecrets.bpdm.client.orchestrator.registration=$BPDM_POOL_ORCH_CLIENT_SECRET \ - --set-value bpdm-cleaning-service-dummy.applicationSecrets.bpdm.client.orchestrator.registration=$BPDM_DUMMY_ORCH_CLIENT_SECRET\ - .charts/bpdm/bpdm +```bash +helm install bpdm --values path/to/values/file.yml ./charts/bpdm ``` -### Insecure Installation +The following sections provide example use cases for default overrides. + + +#### Overriding Postgres Passwords + +If you want to change the default password of the given postgres database you can override the postgres configuration and the connection configuration of each BPDM app that uses the database: + +```yaml +postgres: + password: $PASSWORD +bpdm-gate: + applicationSecrets: + spring: + datasource: + password: $PASSWORD +bpdm-pool: + applicationSecrets: + spring: + datasource: + password: $PASSWORD +bpdm-orchestrator: + applicationSecrets: + spring: + datasource: + password: $PASSWORD +``` + +#### Overriding Central-IDP Secrets + +You can use [seeding](https://github.com/eclipse-tractusx/portal-iam/tree/v3.0.1/charts/centralidp) for adding custom passwords to the Central-IDP dependency. +The new client clients secrets then need to be given to the connections of each BPDM app: + +```yaml +bpdm-gate: + applicationSecrets: + bpdm: + client: + orchestrator: + registration: + client-secret: $GATE_ORCH_CLIENT_SECRET + pool: + registration: + client-secret: $GATE_POOL_CLIENT_SECRET +bpdm-pool: + applicationSecrets: + bpdm: + client: + orchestrator: + registration: + client-secret: $POOL_ORCH_CLIENT_SECRET +bpdm-cleaning-service-dummy: + applicationSecrets: + bpdm: + client: + orchestrator: + registration: + client-secret: $CLEANING_DUMMY_ORCH_CLIENT_SECRET +``` + +#### Insecure Installation For non-production purposes you may want to install BPDM applications that are not authenticated. All BPDM applications offer a Spring profile to quickly remove all authentication configuration for their APIs and client connections. -In this case you can also disable the Keycloak dependency from being deployed. +In this case you can also disable the Central-IDP dependency from being deployed. -```console -helm install bpdm \ - --set-value keycloak.enabled=false - --set-value bpdm-gate.profiles=["no-auth"] \ - --set-value bpdm-orchestrator.profiles=["no-auth"] \ - --set-value bpdm-pool.profiles=["no-auth"] \ - --set-value bpdm-cleaning-service-dummy.profiles=["no-auth"] - .charts/bpdm/bpdm +```yaml +centralIdp: + enabled: false +bpdm-gate: + springProfiles: + - no-auth +bpdm-pool: + springProfiles: + - no-auth +bpdm-cleaning-service-dummy: + springProfiles: + - no-auth +bpdm-orchestrator: + springProfiles: + - no-auth ``` You can also more fine-granulary remove authentication on APIs and BPDM client connections. You can refer to the no-auth profile configurations (for example that of the [Gate](bpdm-gate/src/main/resources/application-no-auth.yml)) as a documentation. -### Use External Dependencies +#### Use External Dependencies The BPDM Charts deploy their own PostgreSQL and Keycloak dependencies. However, for production it is recommended to host dedicated Postgres and Keycloak instances with which the BPDM applications should connect to. -#### Additional Requirements - - * Postgres (15.4.0 supported) - * Keycloak (22.0.3 supported) - -#### Installation +>Additional Requirements +> +> * Postgres (15.4.0 supported) +> * Keycloak (22.0.3 supported) In this case, you can disable the dependencies and configure the connection to external systems in the application configuration. -```console -helm install bpdm \ - --set-value keycloak.enabled=false - --set-value postgres.enabled=false - --set-value bpdm-gate.applicationConfig.bpdm.datasource.host=external-db \ - --set-value bpdm-gate.applicationConfig.bpdm.security.auth-server-url=http://external-keycloak \ - --set-value bpdm-pool.applicationConfig.bpdm.datasource.host=external-db \ - --set-value bpdm-pool.applicationConfig.bpdm.security.auth-server-url=http://external-keycloak \ - --set-value bpdm-orchestrator.applicationConfig.bpdm.security.auth-server-url=http://external-keycloak \ - --set-value bpdm-cleaning-service-dummy.applicationConfig.bpdm.client.orchestrator.provider.issuer-uri= http://external-keycloak/realms/CX-Central \ - .charts/bpdm/bpdm +```yaml +centralIdp: + enabled: false +postgres: + enabled: false +bpdm-gate: + applicationConfig: + bpdm: + datasource: + host: "http://remote-postgres" + security: + auth-server-url: "http://remote-centralIdp/auth" +bpdm-pool: + applicationConfig: + bpdm: + datasource: + host: "http://remote-postgres" + security: + auth-server-url: "http://remote-centralIdp/auth" +bpdm-orchestrator: + applicationConfig: + bpdm: + datasource: + host: "http://remote-postgres" + security: + auth-server-url: "http://remote-centralIdp/auth" +bpdm-cleaning-service-dummy: + applicationConfig: + bpdm: + client: + orchestrator: + provider: + issuer-uri: "http://remote-centralIdp/auth" ``` +You can combine this configuration with the examples for overriding password and secrets to adapt BPDM's connection configuration to you wishes. + ### Fine-granular Configuration You can configure all BPDM applications over Helm values more fine-granulary via the `applicationConfig` and `applicationSecrets`. @@ -183,31 +264,6 @@ As a reference of what can be changed have a look at the respective application - [BPDM Orchestrator](bpdm-orchestrator/src/main/resources/application.yml) - [BPDM Cleaning Service Dummy](bpdm-cleaning-service-dummy/src/main/resources/application.yml) -## Post-Run Configuration - -For the most part BPDM applications don't need to be further configured after they started. -The BPDM Pool is an exception to that rule. -The Pool offers endpoints for operators to regulate available metadata. -Metadata are supporting business partner information like legal forms, identifier types or administrative level 1 areas. -For example, by adding a range of legal forms operators are able to determine the available legal forms business partners in the Pool can have. - -Typically, business partner data references metadata via technical keys. -If a technical key does not exist in the respective metadata the Pool rejects the record. - -Administrative level 1 areas follows the ISO 3166-2 norm and is filled by default. -Such metadata does not need to be added by the operator. - -The [use case Postman collection](docs/postman/BPDM%20Tests.postman_collection.json) shows which metadata can be added and how that is done. -Refer to use cases for Operators(O): - -- CL: Shows how to add available legal forms -- CIL: Shows how to add available identifier types used for legal entities -- CIA: Shows how to add available identifier types used for addresses - -Please note that in the current implementation there are no endpoints to delete metadata. -Deletions would need to be done directly in the database. - - ## EDC Installation This section shows how to make your BPDM Gate and Pool APIs available over an EDC. @@ -242,7 +298,7 @@ This offer allows a company to access the Pool for reading golden record data of 1. Create a policy of type `HasBusinessPartnerNumber` for the company's BPNL (if it not yet exists) 2. Create a policy of type `AcceptPurpose` with usage purpose for the pool (if not yet exists) -3. Create a technical user with role `Pool Cx Member` and the company's BPN identity +3. Create a technical user with role `BPDM Pool Consumer` and the company's BPN identity 4. Create a `ReadAccessPoolForCatenaXMember` asset with the created technical user for client credentials 5. Create a contract definition `ReadAccessPoolForCatenaXMember` referencing the created asset @@ -253,7 +309,7 @@ This offer allows a company to access a Gate API for uploading business partner 1. Create a policy of type `HasBusinessPartnerNumber` for the company's BPNL (if it not already exists) 2. Create a policy of type `AcceptPurpose` with upload usage purpose for the gate (if it not already exists) -3. Create a technical user with role `Gate Input Manager` and the company's BPN identity +3. Create a technical user with role `BPDM Sharing Input Manager` and the company's BPN identity 4. Create a `FullAccessGateInputForSharingMember` asset with the created technical user for client credentials 5. Create a contract definition `FullAccessGateInputForSharingMember` referencing the created asset @@ -264,7 +320,7 @@ This offer explicitly does **not** grant access to read the golden record output 1. Create a policy of type `HasBusinessPartnerNumber` for the company's BPNL (if it not already exists) 2. Create a policy of type `AcceptPurpose` with download usage purpose for the gate (if it not already exists) -3. Create a technical user with role `Gate Input Consumer` and the company's BPN identity +3. Create a technical user with role `BPDM Sharing Input Consumer` and the company's BPN identity 4. Create a `ReadAccessGateInputForSharingMember` asset with the created technical user for client credentials 5. Create a contract definition `ReadAccessGateInputForSharingMember` referencing the created asset @@ -275,7 +331,7 @@ This offer does **not** grant access to the uploaded business partner input data 1. Create a policy of type `HasBusinessPartnerNumber` for the company's BPNL (if it not already exists) 2. Create a policy of type `AcceptPurpose` with download usage purpose for the gate (if it not already exists) -3. Create a technical user with role `Gate Output Consumer` and the company's BPN identity +3. Create a technical user with role `BPDM Sharing Output Consumer` and the company's BPN identity 4. Create a `ReadAccessGateOutputForSharingMember` asset with the created technical user for client credentials 5. Create a contract definition `ReadAccessGateOutputForSharingMember` referencing the created asset @@ -291,22 +347,18 @@ The following instructions assume you are using the BPDM helm chart to deploy th #### Deploy the initial golden record process components -1. Disable the own Keycloak dependency -2. Set the authentication server to [Central-IDP](https://github.com/eclipse-tractusx/portal-iam) -3. Provide the client credentials of pre-existing Central-IDP technical users to the BPDM components to establish communication between the components -4. Overwrite the standard client-ids to match the one in Central-IDP -5. Overwrite the BPDM Orchestrator default permission names to be compatible with the Central-IDP (see [technical debts](docs/architecture/11_Risks_And_Technical_Debts.md)) -6. Expose the Pool over ingress on context path `pool` to make it available to the Portal -7. Expose the Portal Gate over ingress on context path `companies/test-company` to make it available to the Portal -8. Configure the Gate to **not automatically** share newly uploaded business partner data to the golden record process -9. Configure the Gate to have **no** owner as it needs to be used by multiple companies to manage their own sites and addresses +1. Disable the own Central-IDP dependency +2. Set the authentication server to the Central-IDP instance used by the Portal +3. Override the default client secrets with the ones used in the Portal's Central-IDP +4. Expose the Pool over ingress on context path `pool` to make it available to the Portal +5. Expose the Portal Gate over ingress on context path `companies/test-company` to make it available to the Portal +6. Configure the Gate to have **no** owner as it needs to be used by multiple companies to manage their own sites and addresses ```yaml # Helm Values Overwrite -keycloak: +centralidp: enabled: false - bpdm-pool: ingress: enabled: true @@ -321,23 +373,16 @@ bpdm-pool: pathType: "ImplementationSpecific" applicationConfig: server: - # App should take the x-forward-header-prefix into account for Swagger-UI and redirects to work correctly - forward-headers-strategy: "FRAMEWORK" + forward-headers-strategy: "FRAMEWORK" bpdm: security: auth-server-url: "http://central-idp-host-name/auth" - client-id: "Cl7-CX-BPDM" - client: - orchestrator: - registration: - client-id: "sa-cl7-cx-5" applicationSecrets: bpdm: client: orchestrator: registration: - client-secret: "*****" - + client-secret: $POOL_ORCH_CLIENT_SECRET bpdm-gate: ingress: enabled: true @@ -352,53 +397,30 @@ bpdm-gate: pathType: "ImplementationSpecific" applicationConfig: server: - # App should take the x-forward-header-prefix into account for Swagger-UI and redirects to work correctly - forward-headers-strategy: "FRAMEWORK" + forward-headers-strategy: "FRAMEWORK" bpdm: bpn: - owner-bpn-l: + owner-bpn-l: null tasks: creation: fromSharingMember: starts-as-ready: false security: auth-server-url: "http://central-idp-host-name/auth" - client-id: "Cl16-CX-BPDMGate" - client: - orchestrator: - registration: - client-id: "sa-cl7-cx-5" - pool: - registration: - client-id: "sa-cl7-cx-5" applicationSecrets: bpdm: client: orchestrator: registration: - client-secret: "*****" + client-secret: $GATE_ORCH_CLIENT_SECRET pool: registration: - client-secret: "*****" - + client-secret: $GATE_POOL_CLIENT_SECRET bpdm-orchestrator: applicationConfig: bpdm: security: auth-server-url: "http://central-idp-host-name/auth" - client-id: "Cl7-CX-BPDM" - permissions: - createTask: "write_partner" - readTask: "write_partner" - reservation: - clean: "write_partner" - cleanAndSync: "write_partner" - poolSync: "write_partner" - result: - clean: "write_partner" - cleanAndSync: "write_partner" - poolSync: "write_partner" - bpdm-cleaning-service-dummy: applicationConfig: bpdm: @@ -406,62 +428,66 @@ bpdm-cleaning-service-dummy: orchestrator: provider: issuer-uri: "https://central-idp-host-name/auth/realms/CX-Central" - registration: - client-id: "sa-cl7-cx-5" applicationSecrets: bpdm: client: orchestrator: registration: - client-secret: "*****" + client-secret: $CLEANING_DUMMY_ORCH_CLIENT_SECRET ``` -#### Create BPDM marketplace services +#### Create BPDM marketplace apps For giving companies access to the golden record process and/or Pool the operator needs to create appropriate EDC offers. The EDC offers should provide access to the Pool API and Gate API. Companies should not access the APIs directly over technical users but rather use the EDC offers. The most important pre-requisite for creating such EDC offers are technical users which have the correct roles and BPN identities. -The Portal supports creating such technical users by creating marketplace services that companies can subscribe to. +The Portal supports creating such technical users by creating marketplace apps that companies can subscribe to. After the company subscribed to a service a technical user will be created automatically by the Portal containing the BPN identity of the subscribing company and the roles defined in the marketplace service. For BPDM the operator needs to create 3 services: -1. Business Partner Members Pool Service: Access to the Catena-X member golden records -2. Sharing Member Upload Service: Upload company business partner data and share it to the golden record process -3. Sharing Member Download Service: Download golden records for previously shared business partner data +1. Business Partner Members Pool App: Access to the Catena-X member golden records +2. Sharing Member Upload App: Upload company business partner data and share it to the golden record process +3. Sharing Member Download App: Download golden records for previously shared business partner data -When creating the service, the operator needs to specify with which role the technical user should be initialized after a company subscribes to it: +When creating the app, the operator needs to specify with which role the technical user should be initialized after a company subscribes to it: -1. Business Partner Members Pool Service: Pool Cx Member -2. Sharing Member Upload Service: Gate Input Manager -3. Sharing Member Download Service: Gate Output Consumer +1. Business Partner Members Pool Service: BPDM Pool Consumer +2. Sharing Member Upload Service: BPDM Sharing Input Manager +3. Sharing Member Download Service: BPDM Sharing Output Consumer #### Manage BPDM service subscriptions -When a company subscribes to a BPDM service the operator needs to perform the following tasks: +When a company subscribes to a BPDM service the operator needs to [create an EDC](#edc-installation) offer based on the service using the created technical user for that subscription. +If the subscription is a sharing member service you additionally need to create and configure a new Gate for that sharing member: -1. Create a new Gate deployment (if it is a sharing member service) -2. [Create an EDC](#edc-installation) offer based on the service using the created technical user +1. Create a new technical user `BPDM Pool Sharing Consumer` in the Portal +2. Create a new technical user `BPDM Orchestrator Task Creator` in the Portal +3. Create a new Gate deployment (if it is a sharing member service) -If a new Gate has to be deployed it needs to integrate into the existing BPDM components: +The new Gate should be configured in the following way: 1. Disable all components except the Gate to only deploy a new Gate instance -2. Make sure that the Gate is accessible to your EDC (either by ingress or internal host name) -3. Set the subscribing company's BPNL as the owner of that Gate -4. Optionally configure whether uploaded business partner data should be automatically shared to the golden record process according to the subscribing companies wishes -5. Set the Gate's Orchestrator and Pool client base-urls so that the Gate can find the existing components +2. Set the subscribing company's BPNL as the owner of that Gate +3. Set the authentication server to the Central-IDP instance used by the Portal +4. Set the Gate's Orchestrator and Pool client base-urls so that the Gate can find the existing components +5. Use the client credentials of the created technical users to configure the connection to the Orchestrator and Pool +6. Make sure the Gate is accessible to your EDC (either by local host name or ingress) +7. Optionally, configure whether uploaded business partner data should be automatically shared to the golden record process according to the subscribing companies wishes ```yaml # Helm Values Overwrite -keycloak: +centralIdp: enabled: false - bpdm-pool: enabled: false - +bpdm-orchestrator: + enabled: false +bpdm-cleaning-service-dummy: + enabled: false bpdm-gate: applicationConfig: bpdm: @@ -473,31 +499,26 @@ bpdm-gate: starts-as-ready: false/true security: auth-server-url: "http://central-idp-host-name/auth" - client-id: "Cl16-CX-BPDMGate" client: orchestrator: base-url: "orchestrator-api-url" registration: - client-id: "sa-cl7-cx-5" + client-id: $BPDM_ORCHESTRATOR_TASK_CREATOR_CLIENT_ID pool: base-url: "pool-api-url" registration: - client-id: "sa-cl7-cx-5" + client-id: $BPDM_POOL_SHARING_CONSUMER_CLIENT_ID applicationSecrets: bpdm: client: orchestrator: registration: - client-secret: "*****" + client-secret: $BPDM_ORCHESTRATOR_TASK_CREATOR_CLIENT_SECRET pool: registration: - client-secret: "*****" + client-secret: $BPDM_POOL_SHARING_CONSUMER_CLIENT_SECRET -bpdm-orchestrator: - enabled: false -bpdm-cleaning-service-dummy: - enabled: false ``` ## NOTICE