Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add federated catalog samples for an embedded and a standalone federated catalog #342

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions federated-catalog/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Federated Catalog Samples

The samples in this section focus on the topic of Federated Catalogs.

The Federated Catalog (FC) functions as an aggregated repository of
catalogs obtained from multiple
participants in the dataspace. To accomplish this, the FC utilizes crawlers
that periodically crawl the catalogs from each participant and store this list
of catalogs in a local cache.
By maintaining this locally cached version of catalogs, it eliminates the need to query
each participant individually, resulting in faster and more reliable queries.


The following samples shows how to
* implement, build and run different versions of FC e.g.
* standalone,
* embedded.
* implement TargetNodeDirectory and resolve Target Nodes,
* from a static file containing all participants' DSP endpoints,



## Samples

### [FC sample 00](./fc-00-basic/README.md): Federated Catalog Prerequisites
The purpose of this example is to make preparations for implementing Federated Catalog (FC).
We'll set up a basic federated catalog that includes necessary dependencies for triggering the FC,
along with some other modules to demonstrate FC functionalities.

---

### Implement Different Versions of FC
### [FC sample 01](./fc-01-embedded/README.md): Implement an embedded federated catalog
This sample demonstrates how we can implement a federated catalog which is embedded in a connector.
The connector exposes a catalog endpoint that serves the consolidated list of catalogs.
### [FC sample 02](./fc-02-standalone/README.md): Implement a standalone federated catalog

In this sample we focus on the implementation of
a standalone federated catalog. Unlike the previous sample,
a standalone federated catalog will not have the added functionalities of a connector. However, it also
exposes a catalog API that serves the list of catalogs.

---

### Different Implementations of Node Resolver


In the following samples you will learn how to implement a TargetNodeDirectory depending on different use cases.

### [FC sample 03](): Resolve Target Catalog Nodes from static participant file
This sample demonstrates a Catalog Node resolver, that implements TargetNodeDirectory. It resolves the Target Catalog
Nodes from a static participant file containing the DSP endpoints of the participants.


98 changes: 98 additions & 0 deletions federated-catalog/fc-00-basic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Federated Catalog Prerequisites

The purpose of this example is to make preparations for implementing Federated Catalog (FC)
and set up additional requirements for testing the FC functionalities.
For this purpose, we will be covering the following in this scope.
* `federated-catalog-base`: A basic federated catalog that includes necessary dependencies for triggering the FC.
* `participant-connector`: A connector with a contract offer. We will require this for testing the functionalities of the FCs in our later samples.
* `fixed-node-resolver`: A mock node directory resolver.
It provides a fixed Target Catalog Node, which represents the `participant-connector`.



### federated-catalog-base
The [federated-catalog-base](../fc-00-basic/federated-catalog-base) will be used as a foundational module in our upcoming samples to trigger the FC.
It provides a [build.gradle.kts](./federated-catalog-base/build.gradle.kts) file that includes only the dependencies
essential for FC, without any additional functionality.
```shell
...
dependencies {
implementation(libs.edc.fc.spi.crawler)
runtimeOnly(libs.fc.core)
runtimeOnly(libs.fc.ext.api)
}
...
```
Any further dependencies will be added in the later samples based on their use cases.


### fixed-node-resolver
The Federated Catalog requires a list of Target Catalog Nodes, which are essentially the DSP endpoints of the dataspace participants.
The catalog crawler then crawls these listed endpoints to collect their offered catalogs.
This list of Target Nodes is resolved by a Catalog Node Resolver which implements the [TargetNodeDirectory](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNodeDirectory.java).
Check out [eclipse-edc/FederatedCatalog](https://github.com/eclipse-edc/FederatedCatalog/tree/main) for further information on this topic.


In this module, we've included a fixed Node Resolver, [fixed-node-resolver](./fixed-node-resolver)
that simply returns a hard-coded Target Node of the `participant-connector`.
However, we will not cover the implementation of the resolver in this sample; that will be explained in detail later in
[fc-03-static-node-directory](../fc-03-static-node-directory/README.md).


The purpose of including this [`fixed-node-resolver`](./fixed-node-resolver)
as a prerequisite, is the fact that we need to have some form of Target Node Resolver to demonstrate the functionality
of the federated catalogs that we are going to build in sample
[fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone).

### participant connector

When the federated catalog boots up, the crawler begins periodically invoking the Target Nodes returned by the
Node Resolver and collecting the catalogs offered by these nodes. To test whether our federated catalogs
(which we will build in later samples: [fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../../fc/fc-02-standalone)) can successfully request and retrieve these catalogs, we need at least one connector with a contract offer.

Therefore, in this section, we will start a connector and then create a contract
for this connector. In the future samples, we will refer to it as `participant-connector`.
This `participant-connector` will function as a provider.
We will use the resources from the [transfer](../../transfer) sample to set up this connector. In the rest of this section we will,
* run the `participant-connector`
* create an asset for this `participant-connector`
* create a policy
* create a contract offer

Although these topics were covered in the [transfer](../../transfer) section, we’ll document all the necessary commands here for easier execution.


#### Build connector jar
Use the following command to build a connector jar.
```shell
./gradlew transfer:transfer-00-prerequisites:connector:build
```
#### Run the connector
Execute the following to run the connector jar.
```shell
java -Dedc.keystore=transfer/transfer-00-prerequisites/resources/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.fs.config=transfer/transfer-00-prerequisites/resources/configuration/provider-configuration.properties -jar transfer/transfer-00-prerequisites/connector/build/libs/connector.jar
```

---

Once the connector is running, carry out the commands below in sequence.
#### Create an asset
```shell
curl -d @transfer/transfer-01-negotiation/resources/create-asset.json \
-H 'content-type: application/json' http://localhost:19193/management/v3/assets \
-s | jq
```

#### Create a policy
```bash
curl -d @transfer/transfer-01-negotiation/resources/create-policy.json \
-H 'content-type: application/json' http://localhost:19193/management/v3/policydefinitions \
-s | jq
```

#### Create a contract definition
```bash
curl -d @transfer/transfer-01-negotiation/resources/create-contract-definition.json \
-H 'content-type: application/json' http://localhost:19193/management/v3/contractdefinitions \
-s | jq
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright (c) 2024 Fraunhofer-Gesellschaft
*
* This program and the accompanying materials are made available under the
* terms of the Apache License, Version 2.0 which is available at
* https://www.apache.org/licenses/LICENSE-2.0
*
* SPDX-License-Identifier: Apache-2.0
*
* Contributors:
* Fraunhofer-Gesellschaft - initial API and implementation
*
*/

plugins {
`java-library`
id("application")
}

dependencies {
implementation(libs.edc.fc.spi.crawler)
runtimeOnly(libs.fc.core)
runtimeOnly(libs.fc.ext.api)
}


23 changes: 23 additions & 0 deletions federated-catalog/fc-00-basic/fixed-node-resolver/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
* Copyright (c) 2024 Fraunhofer-Gesellschaft
*
* This program and the accompanying materials are made available under the
* terms of the Apache License, Version 2.0 which is available at
* https://www.apache.org/licenses/LICENSE-2.0
*
* SPDX-License-Identifier: Apache-2.0
*
* Contributors:
* Fraunhofer-Gesellschaft - initial API and implementation
*
*/

plugins {
`java-library`
id("application")
}

dependencies {
implementation(libs.edc.fc.spi.crawler)
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/*
* Copyright (c) 2024 Fraunhofer-Gesellschaft
*
* This program and the accompanying materials are made available under the
* terms of the Apache License, Version 2.0 which is available at
* https://www.apache.org/licenses/LICENSE-2.0
*
* SPDX-License-Identifier: Apache-2.0
*
* Contributors:
* Fraunhofer-Gesellschaft - initial API and implementation
*
*/

package org.eclipse.edc.sample.extension.fc;

import org.eclipse.edc.crawler.spi.TargetNode;
import org.eclipse.edc.crawler.spi.TargetNodeDirectory;

import java.util.ArrayList;
import java.util.List;

public class CatalogNodeDirectory implements TargetNodeDirectory {

@Override
public List<TargetNode> getAll() {
List<String> protocolList = new ArrayList<>();
protocolList.add("dataspace-protocol-http");

TargetNode participantNode = new TargetNode("https://w3id.org/edc/v0.0.1/ns/",
"provider",
"http://localhost:19194/protocol", protocolList);

List<TargetNode> targetNodes = new ArrayList<>();
targetNodes.add(participantNode);
return targetNodes;
}

@Override
public void insert(TargetNode targetNode) {

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Copyright (c) 2024 Fraunhofer-Gesellschaft
*
* This program and the accompanying materials are made available under the
* terms of the Apache License, Version 2.0 which is available at
* https://www.apache.org/licenses/LICENSE-2.0
*
* SPDX-License-Identifier: Apache-2.0
*
* Contributors:
* Fraunhofer-Gesellschaft - initial API and implementation
*
*/

package org.eclipse.edc.sample.extension.fc;

import org.eclipse.edc.crawler.spi.TargetNodeDirectory;
import org.eclipse.edc.runtime.metamodel.annotation.Provider;
import org.eclipse.edc.spi.system.ServiceExtension;

public class CatalogNodeDirectoryExtension implements ServiceExtension {
private TargetNodeDirectory nodeDirectory;


@Provider
public TargetNodeDirectory federatedCacheNodeDirectory() {
if (nodeDirectory == null) {
nodeDirectory = new CatalogNodeDirectory();
}
return nodeDirectory;
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
org.eclipse.edc.sample.extension.fc.CatalogNodeDirectoryExtension
Loading
Loading