Skip to content

Commit

Permalink
ci: finalise 3.6.0 (#1441)
Browse files Browse the repository at this point in the history
* feat: add version in header (#1412)

* feat: add version in header

* fix: PR remarks

* fix: missing export

* chore: move controller advice

* fix: broken IT

* fix: duplicate bean

* fix: duplicate bean

* fix: duplicate bean

* chore: additional test

* chore(deps): bump rexml from 3.2.5 to 3.3.9 in /docs (#1406)

Bumps [rexml](https://github.com/ruby/rexml) from 3.2.5 to 3.3.9.
- [Release notes](https://github.com/ruby/rexml/releases)
- [Changelog](https://github.com/ruby/rexml/blob/master/NEWS.md)
- [Commits](ruby/rexml@v3.2.5...v3.3.9)

---
updated-dependencies:
- dependency-name: rexml
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump webrick from 1.8.1 to 1.8.2 in /docs (#1397)

Bumps [webrick](https://github.com/ruby/webrick) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/ruby/webrick/releases)
- [Commits](ruby/webrick@v1.8.1...v1.8.2)

---
updated-dependencies:
- dependency-name: webrick
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: cleanup retired linear caching (#1413)

* fix: misplaced version number (#1415)

chore: bump zonky embedded db

* feat: kafka in + bump 3.6.0 (#1414)

* wip: first setup

* feat: working kafka listener

* feat: working kafka source in admin

* fix: update bean name in tests

* feat: working kafka source in admin

* feat: add integration test

* feat: add integration test

* fix test

* fix test

* chore: add kafka jar

* sonar

* add assembly

* docs: documentation effort

* chore: pr remarks

* ci: prepare release 3.6.0

---------

Co-authored-by: Yalz <[email protected]>
# Conflicts:
#	ldes-server-admin/ldes-server-admin-common/pom.xml
#	ldes-server-admin/ldes-server-admin-rest/pom.xml
#	ldes-server-admin/pom.xml
#	ldes-server-application/pom.xml
#	ldes-server-domain/pom.xml
#	ldes-server-fetch/ldes-server-fetch-common/pom.xml
#	ldes-server-fetch/ldes-server-fetch-rest/pom.xml
#	ldes-server-fetch/pom.xml
#	ldes-server-fragmentation/ldes-server-fragmentation-common/pom.xml
#	ldes-server-fragmentation/ldes-server-fragmentation-geospatial/pom.xml
#	ldes-server-fragmentation/ldes-server-fragmentation-reference/pom.xml
#	ldes-server-fragmentation/ldes-server-fragmentation-timebased-hierarchical/pom.xml
#	ldes-server-fragmentation/ldes-server-pagination/pom.xml
#	ldes-server-fragmentation/pom.xml
#	ldes-server-infra-postgres/pom.xml
#	ldes-server-infra-postgres/postgres-admin-repository/pom.xml
#	ldes-server-infra-postgres/postgres-fetch-repository/pom.xml
#	ldes-server-infra-postgres/postgres-fragmentation-repository/pom.xml
#	ldes-server-infra-postgres/postgres-ingest-repository/pom.xml
#	ldes-server-infra-postgres/postgres-liquibase/pom.xml
#	ldes-server-infra-postgres/postgres-liquibase/src/main/resources/db/changelog/master.xml
#	ldes-server-infra-postgres/postgres-maintenance-repository/pom.xml
#	ldes-server-infra-postgres/postgres-pagination-repository/pom.xml
#	ldes-server-ingest/ldes-server-ingest-common/pom.xml
#	ldes-server-ingest/ldes-server-ingest-rest/pom.xml
#	ldes-server-ingest/pom.xml
#	ldes-server-integration-test/pom.xml
#	ldes-server-maintenance/ldes-server-compaction/pom.xml
#	ldes-server-maintenance/ldes-server-maintenance-common/pom.xml
#	ldes-server-maintenance/ldes-server-retention/pom.xml
#	ldes-server-maintenance/pom.xml
#	pom.xml

* chore: refactor-prefix-adder (#1421)

* feat: addition of well known prefixes
feat: creation of ldes-name and view-name prefixes

* feat: separation of PrefixConstructor
feat: tests added for Prefixes
feat: new impl for PrefixAdder

* feat: cleanup and configuration test added

* chore: small refactor of config and uri creators

* fix: broken tests

* fix: bad property name used

* feat: missing http presence handling in PrefixConstructor

* chore: improve tests

* docs: prefixes

* chore: make version header to all endpoints accessible (#1422)

* chore: upgrade deprecated gh actions (#1423)

* chore: upgrade deprecated gh actions

* chore: upgrade deprecated gh actions

* fix: well-known-prefixes.csv was not loaded in Docker image (#1425)

* fix: unnecessary hibernate deep copy (#1424)

* feat: usage of a projection

* chore: implement PR remarks

* fix: lenient version ingest (#1426)


---------

Co-authored-by: Yalz <[email protected]>

* Revert "fix: lenient version ingest (#1426)"

This reverts commit 072c00e.

* chore: return 404 when updating event source of non existing event stream (#1427)

* fix: lock collection row to prevent interleaved sequence numbers in member_id when ingesting members (#1430)

Co-authored-by: Jan Robert <[email protected]>

* fix: wrong view stats update (#1431)

* fix: wrong member pagination count in view stats
chore: cleanup fragmentation service

* fix: broken test

* chore: implement PR remarks
chore: add extra tests

* fix/ fix admin view shacl shape (#1433)

* chore: dcat catalog repository: remove unreachable logic (#1435)

* chore: remove unreachable logic

* chore: remove unreachable logic

* chore: make flaky test fail less

---------

Co-authored-by: Yalz <[email protected]>

* feat: keep launching jobs until all work is done (#1436)

* feat: use while loop to check if there is still fragmentation work to do

* feat: fire event when there is fragmentation work to do

* chore: extend tests

* chore: delete kafka source in db when listener is removed (#1439)

* chore: delete kafka source in db when listener is removed

* chore: delete kafka source in db when listener is removed

* chore: delete kafka source in db when listener is removed

---------

Co-authored-by: Yalz <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Jonas Bulcke <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yalz <[email protected]>
Co-authored-by: Ranko Orlic <[email protected]>
Co-authored-by: pj-cegeka <[email protected]>
  • Loading branch information
6 people authored Dec 4, 2024
1 parent 55a739e commit 1c61290
Show file tree
Hide file tree
Showing 204 changed files with 2,939 additions and 6,649 deletions.
2 changes: 2 additions & 0 deletions .github/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ COPY ./ldes-server-ingest/ldes-server-ingest-rest/target/ldes-server-ingest-rest
COPY ./ldes-server-fetch/ldes-server-fetch-rest/target/ldes-server-fetch-rest-jar-with-dependencies.jar ./lib/
COPY ./ldes-server-admin/ldes-server-admin-rest/target/ldes-server-admin-rest-jar-with-dependencies.jar ./lib/

COPY ./ldes-server-ingest/ldes-server-ingest-kafka/target/ldes-server-ingest-kafka-jar-with-dependencies.jar ./lib/

# Plugin Fragmentations
COPY ./ldes-server-fragmentation/ldes-server-fragmentation-geospatial/target/ldes-server-fragmentation-geospatial-jar-with-dependencies.jar ./lib/
COPY ./ldes-server-fragmentation/ldes-server-fragmentation-timebased-hierarchical/target/ldes-server-fragmentation-timebased-hierarchical-jar-with-dependencies.jar ./lib/
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/3.approve-snapshot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
echo "Already released. Current version: $VERSION"
fi
- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
uses: peter-evans/create-pull-request@v7
if: ${{ env.NEW_VERSION != '' }}
with:
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/add-issue-to-project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
name: Add issue to project
runs-on: ubuntu-latest
steps:
- uses: actions/add-to-project@v0.4.0
- uses: actions/add-to-project@v1
with:
project-url: https://github.com/orgs/Informatievlaanderen/projects/4
github-token: ${{ secrets.ADD_TO_PROJECT_PAT }}
4 changes: 2 additions & 2 deletions .github/workflows/build-documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
export VERSION=$(mvn help:evaluate -Dexpression="project.version" -q -DforceStdout)
echo "version=$VERSION" >> $>> $GITHUB_OUTPUT
- name: Setup Ruby
uses: ruby/setup-ruby@0a29871fe2b0200a17a4497bae54fe5df0d973aa
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.0' # Not needed with a .ruby-version file
bundler-cache: true # runs 'bundle install' and caches installed gems automatically
Expand All @@ -47,7 +47,7 @@ jobs:
bundle exec jekyll build --baseurl "${{env.base_url}}/${{steps.version.outputs.version}}" -s "./" -d "./_site"
env:
JEKYLL_ENV: production
- uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 16
registry-url: https://npm.pkg.github.com/
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/build-project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,18 +50,18 @@ jobs:
needs: build
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- name: Download JARs
uses: actions/download-artifact@v4
with:
name: artifacts
path: .github
- name: Build and push Docker image
uses: docker/build-push-action@v3
uses: docker/build-push-action@v6
with:
context: .github
push: false
14 changes: 7 additions & 7 deletions .github/workflows/pr-merged.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
id: determine-version
run: |
export VERSION=$(mvn help:evaluate -Dexpression="project.version" -q -DforceStdout)
echo "::set-output name=version::$VERSION"
echo "version=$VERSION" >> $GITHUB_ENV
- name: Analyse & publish package
run: |
mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:sonar deploy -Dsonar.projectKey=Informatievlaanderen_VSDS-LDESServer4J -Pcoverage
Expand All @@ -77,9 +77,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- name: Download JARs
uses: actions/download-artifact@v4
with:
Expand All @@ -100,28 +100,28 @@ jobs:
fi
- name: Log in to the DockerHub Container registry
if: github.ref_name == 'main'
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Log in to the GitHub Container registry
if: github.ref_name != 'main'
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: Informatievlaanderen
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4
uses: docker/metadata-action@v5
with:
images: ${{ env.IMAGES }}
tags: |
type=raw,value=${{env.IMAGE_TAG}}
type=raw,value=${{env.LATEST}}
- name: Build and push Docker image
uses: docker/build-push-action@v3
uses: docker/build-push-action@v6
with:
context: .github
push: true
Expand Down
25 changes: 13 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,18 +52,19 @@ Each maven profile represents a different functionality of the LDES server that
The default exported image contains all the profiles, but a custom image can be created with only the needed
dependencies.

| Profile | Dependencies | Description |
|----------------------------|--------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| `fragmentation` | `postgres-pagination-repository` | Allows basic fragmentation (pagination) |
| `maintenance` | `postgres-maintenance-repository` | Allows the LDES server to perform maintenance operations on its Event Streams and Members: compaction, retention, deletion. |
| **Interfaces** | | |
| `http-admin` | `ldes-server-admin-rest`,`postgres-admin-repository`* | Gives access to REST API to create and manage Event Streams and Views. |
| `http-ingest` | `ldes-server-ingest-rest`,`postgres-ingest-repository` | Gives access to REST API to ingest members into the LDES. |
| `http-fetch` | `ldes-server-fetch-rest`,`postgres-fetch-repository` | Gives access to REST API to fetch Event Streams, its Views and pages. |
| **Plugin Fragmentations** | | |
| `fragmentation-timebased` | `ldes-server-fragmentation-timebased-hierarchical` | Allows fragmentation in based on a timebased property. |
| `fragmentation-geospatial` | `ldes-server-fragmentation-geospatial` | Allows fragmentation in based on a geospatial property. |
| `fragmentation-reference` | `ldes-server-fragmentation-reference` | Allows fragmentation in based on a textual property. |
| Profile | Dependencies | Description |
|----------------------------|---------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| `fragmentation` | `postgres-pagination-repository` | Allows basic fragmentation (pagination) |
| `maintenance` | `postgres-maintenance-repository` | Allows the LDES server to perform maintenance operations on its Event Streams and Members: compaction, retention, deletion. |
| **Interfaces** | | |
| `http-admin` | `ldes-server-admin-rest`,`postgres-admin-repository`* | Gives access to REST API to create and manage Event Streams and Views. |
| `http-ingest` | `ldes-server-ingest-rest`,`postgres-ingest-repository` | Gives access to REST API to ingest members into the LDES. |
| `kafka-ingest` | `ldes-server-ingest-kafka`,`postgres-ingest-repository` | Allows Kafka member ingestion into the LDES. |
| `http-fetch` | `ldes-server-fetch-rest`,`postgres-fetch-repository` | Gives access to REST API to fetch Event Streams, its Views and pages. |
| **Plugin Fragmentations** | | |
| `fragmentation-timebased` | `ldes-server-fragmentation-timebased-hierarchical` | Allows fragmentation in based on a timebased property. |
| `fragmentation-geospatial` | `ldes-server-fragmentation-geospatial` | Allows fragmentation in based on a geospatial property. |
| `fragmentation-reference` | `ldes-server-fragmentation-reference` | Allows fragmentation in based on a textual property. |


*: The `postgres-admin-repository`, as shown by the dependency graph, will be loaded in by the other above-mentioned functionality profiles.
Expand Down
4 changes: 4 additions & 0 deletions content/dependency-graph.dot
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ digraph "ldes-server" {
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-common:jar:compile"[label=<ldes-server-ingest-common>]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-fetch-rest:jar:compile"[label=<ldes-server-fetch-rest>, fillcolor="#87CEFA", style="filled"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-fetch-common:jar:compile"[label=<ldes-server-fetch-common>]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-kafka:jar:compile"[label=<ldes-server-ingest-kafka>, fillcolor="#87CEFA", style="filled"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-retention:jar:compile"[label=<ldes-server-retention>]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-maintenance-common:jar:compile"[label=<ldes-server-maintenance-common>]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-compaction:jar:compile"[label=<ldes-server-compaction>]
Expand All @@ -33,6 +34,7 @@ digraph "ldes-server" {
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-fragmentation-reference:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-fragmentation-common:jar:compile"
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-admin-rest:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-admin-common:jar:compile"
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-rest:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-common:jar:compile"
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-kafka:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-common:jar:compile"
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-fetch-rest:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-fetch-common:jar:compile"
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-fetch-rest:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-admin-common:jar:compile"
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-retention:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-maintenance-common:jar:compile"
Expand All @@ -52,6 +54,7 @@ digraph "ldes-server" {
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:postgres-liquibase:jar:compile" [label="Database Changes"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-rest:jar:compile" [label="Ingest REST API"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:postgres-ingest-repository:jar:compile" [label="Ingest Implementation"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-ingest-kafka:jar:compile" [label="Ingest Kafka"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-fetch-rest:jar:compile" [label="Fetch REST API"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:postgres-fetch-repository:jar:compile" [label="Fetch Implementation"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:postgres-pagination-repository:jar:compile" [label="Basic Fragmentation Implementation"]
Expand All @@ -60,4 +63,5 @@ digraph "ldes-server" {
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-fragmentation-reference:jar:compile" [label="Plugin Fragmentation"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:ldes-server-admin-rest:jar:compile" [label="Admin REST API"]
"be.vlaanderen.informatievlaanderen.vsds:ldes-server-application:jar:compile" -> "be.vlaanderen.informatievlaanderen.vsds:postgres-maintenance-repository:jar:compile" [label="Maintenance Implementation"]

}
Binary file modified content/ldes-server-graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
71 changes: 71 additions & 0 deletions docker-compose/kafka/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
version: "3.3"
services:
ldes-server:
container_name: basic_ldes-server
image: ldes/ldes-server
environment:
- SPRING_CONFIG_LOCATION=/config/
- PYROSCOPE_CONFIGURATION_FILE=/config/pyroscope.properties
volumes:
- ./docker-compose/server.config.yml:/config/application.yml:ro
- ./docker-compose/pyroscope.properties:/config/pyroscope.properties:ro
ports:
- "8080:8080"
- "8087:8087"
- "8088:8088"
- "8089:8089"
networks:
- ldes
depends_on:
- postgres
postgres:
container_name: ldes-postgres
image: postgres:14-alpine
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=admin
- POSTGRES_USER=admin
- POSTGRES_DB=test
networks:
- ldes
pyroscope:
image: grafana/pyroscope:latest
ports:
- 4040:4040
networks:
- ldes

zookeeper:
image: zookeeper:3.7.0
ports:
- "2181:2181"

kafka:
image: confluentinc/cp-kafka:7.3.2
ports:
- "9092:9092"
- "29092:29092"
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://0.0.0.0:29092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
ALLOW_PLAINTEXT_LISTENER: 'yes'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

kafka-ui:
image: provectuslabs/kafka-ui:latest
depends_on:
- kafka
ports:
- "9200:8080"
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
networks:
ldes:
4 changes: 2 additions & 2 deletions docs/Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ GEM
rb-fsevent (0.11.2)
rb-inotify (0.10.1)
ffi (~> 1.0)
rexml (3.2.5)
rexml (3.3.9)
rouge (3.30.0)
rubyzip (2.3.2)
safe_yaml (1.0.5)
Expand All @@ -68,7 +68,7 @@ GEM
sass-listen (4.0.0)
rb-fsevent (~> 0.9, >= 0.9.4)
rb-inotify (~> 0.9, >= 0.9.7)
webrick (1.8.1)
webrick (1.8.2)

PLATFORMS
x64-mingw-ucrt
Expand Down
4 changes: 4 additions & 0 deletions docs/_configuration/event-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ server:generic-eventstream a ldes:EventStream ;
genericES:shape a sh:NodeShape .
````

## Configuring an Event Stream with a Kafka source

To configure an Event Stream that ingests members from a Kafka topic, please visit the [Kafka Ingest documentation](../ingest/kafka).

## Configuring a SHACL Shape

[SHACL (Shapes Constraint Language)](https://www.w3.org./TR/shacl/) is a standard for validating RDF data and ensuring
Expand Down
32 changes: 0 additions & 32 deletions docs/_configuration/fragmentations/timebased.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ Timebased fragmentation will create fragments based on a time selected from the
tree:fragmentationStrategy [
a tree:HierarchicalTimeBasedFragmentation ;
tree:maxGranularity { Mandatory: defines the depth level of the fragments } ;
tree:linearTimeCachingEnabled { Optional: indicates if fragments should be cached by linear time (true/false) } ;
tree:fragmentationPath { Mandatory: defines which property will be used for bucketizing } ;
tree:fragmenterSubjectFilter { Optional: regex to filter the subjects matching the fragmentationPath } ;
] .
Expand All @@ -30,37 +29,6 @@ For maxGranularity the following values are allowed:
* minute,
* second.

### Linear time caching
This configuration which is off by default makes it possible to activate intelligent caching for time based data.
When you have a data with timestamps that keep increasing and are always current or newer it possible to improve
the caching. **We always assume that future members will have future timestamps.**

**For example**
timestamp: 21/02/2024

Will result in fragments:
- root
- year=2024
- year=2024&month=02
- year=2024&month=02&day=21

We will assume that no new year will be added to the root until 31/12/2024.
We will assume that no new month will be added to year=2024 until 29/02/2024.
We will assume that no new day will be added to year=2024&month=02 until 21/02/2024 23:59:59.

When a new day is added later, for example 22/02/2024. All other days and all its pages in year=2024&month=02 will
become immutable. We always assume that future members will have future timestamps.

The above logic makes the server a lot more efficient for this kind of timebased data. Clients won't have to
keep track of an increasing number of open fragments.

This optional feature is disabled by default and needs to be activated
by adding `tree:linearTimeCachingEnabled true ;` to the config.

**Historical data seeding**
This feature will still work when you seed a new server with historical data as long as you ingest the data
in chronological order. This is because fragments do not become immutable until a fragment further in time is added.

## Algorithm

1. The fragmentationObjects of the member are determined
Expand Down
Loading

0 comments on commit 1c61290

Please sign in to comment.