[#19] Enhance device communication API

* Add support for device states * Add support for request/response commands * Add support for ack-required command type and change configs to use ack-required type * Add retry mechanism for sending configs * Add automatic Pub/Sub tenant topic & subscription creation and deletion * Add automatically sending of a config in case a config is requested or a new one created * Add possibility to change log level * Update Quarkus dependency and migrated javax dependencies to jakarta * Improve SQL connection handling * Format and clean up device communication api code Signed-off-by: Matthias Kaemmer <[email protected]>
sotec-iot · Jun 3, 2024 · 8f1a11e · 8f1a11e
1 parent c048ec4
commit 8f1a11e
Show file tree

Hide file tree

Showing 96 changed files with 5,097 additions and 1,401 deletions.
diff --git a/device-communication/README.md b/device-communication/README.md
@@ -1,18 +1,18 @@
 # Device Communication API
 
 Device communication API enables users and applications to send configurations and commands to devices via HTTP(S)
-endpoints.
+endpoints as well as retrieve states of devices.
 
 ![img.png](img.png)
 
 ### Application
 
-The application is reactive and uses Quarkus Framework for the application and Vertx tools for the HTTP server.
+The application is reactive and uses the Quarkus framework for the application and Vert.x tools for the HTTP server.
 
 ### Hono internal communication
 
-API uses [Google's PubSub](https://cloud.google.com/pubsub/docs/overview?hl=de) service to communicate with the command
-router.
+The application uses [Google's PubSub](https://cloud.google.com/pubsub/docs/overview?hl=de) service to communicate with
+the command router.
 
 ## API endpoints
 
@@ -32,32 +32,41 @@ router.
 
 - POST: create a device config version
 
-For more information please see resources/api/openApi file.
+For more information please see resources/api/hono-endpoint.yaml file.
 
 ## Pub/Sub - Internal Messaging
 
-API communicates with hono components via the internal messaging interface (implemented from Google's PubSub).
-All the settings for the InternalMessaging component are in the application.yaml file. By publish/subscribe to a topic
-application sends or expects some message attributes.
+The application communicates with hono components via the internal messaging interface (implemented for Google PubSub).
+The settings for the internal messaging component can be found in the application.yaml file.
 
 ### Events
 
-API will subscribe to all tenants' event topic at startup.
+The application subscribes to all tenants' event topic at startup to listen for config requests.
 
-Expected message Attributes:
+#### MQTT config request (empty notification)
+
+Expected message attributes:
 
 - deviceId
 - tenantId
-- content-type
+- content-type (must be "application/vnd.eclipse-hono-empty-notification")
+- ttd (must be -1)
 
-Application will <b>proceed only empty Notifications events (content-type is
-application/vnd.eclipse-hono-empty-notification)</b>.
+#### HTTP config request
+
+Expected message attributes:
+
+- deviceId
+- tenantId
+- ttd (must not be blank or empty)
+- orig_adapter (must be "hono-http")
+- orig_address (must contain "config")
 
 ### States
 
-API will subscribe to all tenants' state topic at startup.
+The application subscribes to all tenants' state topic at startup.
 
-Expected message Attributes:
+Expected message attributes:
 
 - deviceId
 - tenantId
@@ -66,49 +75,71 @@ States are read only.
 
 ### Configs
 
-Application will publish the latest device configuration when:
+The application publishes the latest device configuration when:
 
-- an empty Notifications event was received
-- a new device config was created
+- An empty notification event is received (MQTT device subscribed to the command topic)
+- A config request from an HTTP device is received
+- A new device config is created
 
-Message will be published with the following attributes:
+Message attributes:
 
 - deviceId
 - tenantId
-- config-version
+- correlation-id (the config version)
+- subject (always set to "config")
+- ack-required (always set to true)
+
+Body:
 
-The Body will be a JSON object with the device config object.
+A JSON object with the device config object.
 
-After publishing device configs, application subscribes to config_response topic and waits for the device to ack the
-configs.
+After publishing a device config, the application waits internally for an acknowledgement from the device on the
+command_response topic to update the "device_ack_time" in the database.
 
 ### Config ACK
 
 Expected message attributes:
 
 - deviceId
 - tenantId
-- configVersion (the config version received from device)
-
-If configVersion is not set, application will ack always the latest config.
+- correlation-id (the config version)
+- content-type (must be "application/vnd.eclipse-hono-delivery-success-notification+json")
+- subject (must be "config")
 
 ### Commands
 
-A command will be published from API to the command topic.
+A command will be published from the application to the command topic.
 
 Attributes:
 
 - deviceId
 - tenantId
-- subject (always set to "command")
+- subject (if not specified set to "command")
+- response-required (optional)
+- ack-required (optional)
+- correlation-id (optional)
+- timeout (optional)
 
 Body:
 
 The command as string.
 
+If ack-required is set to "true", the request will wait for the duration of the specified timeout (in milliseconds)
+for an acknowledgement from the device on the command_response topic.
+
+### Command ACK
+
+Expected message attributes:
+
+- deviceId
+- tenantId
+- correlation-id
+- content-type (must be "application/vnd.eclipse-hono-delivery-success-notification+json")
+- subject (must not be "config")
+
 ## Database
 
-Application uses PostgresSQL database. All the database configurations can be found in application.yaml file.
+The application uses a PostgreSQL database. All the database configurations can be found in the application.yaml file.
 
 ### Tables
 
@@ -121,29 +152,29 @@ Application uses PostgresSQL database. All the database configurations can be fo
 
 ### Migrations
 
-When Applications starts tables will be created by the DatabaseSchemaCreator service.
+When the application starts the tables will be created by the DatabaseSchemaCreator service.
 
-### Running postgresSQL container local
+### Running PostgresSQL container locally
 
-For running the PostgresSQL Database local with docker run:
+The docker run command for running the PostgreSQL database locally:
 
 ``````
 
 docker run -p 5432:5432 --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres
 
 ``````
 
-After the container is running, log in to the container and with psql create the database. Then we have
-to set the application settings.
+After the container is running, log in to the container and with psql create the database. Then we have to set the
+application settings.
 
-Default postgresSQl values:
+Default PostgreSQl values:
 
 - userName = postgres
 - password = mysecretpassword
 
-## Build and Push API Docker Image
+## Build and push the application's Docker image
 
-Mavens auto build and push functionality can be enabled from application.yaml settings:
+Mavens auto build and push functionality can be enabled in the application.yaml settings:
 
 ````
 quarkus:
@@ -154,45 +185,49 @@ quarkus:
   image: "eclipse/hono-device-communication"
 ````
 
-By running maven package, install or deploy will automatically build the docker image and if push is enabled it will
-push the image
-to the given registry.
+By running maven package, install or deploy the docker image will automatically be built and if push is enabled it will
+push the image to the registry specified in the image path.
 
 ## OpenApi Contract-first
 
-For creating the endpoints, Vertx takes the openApi definition file and maps every endpoint operation-ID with a specific
-Handler
-function.
+For creating the endpoints, Vert.x takes the openApi definition file and maps every endpoint operation-ID with a
+specific handler function.
 
 ## Handlers
 
-Handlers are providing callBack functions for every endpoint. Functions are going to be called automatically from vertx
-server every time a request is received.
+Handlers are providing callBack functions for every endpoint. Functions are going to be called automatically from the
+Vert.x server every time a request is received.
 
-## Adding a new Endpoint
+## Adding a new endpoint
 
-Adding new Endpoint steps:
+Adding new endpoint steps:
 
-1. Add Endpoint in openApi file and set an operationId
-2. Use an existing const Class or create a new one under /config and set the operation id name
-3. Implement an HttpEndpointHandler and set the Routes
+1. Add endpoint in openApi (hono-device-communication-v1.yaml) and swagger (hono-endpoint.yaml) file and set an
+   operationId in the openApi file
+2. Depending on the endpoint, create a new HttpEndpointHandler or use an existing one (in case it's a new one, add it to
+   the availableHandlerServices within the VertxHttpHandlerManagerService)
+3. Implement the method and set the routes within the handler
 
 ## PubSub Events
 
-Application subscribes and uses to the following topics:
+The application publishes and subscribes to the following topics:
 
-1. TENANT_ID.command
-2. TENANT_ID.command_response
-3. TENANT_ID.event
-4. TENANT_ID.event.state
-5. registry-tenant.notification
+publish:
+* TENANT_ID.command
 
-## Automatically create PubSub topics and subscriptions
+subscribe:
+* TENANT_ID.command_response
+* TENANT_ID.event
+* TENANT_ID.event.state
+* registry-tenant.notification
 
-Application creates all tenants topics and subscriptions when:
+## Automatically create and delete PubSub topics and subscriptions
 
-1. Application starts if are not exist
-2. New tenant is created
+The application creates tenant topics and subscriptions when:
 
+* The application starts, if they do not exist already
+* A new tenant is created
 
+The application deletes tenant topics and subscriptions when:
 
+* A tenant is deleted
diff --git a/device-communication/pom.xml b/device-communication/pom.xml
@@ -13,18 +13,19 @@
         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
         <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
         <quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
-        <quarkus.platform.version>2.15.3.Final</quarkus.platform.version>
+        <quarkus.platform.version>3.2.6.Final</quarkus.platform.version>
         <skipITs>true</skipITs>
         <surefire-plugin.version>3.0.0-M9</surefire-plugin.version>
         <org.mapstruct.version>1.5.3.Final</org.mapstruct.version>
         <hono.version>2.1.0</hono.version>
-        <vertx-pg-client.version>4.3.7</vertx-pg-client.version>
+        <vertx-pg-client.version>4.4.6</vertx-pg-client.version>
         <jakarta.persistence-api.version>2.2.3</jakarta.persistence-api.version>
         <com.ongres.scram.client.version>2.1</com.ongres.scram.client.version>
-        <quarkus-container-image-docker.version>2.16.0.Final</quarkus-container-image-docker.version>
-        <quarkus-google-cloud-pubsub.version>1.3.0</quarkus-google-cloud-pubsub.version>
+        <quarkus-container-image-docker.version>3.2.6.Final</quarkus-container-image-docker.version>
+        <quarkus-google-cloud-pubsub.version>2.6.0</quarkus-google-cloud-pubsub.version>
         <maven-surefire-plugin.version>3.0.0-M9</maven-surefire-plugin.version>
         <com.google.guava.version>31.1-jre</com.google.guava.version>
+        <mockito-inline.version>5.2.0</mockito-inline.version>
     </properties>
     <dependencyManagement>
         <dependencies>
@@ -84,6 +85,7 @@
         <dependency>
             <groupId>org.mockito</groupId>
             <artifactId>mockito-inline</artifactId>
+            <version>${mockito-inline.version}</version>
             <scope>test</scope>
         </dependency>
 
@@ -97,6 +99,14 @@
             <artifactId>jakarta.persistence-api</artifactId>
             <version>${jakarta.persistence-api.version}</version>
         </dependency>
+        <dependency>
+            <groupId>jakarta.inject</groupId>
+            <artifactId>jakarta.inject-api</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>jakarta.enterprise</groupId>
+            <artifactId>jakarta.enterprise.cdi-api</artifactId>
+        </dependency>
         <dependency>
             <groupId>com.ongres.scram</groupId>
             <artifactId>client</artifactId>
@@ -152,25 +162,23 @@
         <dependency>
             <groupId>org.eclipse.hono</groupId>
             <artifactId>hono-core</artifactId>
-            <version>2.4.0-SNAPSHOT</version>
+            <version>2.5.0-SNAPSHOT</version>
         </dependency>
         <dependency>
             <groupId>org.eclipse.hono</groupId>
             <artifactId>hono-client-pubsub-common</artifactId>
-            <version>2.4.0-SNAPSHOT</version>
+            <version>2.5.0-SNAPSHOT</version>
         </dependency>
         <dependency>
             <groupId>org.eclipse.hono</groupId>
             <artifactId>hono-client-notification</artifactId>
-            <version>2.4.0-SNAPSHOT</version>
+            <version>2.5.0-SNAPSHOT</version>
         </dependency>
         <dependency>
             <groupId>commons-codec</groupId>
             <artifactId>commons-codec</artifactId>
             <version>1.15</version>
         </dependency>
-
-
     </dependencies>
     <build>
         <plugins>

diff --git a/device-communication/src/main/docker/Dockerfile.jvm b/device-communication/src/main/docker/Dockerfile.jvm
@@ -88,6 +88,6 @@ COPY --chown=185 target/quarkus-app/quarkus/ /deployments/quarkus/
 
 EXPOSE 8080
 USER 185
-ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
+ENV JAVA_OPTS="-Djava.util.logging.manager=org.jboss.logmanager.LogManager"
 ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"
 
diff --git a/device-communication/src/main/docker/Dockerfile.legacy-jar b/device-communication/src/main/docker/Dockerfile.legacy-jar
@@ -90,5 +90,5 @@ COPY target/*-runner.jar /deployments/quarkus-run.jar
 
 EXPOSE 8080
 USER 185
-ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
+ENV JAVA_OPTS="-Djava.util.logging.manager=org.jboss.logmanager.LogManager"
 ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"
diff --git a/device-communication/src/main/docker/Dockerfile.native b/device-communication/src/main/docker/Dockerfile.native
@@ -24,4 +24,4 @@ COPY --chown=1001:root target/*-runner /work/application
 EXPOSE 8080
 USER 1001
 
-CMD ["./application", "-Dquarkus.http.host=0.0.0.0"]
+CMD ["./application"]
diff --git a/device-communication/src/main/docker/Dockerfile.native-micro b/device-communication/src/main/docker/Dockerfile.native-micro
@@ -27,4 +27,4 @@ COPY --chown=1001:root target/*-runner /work/application
 EXPOSE 8080
 USER 1001
 
-CMD ["./application", "-Dquarkus.http.host=0.0.0.0"]
+CMD ["./application"]