Add documentation

README.md

@@ -1,4 +1,182 @@
-# RADAR Manual data upload Kafka connector
+# RADAR-base Data Uploader
+
+RADAR-base Data Uploader is a web-application that enables uploading data to RADAR-Base. It has a Kafka Connect source connector that processes the uploaded data, parse it in appropriate format and send them to Kafka.
+
+This can be used to 
+1. Upload data from devices and other sources using a web-interface.
+2. Monitor status of the records created
+
+# Components 
+The project contains three components:
+1. A web-application where user can upload data [radar-upload-frontend](radar-upload-frontend)
+2. A back-end webservice application that stores the uploaded data with metadata. [radar-upload-backend](radar-upload-backend)
+3. A Kafka Connect source-connector that loads data from backend connector and sends it to Kafka. [kafka-connect-upload-source](kafka-connect-upload-source)
+
+
+# Screenshots
+![login using managementportal credentials](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_003.png)
+
+![create and upload files](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_010.png)
+
+![view records and status](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_011.png)
+
+![view participants and records](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_012.png)
+
+# Usage
+If you are a user who would like to upload data to RADAR-base, you require an account on ManagementPortal application of your environment. Please request an account from your System administer if you do not have done.
+
+1. Please login to the uploader application using your account credentials from ManagementPortal.
+2. Authorize the uploading application to to perform mentioned operations by clicking on `"Approve"`.
+3. Select the project you want to work with.
+4. Click on the `"UPLOAD"` button to upload files.
+5. Select the participant and type of the file data source, then click on `"CREATE RECORD"`.
+6. Once the record is created, upload the relevant file(s), then click on `"UPLOAD"`.
+7. Once you are done with uploading files, click on `"DONE"` to finalise the process.
+8. You will see a new record created for the corresponding participant with `READY`
+9. You can switch to `RECORD` tab and monitor the status of the record.
+
+Please see the [step-by-step guide](https://radar-base.org/index.php/2019/10/31/a-step-by-step-on-how-to-manually-upload-data-radar-base/) with screenshots if you like more information.
+
+# Installation
+
+To install fully functional RADAR-base data uploader from source, please use the `docker-compose.yml` under the root directory
+
+```bash
+docker-compose up -d --build
+```
+
+
+# Usage
+
+## Configuring RADAR-base Data Uploader
+
+**Disclaimer:** This section does not cover how to install or configure the complete RADAR-Base stack. Please visit [RADAR-Docker](https://github.com/RADAR-base/RADAR-Docker) or [RADAR-Kubernetes](https://github.com/RADAR-base/RADAR-Kubernetes) for complete installation guidelines. 
+
+**Note:** Some configuration values are specified under the assumption that the docker containers will have the same name on deployment as well.
+
+Configuring Data Uploader involves configuring the three components mentioned above.
+
+### 1. Configuring radar-upload-backend
+Please copy `etc/upload.yml.template` to `etc/upload.yml` and modify the database credentials and the oauth client credentials. The following tables shows the possible properties and explanation.
+
+```yaml
+
+baseUri: "http://0.0.0.0:8085/upload/api/"
+managementPortalUrl: "http://managementportal-app:8080/managementportal/"
+jdbcDriver: "org.postgresql.Driver"
+jdbcUrl: "jdbc:postgresql://radarbase-postgresql:5432/uploadconnector"
+jdbcUser: "username" # change this to your database username
+jdbcPassword: "password" # change this to your database password
+additionalPersistenceConfig:
+  "hibernate.dialect": "org.hibernate.dialect.PostgreSQL95Dialect"
+enableCors: yes # if you want to enable cors filter to this component specify yes
+clientId: "radar_upload_backend" # configure your oauth client id
+clientSecret: "secret" # configure the client-secret
+sourceTypes: # these are the data source types that are supported to upload data
+  - name: "phone-acceleration"
+    topics:
+      - "android_phone_acceleration"
+    contentTypes:
+      - "text/csv"
+    timeRequired: false
+    sourceIdRequired: false
+    configuration:
+      "setting1": "value1"
+      "setting2": "value2"
+  - name: "acceleration-zip"
+    topics:
+      - "android_phone_acceleration"
+    contentTypes:
+      - "application/zip"
+    timeRequired: false
+    sourceIdRequired: false
+  - name: "altoida-zip"
+    topics:
+      - "connect_upload_altoida_acceleration"
+    contentTypes:
+      - "application/zip"
+    timeRequired: false
+    sourceIdRequired: false
+
+```
+
+#### Adding support to new device type
+To add support to additional device types, add a new entry of sourceType to the `sourceTypes` list.
+A single sourceType entry is defined as below.
+
+```yaml
+
+- name: "acceleration-zip" # unique identifier of the data source or device (*required)
+    topics: # list of topics to send data
+      - "android_phone_acceleration" 
+    contentTypes: # content types of the data
+      - "application/zip"
+    timeRequired: false # whether the user should supply a date-time for the upload e.g. when the data is not timestamped. 
+    sourceIdRequired: false # if source-id is compulsory to upload data, specify true.    
+``` 
+
+If the `upload.yml` file has been modified after starting the serve, restart the service
+
+```bash
+docker-compose restart radar-upload-backend
+```
+
+### 2. Configuring Kafka Source Connector
+
+Please copy `etc/source-upload.properties.template` to `etc/source-upload.properties` and modify the oauth client credentials and supported converter classes. The following tables shows the possible properties and explanation.
+
+
+<table class="data-table"><tbody>
+<tr>
+<th>Name</th>
+<th>Description</th>
+<th>Type</th>
+<th>Default</th>
+<th>Valid Values</th>
+<th>Importance</th>
+</tr>
+<tr>
+<td>upload.source.poll.interval.ms</td></td><td>How often to poll the records to process.</td></td><td>long</td></td><td>60000</td></td><td></td></td><td>low</td></td></tr>
+<tr>
+<td>upload.source.client.id</td></td><td>OAuth Client-id of the upload kafka connector.</td></td><td>string</td></td><td>radar-upload-connector-client</td></td><td></td></td><td>high</td></td></tr>
+<tr>
+<td>upload.source.client.secret</td></td><td>OAuth client-secret of the upload kafka connector.</td></td><td>string</td></td><td></td></td><td></td></td><td>high</td></td></tr>
+<tr>
+<td>upload.source.client.tokenUrl</td></td><td>Token URL of ManagementPortal to get access token.</td></td><td>string</td></td><td></td></td><td></td></td><td>high</td></td></tr>
+<tr>
+<td>upload.source.backend.baseUrl</td></td><td>URL of the radar-upload-backend where uploaded files are stored.</td></td><td>string</td></td><td>http://radar-upload-connect-backend:8085/radar-upload/</td></td><td></td></td><td>high</td></td></tr>
+<tr>
+<td>upload.source.record.converter.classes</td></td><td>List of classes to be used to convert a record.</td></td><td>list</td></td><td>org.radarbase.connect.upload.converter.phone.AccelerometerConverterFactory</td></td><td>Class extending org.radarbase.connect.upload.converter.ConverterFactory</td></td><td>high</td></td></tr>
+</tbody></table>
+
+#### Adding support to new device type
+To add processing data from new device type, please implement a ConverterFactory that can process the data from corresponding device and add the name of the class to the list of `upload.source.record.converter.classes`.
+
+### 3. Configuring radar-upload-frontend
+
+Configuring radar-upload-frontend requires configuring the right environment variables for the docker container.
+
+<table class="data-table"><tbody>
+<tr>
+<th>Name</th>
+<th>Description</th>
+<th>Values</th>
+</tr>
+<tr>
+<td>VUE_APP_BASE_URL</td><td>Base URL of the web-application.</td><td>/upload</td>
+<tr>
+<tr>
+<td>VUE_APP_API_BASE_URL</td><td>Advertised URL of the radar-upload-backend.</td><td>http://localhost:8080/upload/api/ or https://radarbase-test.domain.com/upload/api/</td>
+<tr>
+<tr>
+<td>VUE_APP_AUTH_API</td><td>Advertised URL of ManagementPortal Authorization.</td><td>http://localhost:8080/managementportal/oauth or https://radarbase-test.domain.com/managementportal/oauth</td>
+<tr>
+<tr>
+<td>VUE_APP_AUTH_CALLBACK</td><td>Callback URL to send the `authorization_code`. This should be correctly configured on ManagementPortal.</td><td>http://localhost:8080/upload/login or https://radarbase-test.domain.com/upload/login</td>
+<tr>
+<tr>
+<td>VUE_APP_CLIENT_ID</td><td>OAuth Client Id of the web application configured on ManagementPortal. </td><td>radar_upload_frontend</td>
+<tr>
+</tbody></table>
 
-Work in progress to enable direct data uploads to Kafka. Architecture plans are described in the [docs](https://github.com/RADAR-base/radar-upload-source-connector/tree/master/docs).
 

etc/source-upload.properties.template

@@ -0,0 +1,9 @@
+name=radar-upload-source
+connector.class=org.radarbase.connect.upload.UploadSourceConnector
+tasks.max=4
+upload.source.client.id=radar_upload_connect
+upload.source.client.secret=upload_secret
+upload.source.client.tokenUrl=http://managementportal-app:8080/managementportal/oauth/token
+upload.source.backend.baseUrl=http://radar-upload-connect-backend:8085/radar-upload/
+upload.source.poll.interval.ms=1800000
+upload.source.record.converter.classes=org.radarbase.connect.upload.converter.phone.AccelerometerConverterFactory,org.radarbase.connect.upload.converter.altoida.AltoidaZipConverterFactory

etc/upload.yml.template

@@ -0,0 +1,37 @@
+baseUri: "http://0.0.0.0:8085/upload/api/"
+managementPortalUrl: "http://managementportal-app:8080/managementportal/"
+jdbcDriver: "org.postgresql.Driver"
+jdbcUrl: "jdbc:postgresql://radarbase-postgresql:5432/uploadconnector"
+jdbcUser: "username"
+jdbcPassword: "password"
+additionalPersistenceConfig:
+  "hibernate.dialect": "org.hibernate.dialect.PostgreSQL95Dialect"
+sourceTypes:
+  - name: "phone-acceleration"
+    topics:
+      - "android_phone_acceleration"
+    contentTypes:
+      - "text/csv"
+    timeRequired: false
+    sourceIdRequired: false
+    configuration:
+      "setting1": "value1"
+      "setting2": "value2"
+  - name: "acceleration-zip"
+    topics:
+      - "android_phone_acceleration"
+    contentTypes:
+      - "application/zip"
+    timeRequired: false
+    sourceIdRequired: false
+  - name: "altoida-zip"
+    topics:
+      - "connect_upload_altoida_acceleration"
+    contentTypes:
+      - "application/zip"
+    timeRequired: false
+    sourceIdRequired: false
+enableCors: yes
+clientSecret: "secret"
+clientId: "radar_upload_backend"
+

kafka-connect-upload-source/src/main/java/org/radarbase/connect/upload/UploadSourceConnectorConfig.kt

@@ -93,7 +93,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map<String, S
         const val UPLOAD_SOURCE_SERVER_BASE_URL_CONFIG = "upload.source.backend.baseUrl"
         private const val UPLOAD_SOURCE_SERVER_BASE_URL_DOC = "Base URL of the file upload backend"
         private const val UPLOAD_SOURCE_SERVER_BASE_URL_DISPLAY = "Base URL of the file upload backend"
-        private const val UPLOAD_SOURCE_SERVER_BASE_URL_DEFAULT = "http://radar-upload-backend:8080/"
+        private const val UPLOAD_SOURCE_SERVER_BASE_URL_DEFAULT = "http://radar-upload-connect-backend:8085/radar-upload/"
 
         const val SOURCE_POLL_INTERVAL_CONFIG = "upload.source.poll.interval.ms"
         private const val SOURCE_POLL_INTERVAL_DOC = "How often to poll the source URL."
@@ -102,7 +102,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map<String, S
 
         const val UPLOAD_SOURCE_CONVERTERS_CONFIG = "upload.source.record.converter.classes"
         private const val UPLOAD_SOURCE_CONVERTERS_DOC = "List record converter classes that are in class-path"
-        private const val UPLOAD_SOURCE_CONVERTERS_DISPLAY = "List of record converter class"
+        private const val UPLOAD_SOURCE_CONVERTERS_DISPLAY = "List of record converter factory class"
         private val UPLOAD_SOURCE_CONVERTERS_DEFAULT: List<String> = listOf(
                 AccelerometerConverterFactory()::class.java.name,
                 AcceleratometerZipConverterFactory()::class.java.name,
@@ -125,7 +125,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map<String, S
                     .define(SOURCE_POLL_INTERVAL_CONFIG,
                             ConfigDef.Type.LONG,
                             SOURCE_POLL_INTERVAL_DEFAULT,
-                            ConfigDef.Importance.LOW,
+                            ConfigDef.Importance.HIGH,
                             SOURCE_POLL_INTERVAL_DOC,
                             groupName,
                             ++orderInGroup,
@@ -135,7 +135,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map<String, S
                     .define(UPLOAD_SOURCE_CLIENT_CONFIG,
                             ConfigDef.Type.STRING,
                             UPLOAD_SOURCE_MP_CLIENT_DEFAULT,
-                            ConfigDef.Importance.LOW,
+                            ConfigDef.Importance.HIGH,
                             UPLOAD_SOURCE_CLIENT_DOC,
                             groupName,
                             ++orderInGroup,
@@ -155,7 +155,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map<String, S
                     .define(UPLOAD_SOURCE_SERVER_BASE_URL_CONFIG,
                             ConfigDef.Type.STRING,
                             UPLOAD_SOURCE_SERVER_BASE_URL_DEFAULT,
-                            ConfigDef.Importance.HIGH,
+                            ConfigDef.Importance.LOW,
                             UPLOAD_SOURCE_SERVER_BASE_URL_DOC,
                             groupName,
                             ++orderInGroup,
@@ -165,7 +165,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map<String, S
                     .define(UPLOAD_SOURCE_CLIENT_TOKEN_URL_CONFIG,
                             ConfigDef.Type.STRING,
                             UPLOAD_SOURCE_CLIENT_TOKEN_URL_DEFAULT,
-                            ConfigDef.Importance.HIGH,
+                            ConfigDef.Importance.LOW,
                             UPLOAD_SOURCE_CLIENT_TOKEN_URL_DOC,
                             groupName,
                             ++orderInGroup,