diff --git a/README.md b/README.md index 461d0e47..35928474 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,182 @@ -# RADAR Manual data upload Kafka connector +# RADAR-base Data Uploader + +RADAR-base Data Uploader is a web-application that enables uploading data to RADAR-Base. It has a Kafka Connect source connector that processes the uploaded data, parse it in appropriate format and send them to Kafka. + +This can be used to +1. Upload data from devices and other sources using a web-interface. +2. Monitor status of the records created + +# Components +The project contains three components: +1. A web-application where user can upload data [radar-upload-frontend](radar-upload-frontend) +2. A back-end webservice application that stores the uploaded data with metadata. [radar-upload-backend](radar-upload-backend) +3. A Kafka Connect source-connector that loads data from backend connector and sends it to Kafka. [kafka-connect-upload-source](kafka-connect-upload-source) + + +# Screenshots +![login using managementportal credentials](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_003.png) + +![create and upload files](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_010.png) + +![view records and status](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_011.png) + +![view participants and records](https://raw.githubusercontent.com/RADAR-base/radar-upload-source-connector/master/docs/assets/Selection_012.png) + +# Usage +If you are a user who would like to upload data to RADAR-base, you require an account on ManagementPortal application of your environment. Please request an account from your System administer if you do not have done. + +1. Please login to the uploader application using your account credentials from ManagementPortal. +2. Authorize the uploading application to to perform mentioned operations by clicking on `"Approve"`. +3. Select the project you want to work with. +4. Click on the `"UPLOAD"` button to upload files. +5. Select the participant and type of the file data source, then click on `"CREATE RECORD"`. +6. Once the record is created, upload the relevant file(s), then click on `"UPLOAD"`. +7. Once you are done with uploading files, click on `"DONE"` to finalise the process. +8. You will see a new record created for the corresponding participant with `READY` +9. You can switch to `RECORD` tab and monitor the status of the record. + +Please see the [step-by-step guide](https://radar-base.org/index.php/2019/10/31/a-step-by-step-on-how-to-manually-upload-data-radar-base/) with screenshots if you like more information. + +# Installation + +To install fully functional RADAR-base data uploader from source, please use the `docker-compose.yml` under the root directory + +```bash +docker-compose up -d --build +``` + + +# Usage + +## Configuring RADAR-base Data Uploader + +**Disclaimer:** This section does not cover how to install or configure the complete RADAR-Base stack. Please visit [RADAR-Docker](https://github.com/RADAR-base/RADAR-Docker) or [RADAR-Kubernetes](https://github.com/RADAR-base/RADAR-Kubernetes) for complete installation guidelines. + +**Note:** Some configuration values are specified under the assumption that the docker containers will have the same name on deployment as well. + +Configuring Data Uploader involves configuring the three components mentioned above. + +### 1. Configuring radar-upload-backend +Please copy `etc/upload.yml.template` to `etc/upload.yml` and modify the database credentials and the oauth client credentials. The following tables shows the possible properties and explanation. + +```yaml + +baseUri: "http://0.0.0.0:8085/upload/api/" +managementPortalUrl: "http://managementportal-app:8080/managementportal/" +jdbcDriver: "org.postgresql.Driver" +jdbcUrl: "jdbc:postgresql://radarbase-postgresql:5432/uploadconnector" +jdbcUser: "username" # change this to your database username +jdbcPassword: "password" # change this to your database password +additionalPersistenceConfig: + "hibernate.dialect": "org.hibernate.dialect.PostgreSQL95Dialect" +enableCors: yes # if you want to enable cors filter to this component specify yes +clientId: "radar_upload_backend" # configure your oauth client id +clientSecret: "secret" # configure the client-secret +sourceTypes: # these are the data source types that are supported to upload data + - name: "phone-acceleration" + topics: + - "android_phone_acceleration" + contentTypes: + - "text/csv" + timeRequired: false + sourceIdRequired: false + configuration: + "setting1": "value1" + "setting2": "value2" + - name: "acceleration-zip" + topics: + - "android_phone_acceleration" + contentTypes: + - "application/zip" + timeRequired: false + sourceIdRequired: false + - name: "altoida-zip" + topics: + - "connect_upload_altoida_acceleration" + contentTypes: + - "application/zip" + timeRequired: false + sourceIdRequired: false + +``` + +#### Adding support to new device type +To add support to additional device types, add a new entry of sourceType to the `sourceTypes` list. +A single sourceType entry is defined as below. + +```yaml + +- name: "acceleration-zip" # unique identifier of the data source or device (*required) + topics: # list of topics to send data + - "android_phone_acceleration" + contentTypes: # content types of the data + - "application/zip" + timeRequired: false # whether the user should supply a date-time for the upload e.g. when the data is not timestamped. + sourceIdRequired: false # if source-id is compulsory to upload data, specify true. +``` + +If the `upload.yml` file has been modified after starting the serve, restart the service + +```bash +docker-compose restart radar-upload-backend +``` + +### 2. Configuring Kafka Source Connector + +Please copy `etc/source-upload.properties.template` to `etc/source-upload.properties` and modify the oauth client credentials and supported converter classes. The following tables shows the possible properties and explanation. + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionTypeDefaultValid ValuesImportance
upload.source.poll.interval.msHow often to poll the records to process.long60000low
upload.source.client.idOAuth Client-id of the upload kafka connector.stringradar-upload-connector-clienthigh
upload.source.client.secretOAuth client-secret of the upload kafka connector.stringhigh
upload.source.client.tokenUrlToken URL of ManagementPortal to get access token.stringhigh
upload.source.backend.baseUrlURL of the radar-upload-backend where uploaded files are stored.stringhttp://radar-upload-connect-backend:8085/radar-upload/high
upload.source.record.converter.classesList of classes to be used to convert a record.listorg.radarbase.connect.upload.converter.phone.AccelerometerConverterFactoryClass extending org.radarbase.connect.upload.converter.ConverterFactoryhigh
+ +#### Adding support to new device type +To add processing data from new device type, please implement a ConverterFactory that can process the data from corresponding device and add the name of the class to the list of `upload.source.record.converter.classes`. + +### 3. Configuring radar-upload-frontend + +Configuring radar-upload-frontend requires configuring the right environment variables for the docker container. + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionValues
VUE_APP_BASE_URLBase URL of the web-application./upload
VUE_APP_API_BASE_URLAdvertised URL of the radar-upload-backend.http://localhost:8080/upload/api/ or https://radarbase-test.domain.com/upload/api/
VUE_APP_AUTH_APIAdvertised URL of ManagementPortal Authorization.http://localhost:8080/managementportal/oauth or https://radarbase-test.domain.com/managementportal/oauth
VUE_APP_AUTH_CALLBACKCallback URL to send the `authorization_code`. This should be correctly configured on ManagementPortal.http://localhost:8080/upload/login or https://radarbase-test.domain.com/upload/login
VUE_APP_CLIENT_IDOAuth Client Id of the web application configured on ManagementPortal. radar_upload_frontend
-Work in progress to enable direct data uploads to Kafka. Architecture plans are described in the [docs](https://github.com/RADAR-base/radar-upload-source-connector/tree/master/docs). diff --git a/docs/assets/Selection_003.png b/docs/assets/Selection_003.png new file mode 100644 index 00000000..26d6b05c Binary files /dev/null and b/docs/assets/Selection_003.png differ diff --git a/docs/assets/Selection_010.png b/docs/assets/Selection_010.png new file mode 100644 index 00000000..5e897b62 Binary files /dev/null and b/docs/assets/Selection_010.png differ diff --git a/docs/assets/Selection_011.png b/docs/assets/Selection_011.png new file mode 100644 index 00000000..038e3809 Binary files /dev/null and b/docs/assets/Selection_011.png differ diff --git a/docs/assets/Selection_012.png b/docs/assets/Selection_012.png new file mode 100644 index 00000000..ee35b5de Binary files /dev/null and b/docs/assets/Selection_012.png differ diff --git a/etc/source-upload.properties.template b/etc/source-upload.properties.template new file mode 100644 index 00000000..84315e75 --- /dev/null +++ b/etc/source-upload.properties.template @@ -0,0 +1,9 @@ +name=radar-upload-source +connector.class=org.radarbase.connect.upload.UploadSourceConnector +tasks.max=4 +upload.source.client.id=radar_upload_connect +upload.source.client.secret=upload_secret +upload.source.client.tokenUrl=http://managementportal-app:8080/managementportal/oauth/token +upload.source.backend.baseUrl=http://radar-upload-connect-backend:8085/radar-upload/ +upload.source.poll.interval.ms=1800000 +upload.source.record.converter.classes=org.radarbase.connect.upload.converter.phone.AccelerometerConverterFactory,org.radarbase.connect.upload.converter.altoida.AltoidaZipConverterFactory diff --git a/etc/upload.yml.template b/etc/upload.yml.template new file mode 100644 index 00000000..6d464a89 --- /dev/null +++ b/etc/upload.yml.template @@ -0,0 +1,37 @@ +baseUri: "http://0.0.0.0:8085/upload/api/" +managementPortalUrl: "http://managementportal-app:8080/managementportal/" +jdbcDriver: "org.postgresql.Driver" +jdbcUrl: "jdbc:postgresql://radarbase-postgresql:5432/uploadconnector" +jdbcUser: "username" +jdbcPassword: "password" +additionalPersistenceConfig: + "hibernate.dialect": "org.hibernate.dialect.PostgreSQL95Dialect" +sourceTypes: + - name: "phone-acceleration" + topics: + - "android_phone_acceleration" + contentTypes: + - "text/csv" + timeRequired: false + sourceIdRequired: false + configuration: + "setting1": "value1" + "setting2": "value2" + - name: "acceleration-zip" + topics: + - "android_phone_acceleration" + contentTypes: + - "application/zip" + timeRequired: false + sourceIdRequired: false + - name: "altoida-zip" + topics: + - "connect_upload_altoida_acceleration" + contentTypes: + - "application/zip" + timeRequired: false + sourceIdRequired: false +enableCors: yes +clientSecret: "secret" +clientId: "radar_upload_backend" + diff --git a/kafka-connect-upload-source/src/main/java/org/radarbase/connect/upload/UploadSourceConnectorConfig.kt b/kafka-connect-upload-source/src/main/java/org/radarbase/connect/upload/UploadSourceConnectorConfig.kt index b2450d0a..079ade61 100644 --- a/kafka-connect-upload-source/src/main/java/org/radarbase/connect/upload/UploadSourceConnectorConfig.kt +++ b/kafka-connect-upload-source/src/main/java/org/radarbase/connect/upload/UploadSourceConnectorConfig.kt @@ -93,7 +93,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map = listOf( AccelerometerConverterFactory()::class.java.name, AcceleratometerZipConverterFactory()::class.java.name, @@ -125,7 +125,7 @@ class UploadSourceConnectorConfig(config: ConfigDef, parsedConfig: Map