Merge pull request #125 from RADAR-base/release-1.1.0

Release-1.1.0

.travis.yml

@@ -35,10 +35,10 @@ script:
   - echo $"SMARTHOST_ADDRESS=mail.example.com\nSMARTHOST_PORT=587\[email protected]\nSMARTHOST_PASSWORD=XXXXXXXX" > etc/smtp.env
   - sudo docker volume create certs
   - sudo docker volume create certs-data
-  - cp etc/radar.yml.template etc/radar.yml
+  - cp etc/radar-backend/radar.yml.template etc/radar-backend/radar.yml
   - cp etc/webserver/nginx.conf.template etc/webserver/nginx.conf
-  - cp etc/sink-hdfs.properties.template etc/sink-hdfs.properties
-  - cp etc/sink-mongo.properties.template etc/sink-mongo.properties
+  - cp etc/hdfs-connector/sink-hdfs.properties.template etc/hdfs-connector/sink-hdfs.properties
+  - cp etc/mongodb-connector/sink-mongo.properties.template etc/mongodb-connector/sink-mongo.properties
   - sudo $HOME/bin/docker-compose up -d --build && sleep 15 && [ -z "$($HOME/bin/docker-compose ps | tail -n +3 | grep " Exit ")" ]
   - sudo $HOME/bin/docker-compose down
   - sudo docker network rm hadoop

README.md

@@ -1,12 +1,12 @@
 # RADAR-Docker
 
-The dockerized RADAR stack for deploying the RADAR-CNS platform. Component repositories can be found at [RADAR-CNS DockerHub org](https://hub.docker.com/u/radarcns/dashboard/)
+The dockerized RADAR stack for deploying the RADAR-base platform. Component repositories can be found at [RADAR-base DockerHub org](https://hub.docker.com/u/radarbase/dashboard/)
 
 ## Installation instructions 
-To install RADAR-CNS stack, do the following: 
+To install RADAR-base stack, do the following: 
 
 1. Install [Docker Engine](https://docs.docker.com/engine/installation/)
-2. Install `docker-compose` using the [installation guide](https://docs.docker.com/compose/install/) or by following our [wiki](https://github.com/RADAR-CNS/RADAR-Docker/wiki/How-to-set-up-docker-on-ubuntu#install-docker-compose).
+2. Install `docker-compose` using the [installation guide](https://docs.docker.com/compose/install/) or by following our [wiki](https://github.com/RADAR-base/RADAR-Docker/wiki/How-to-set-up-docker-on-ubuntu#install-docker-compose).
 3. Verify the Docker installation by running on the command-line:
 
     ```shell
@@ -21,10 +21,10 @@ To install RADAR-CNS stack, do the following:
         sudo apt-get install git
         ```
 
-5. Clone [RADAR-Docker](https://github.com/RADAR-CNS/RADAR-Docker) repository from GitHub.
+5. Clone [RADAR-Docker](https://github.com/RADAR-base/RADAR-Docker) repository from GitHub.
 
     ```shell
-    git clone https://github.com/RADAR-CNS/RADAR-Docker.git
+    git clone https://github.com/RADAR-base/RADAR-Docker.git
     ```
 
 6. Install required component stack following the instructions below.
@@ -34,7 +34,7 @@ To install RADAR-CNS stack, do the following:
 RADAR-Docker currently offers two component stacks to run.
 
 1. A Docker-compose for components from [Confluent Kafka Platform](http://docs.confluent.io/3.1.0/) community 
-2. A Docker-compose for components from RADAR-CNS platform.
+2. A Docker-compose for components from RADAR-base platform.
 
 > **Note**: on macOS, remove `sudo` from all `docker` and `docker-compose` commands in the usage instructions below.
 
@@ -54,23 +54,23 @@ To stop this stack, run:
 sudo docker-compose down
 ```
 
-### RADAR-CNS platform
+### RADAR-base platform
 
-In addition to Confluent Kafka platform components, RADAR-CNS platform offers
+In addition to Confluent Kafka platform components, RADAR-base platform offers
 
 * RADAR-HDFS-Connector - Cold storage of selected streams in Hadoop data storage,
 * RADAR-MongoDB-Connector - Hot storage of selected streams in MongoDB,
-* [RADAR-Dashboard](https://github.com/RADAR-CNS/RADAR-Dashboard),
+* [RADAR-Dashboard](https://github.com/RADAR-base/RADAR-Dashboard),
 * RADAR-Streams - real-time aggregated streams,
 * RADAR-Monitor - Status monitors,
-* [RADAR-HotStorage](https://github.com/RADAR-CNS/RADAR-HotStorage) via MongoDB, 
-* [RADAR-REST API](https://github.com/RADAR-CNS/RADAR-RestApi),
+* [RADAR-HotStorage](https://github.com/RADAR-base/RADAR-HotStorage) via MongoDB, 
+* [RADAR-REST API](https://github.com/RADAR-base/RADAR-RestApi),
 * A Hadoop cluster, and
 * An email server.
 * Management Portal - A web portal to manage patient monitoring studies.
 * RADAR-Gateway - A validating gateway to allow only valid and authentic data to the platform
 * Catalog server - A Service to share source-types configured in the platform.
-To run RADAR-CNS stack in a single node setup:
+To run RADAR-base stack in a single node setup:
 
 1. Navigate to `radar-cp-hadoop-stack`:
 

dcompose-stack/radar-cp-hadoop-stack/README.md

@@ -5,6 +5,8 @@ This docker-compose stack contains the full operational RADAR platform. Once con
 ## Configuration
 
 1. First copy `etc/env.template` file to `./.env` and check and modify all its variables.
+
+
    1.1. To have a valid HTTPS connection for a public host, set `SELF_SIGNED_CERT=no`. You need to provide a public valid DNS name as `SERVER_NAME` for SSL certificate to work. IP addresses will not work. 
 
    1.2. Set `MANAGEMENTPORTAL_FRONTEND_CLIENT_SECRET` to a secret to be used by the Management Portal frontend.
@@ -13,16 +15,16 @@ This docker-compose stack contains the full operational RADAR platform. Once con
 
 2. Copy `etc/smtp.env.template` to `etc/smtp.env` and configure your email settings. Configure alternative mail providers like Amazon SES or Gmail by using the parameters of the [`namshi/smtp` Docker image](https://hub.docker.com/r/namshi/smtp/).
 
-3. Copy `etc/redcap-integration/radar.yml.template` to `etc/redcap-integration/radar.yml` and modify it to configure the properties of Redcap instance and the management portal. For reference on configuration of this file look at the Readme file here - <https://github.com/RADAR-CNS/RADAR-RedcapIntegration#configuration>. In the REDcap portal under Project Setup, define the Data Trigger as `https://<YOUR_HOST_URL>/redcapint/trigger`
+3. Copy `etc/redcap-integration/radar.yml.template` to `etc/redcap-integration/radar.yml` and modify it to configure the properties of Redcap instance and the management portal. For reference on configuration of this file look at the Readme file here - <https://github.com/RADAR-base/RADAR-RedcapIntegration#configuration>. In the REDcap portal under Project Setup, define the Data Trigger as `https://<YOUR_HOST_URL>/redcapint/trigger`
 
 4. Copy `etc/managementportal/config/oauth_client_details.csv.template` to `etc/managementportal/config/oauth_client_details.csv` and change OAuth client credentials for production MP. The OAuth client for the frontend will be loaded automatically and does not need to be listed in this file. This file will be read at each startup. The current implementation overwrites existing clients with the same client ID, so be aware of this if you have made changes to a client listed in this file using the Management Portal frontend. This behaviour might change in the future.
 
-5. Finally, copy `etc/radar.yml.template` to `etc/radar.yml` and edit it, especially concerning the monitor email address configuration.
+5. Finally, copy `etc/radar-backend/radar.yml.template` to `etc/radar-backend/radar.yml` and edit it, especially concerning the monitor email address configuration.
 
 6. (Optional) Note: To have different flush.size for different topics, you can create multipe property configurations for a single connector. To do that,
 
 	6.1 Create multipe property files that have different `flush.size` for given topics.
-	Examples [sink-hdfs-high.properties](https://github.com/RADAR-CNS/RADAR-Docker/blob/dev/dcompose-stack/radar-cp-hadoop-stack/etc/sink-hdfs-high.properties) , [sink-hdfs-low.properties](https://github.com/RADAR-CNS/RADAR-Docker/blob/dev/dcompose-stack/radar-cp-hadoop-stack/etc/sink-hdfs-low.properties)
+	Examples [sink-hdfs-high.properties](https://github.com/RADAR-base/RADAR-Docker/blob/dev/dcompose-stack/radar-cp-hadoop-stack/etc/sink-hdfs-high.properties) , [sink-hdfs-low.properties](https://github.com/RADAR-base/RADAR-Docker/blob/dev/dcompose-stack/radar-cp-hadoop-stack/etc/sink-hdfs-low.properties)
 
 	6.2 Add `CONNECTOR_PROPERTY_FILE_PREFIX: <prefix-value>` environment variable to `radar-hdfs-connector` service in `docker-compose` file.
 
@@ -104,7 +106,7 @@ CSV-structured data can be gotten from HDFS by running
 ```shell
 ./hdfs_restructure.sh /topicAndroidNew <destination directory>
 ```
-This will put all CSV files in the destination directory, with subdirectory structure `PatientId/SensorType/Date_Hour.csv`.
+This will put all CSV files in the destination directory, with subdirectory structure `ProjectId/SubjectId/SensorType/Date_Hour.csv`.
 
 ## Cerificate
 
@@ -128,7 +130,7 @@ Portainer provides simple interactive UI-based docker management. If running loc
 
 ### Kafka Manager
 
-The [kafka-manager](https://github.com/yahoo/kafka-manager) is an interactive web based tool for managing Apache Kafka. Kafka manager has beed integrated in the stack. It is accessible at <http://localhost/kafkamanager/>
+The [kafka-manager](https://github.com/yahoo/kafka-manager) is an interactive web based tool for managing Apache Kafka. Kafka manager has beed integrated in the stack. It is accessible at `http://<your-host>/kafkamanager/`
 
 ### Check Health
 Each of the containers in the stack monitor their own health and show the output as healthy or unhealthy. A script called check-health.sh is used to check this output and send an email to the maintainer if a container is unhealthy.
@@ -137,13 +139,13 @@ First check that the `MAINTAINER_EMAIL` in the .env file is correct.
 
 Then make sure that the SMTP server is configured properly and running.
 
-If systemd integration is enabled, the check-health.sh script will check health of containers every five minutes. It can then be run directly by running
+If systemd integration is enabled, the check-health.sh script will check health of containers every five minutes. It can then be run directly by running if systemd wrappers have been installed
 ```
 sudo systemctl start radar-check-health.service
 ```
 Otherwise, the following manual commands can be invoked.
 
-Then just add a cron job to run the `check-health.sh` script periodically like -
+Add a cron job to run the `check-health.sh` script periodically like -
 1. Edit the crontab file for the current user by typing `$ crontab -e`
 2. Add your job and time interval. For example, add the following for checking health every 5 mins - 
 

dcompose-stack/radar-cp-hadoop-stack/check-health.sh

@@ -45,10 +45,12 @@ while read service; do
     fi
 done <<< "$(sudo-linux docker-compose config --services)"
 
+display_host="${SERVER_NAME} ($(hostname -f), $(curl -s http://ipecho.net/plain))"
+
 if [ "${#unhealthy[@]}" -eq 0 ]; then
     if [ -f .unhealthy ]; then
          rm -f .unhealthy
-         hipchat_notify green "All services are healthy again"
+         hipchat_notify green "All services on ${display_host} are healthy again"
     fi
     echo "All services are healthy"
 else
@@ -57,14 +59,15 @@ else
     # Send notification to MAINTAINER
     # start up the mail container if not already started
     sudo-linux docker-compose up -d smtp
+    # ensure that all topics are available
+    sudo-linux docker-compose run --rm kafka-init
     # save the container, so that we can use exec to send an email later
     container=$(sudo-linux docker-compose ps -q smtp)
     SAVEIFS=$IFS
     IFS=,
     display_services="[${unhealthy[*]}]"
     IFS=$SAVEIFS
-    display_host="${SERVER_NAME} ($(hostname -f), $(curl -s http://ipecho.net/plain))"
-    body="Services on $display_host are unhealthy. Services $display_services have been restarted. Please log in for further information."
+    body="Services on ${display_host} are unhealthy. Services $display_services have been restarted. Please log in for further information."
     echo "Sent notification to $MAINTAINER_EMAIL"
     echo "$body" | sudo-linux docker exec -i ${container} mail -aFrom:$FROM_EMAIL "-s[RADAR] Services on ${SERVER_NAME} unhealthy" $MAINTAINER_EMAIL