Merge pull request #74 from robmarkcole/add-boxes-to-events

Adds box to event data

README.md

@@ -14,7 +14,7 @@ docker pull deepquestai/deepstack:noavx
 
 **Legacy machine users** If you are using a machine that doesn't support avx or you are having issues with making requests, Deepstack has a specific build for these systems. Use `deepquestai/deepstack:noavx` instead of `deepquestai/deepstack` when you are installing or running Deepstack. I expect many users will be using noavx mode so I will use it in the examples below.
 
-## Activating the API
+## Activating the Deepstack API
 Before you get started, you will need to activate the Deepstack API. First, go to www.deepstack.cc and sign up for an account. Choose the basic plan which will give us unlimited access for one installation. You will then see an activation key in your portal.
 
 On your machine with docker, run Deepstack (noavx mode) without any recognition so you can activate the API on port `5000`:
@@ -27,27 +27,26 @@ Now go to http://YOUR_SERVER_IP_ADDRESS:5000/ on another computer or the same on
 docker run -e VISION-DETECTION=True -e API-KEY="Mysecretkey" -v localstorage:/datastore -p 5000:5000 --name deepstack -d deepquestai/deepstack:noavx
 ```
 
+## Usage of this component
+The `deepstack_object` component adds an `image_processing` entity where the state of the entity is the total number of `target` objects that are above a `confidence` threshold which has a default value of 80%. The time of the last detection of the `target` object is in the `last detection` attribute. The type and number of objects (of any confidence) is listed in the `summary` attributes. Optionally the processed image can be saved to disk. If `save_file_folder` is configured two images are created, one with the filename of format `deepstack_latest_{target}.jpg` which is over-written on each new detection of the `target`, and another with a unique filename including the timestamp. An event `image_processing.object_detected` is fired for each object detected. If you are a power user with advanced needs such as zoning detections or you want to track multiple object types, you will need to use the `image_processing.object_detected` events.
+
 ## Home Assistant setup
 Place the `custom_components` folder in your configuration directory (or add its contents to an existing `custom_components` folder). Then configure object detection. **Important:** It is necessary to configure only a single camera per `deepstack_object` entity. If you want to process multiple cameras, you will therefore need multiple `deepstack_object` `image_processing` entities. **Note** that at we can use `scan_interval` to (optionally) limit computation, [as described here](https://www.home-assistant.io/components/image_processing/#scan_interval-and-optimising-resources).
 
-The `deepstack_object` component adds an `image_processing` entity where the state of the entity is the total number of `target` objects that are above a `confidence` threshold which has a default value of 80%. The time of the last positive detection of the `target` object is in the `last detection` attribute. The class and number of objects all objects detected (of any confidence) is listed in the `summary` attributes. An event `image_processing.object_detected` is fired for each object detected. Optionally the processed image can be saved to disk. If `save_file_folder` is configured two images are created, one with the filename of format `deepstack_latest_{target}.jpg` which is over-written on each new detection of the `target`, and another with a unique filename including the timestamp. You can use a [local_file](https://www.home-assistant.io/integrations/local_file/) camera to display the `deepstack_latest_{target}.jpg` image on the HA front-end.
-
 Add to your Home-Assistant config:
+
 ```yaml
 image_processing:
   - platform: deepstack_object
     ip_address: localhost
     port: 5000
     api_key: Mysecretkey
-    timeout: 5
-    scan_interval: 20
     save_file_folder: /config/www/deepstack_person_images
-    target: person
-    confidence: 50
     source:
       - entity_id: camera.local_file
         name: person_detector
 ```
+
 Configuration variables:
 - **ip_address**: the ip address of your deepstack instance.
 - **port**: the port of your deepstack instance.
@@ -67,12 +66,36 @@ Configuration variables:
 <img src="https://github.com/robmarkcole/HASS-Deepstack-object/blob/master/docs/object_detail.png" width="350">
 </p>
 
+#### Event `image_processing.file_saved`
+If `save_file_folder` is configured, an new image will be saved with bounding boxes of detected `target` objects, and the filename will include the time of the image capture. On saving this image a `image_processing.file_saved` event is fired, with a payload that includes:
+
+- `entity_id` : the entity id responsible for the event
+- `file` : the full path to the saved file
+
+An example automation using the `image_processing.file_saved` event is given below, which sends a Telegram message with the saved file:
+
+```yaml
+- action:
+  - data_template:
+      caption: "Captured {{ trigger.event.data.file }}"
+      file: "{{ trigger.event.data.file }}"
+    service: telegram_bot.send_photo
+  alias: New person alert
+  condition: []
+  id: '1120092824611'
+  trigger:
+  - platform: event
+    event_type: image_processing.file_saved
+```
+
 #### Event `image_processing.object_detected`
 An event `image_processing.object_detected` is fired for each object detected above the configured `confidence` threshold. This is the recommended way to check the confidence of detections, and to keep track of objects that are not configured as the `target` (configure logger level to `debug` to observe events in the Home Assistant logs). An example use case for event is to get an alert when some rarely appearing object is detected, or to increment a [counter](https://www.home-assistant.io/components/counter/). The `image_processing.object_detected` event payload includes:
 
 - `entity_id` : the entity id responsible for the event
 - `object` : the object detected
 - `confidence` : the confidence in detection in the range 0 - 1 where 1 is 100% confidence.
+- `box` : the bounding box of the object
+- `centroid` : the centre point of the object
 
 An example automation using the `image_processing.object_detected` event is given below:
 
@@ -92,27 +115,11 @@ An example automation using the `image_processing.object_detected` event is give
       object: person
 ```
 
-#### Event `image_processing.file_saved`
-If `save_file_folder` is configured, an new image will be saved with bounding boxes of detected `target` objects, and the filename will include the time of the image capture. On saving this image a `image_processing.file_saved` event is fired, with a payload that includes:
+The `box` coordinates and the box center (`centroid`) can be used to determine whether an object falls within a defined region-of-interest (ROI). This can be useful to include/exclude objects by their location in the image.
 
-- `entity_id` : the entity id responsible for the event
-- `file` : the full path to the saved file
-
-An example automation using the `image_processing.file_saved` event is given below, which sends a Telegram message with the saved file:
+* The `box` is defined by the tuple `(y_min, x_min, y_max, x_max)` (equivalent to image top, left, bottom, right) where the coordinates are floats in the range `[0.0, 1.0]` and relative to the width and height of the image.
+* The centroid is in `(x,y)` coordinates where `(0,0)` is the top left hand corner of the image and `(1,1)` is the bottom right corner of the image.
 
-```yaml
-- action:
-  - data_template:
-      caption: "Captured {{ trigger.event.data.file }}"
-      file: "{{ trigger.event.data.file }}"
-    service: telegram_bot.send_photo
-  alias: New person alert
-  condition: []
-  id: '1120092824611'
-  trigger:
-  - platform: event
-    event_type: image_processing.file_saved
-```
 
 ## Displaying the `deepstack_latest_{target}.jpg` file
 It easy to display the `deepstack_latest_{target}.jpg` image with a [local_file](https://www.home-assistant.io/components/local_file/) camera. An example configuration is:

custom_components/deepstack_object/image_processing.py

@@ -51,6 +51,8 @@
 DEFAULT_TIMEOUT = 10
 EVENT_OBJECT_DETECTED = "image_processing.object_detected"
 EVENT_FILE_SAVED = "image_processing.file_saved"
+BOX = "box"
+CENTROID = "centroid"
 FILE = "file"
 OBJECT = "object"
 
@@ -86,6 +88,20 @@ def get_box(prediction: dict, img_width: int, img_height: int):
     return box
 
 
+def get_box_centroid(box: Tuple) -> Tuple:
+    """
+    Locate the box centroid in (x,y) coordinates where
+    (0,0) is the top left hand corner of the image and 
+    (1,1) is the bottom right corner of the image.
+    """
+    rounding_decimals = 3
+
+    y_min, x_min, y_max, x_max = box
+    centroid = ((x_max + x_min) / 2, (y_max + y_min) / 2)
+    centroid = [round(coord, rounding_decimals) for coord in centroid]
+    return centroid
+
+
 def draw_box(
     draw: ImageDraw,
     box: Tuple[float, float, float, float],
@@ -261,12 +277,15 @@ def fire_prediction_events(self, predictions, confidence):
 
         for prediction in predictions:
             if ds.format_confidence(prediction["confidence"]) > confidence:
+                box = get_box(prediction, self._image_width, self._image_height)
                 self.hass.bus.fire(
                     EVENT_OBJECT_DETECTED,
                     {
                         ATTR_ENTITY_ID: self.entity_id,
                         OBJECT: prediction["label"],
                         ATTR_CONFIDENCE: ds.format_confidence(prediction["confidence"]),
+                        BOX: box,
+                        CENTROID: get_box_centroid(box),
                     },
                 )
 
@@ -303,8 +322,9 @@ def unit_of_measurement(self):
     def device_state_attributes(self):
         """Return device specific state attributes."""
         attr = {}
-        attr["target"] = self._target
         if self._last_detection:
-            attr["last_detection"] = self._last_detection.strftime("%Y-%m-%d %H:%M:%S")
+            attr[
+                "last_{}_detection".format(self._target)
+            ] = self._last_detection.strftime("%Y-%m-%d %H:%M:%S")
         attr["summary"] = self._summary
         return attr