diff --git a/.gitignore b/.gitignore old mode 100644 new mode 100755 diff --git a/Q08_object_counter/README.md b/Q08_object_counter/README.md new file mode 100755 index 0000000..8c6d7b4 --- /dev/null +++ b/Q08_object_counter/README.md @@ -0,0 +1,471 @@ +# Object Counter Application + +## Application: Overview +The Object Counter Application is a user-friendly and efficient generic software tool that can be used to create custom counting applications for any scenario. This application uses the advanced YOLOV3/Tiny YOLOv3 algorithm to identify and count objects in images or videos. + +### Use Cases +The Generic Counter Application is a powerful tool that can be used to count objects in a variety of settings, including: + +- **Animal Counting**: The application can be fine tuned to count the animals only. This application can be used for zoo or farm monitoring, also could be used to prevent the road hazards due to animal interference. The list of animals on which the AI model is trained is available in [animal_class.txt](./exe/animal/animal_class.txt) + +- **Vehicle Counting**: The application can be fine tuned to count the vehicle instances per frame. This application can then be used for traffic monitoring at government/corporate buildings.The list of vehicles on which the AI model is trained is available in [vehicle_class.txt](./exe/vehicle/vehicle_class.txt) + +- **General Counting**: The general counting applications can be used to count any type of object, from people and cars to inventory and products. They are often used in businesses to track customer traffic, inventory levels, and employee productivity. The list of objects on which the AI model is trained is available in [coco_class.txt](./exe/coco/coco_class.txt) + +The other use cases could be: + +- **Manufacturing**: The application can be used to count parts on a production line or to measure the output of a machine. +- **Retail**: The application can be used to count products on a shelf or to track the number of customers in a store. +- **Safety**: The application can be used to count people in a crowd or to monitor the traffic flow in a city. + +### Key Features +Here are some of the key features of the Generic Counter Application: + +- **Automatic Object Detection**: + The application utilizes YOLOv3/Tiny YOLOv3 model for detection, identifying and localizing objects specified within the provided frame. +- **Flexible**: + The application can be customized to meet the specific needs of any counting scenario. +- **Customizable Settings**: + Users can adjust the detection and classification parameters by using the config file provided in the repository. + +It has following camera input modes. +- Using MIPI Camera +- Using USB Camera + +Users can select detection target from following list +- Animal +- Vehicle +- General (COCO dataset) + +### Supported Product +- RZ/V2L Evaluation Board Kit (RZ/V2L EVK) +- RZ/V2H Evaluation Board Kit (RZ/V2H EVK) + +### Demo +Following is the demo for RZ/V2L EVK. + + +## Application: Requirements + +### Hardware Requirements + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ForEquipmentDetails
RZ/V2LRZ/V2L EVKEvaluation Board Kit for RZ/V2L.
Includes followings. +
    +
  • + MIPI Camera Module(Google Coral Camera)
    + Used as a camera input source. +
    • +
  • MicroUSB to Serial Cable for serial communication.
    • +
+
AC AdapterUSB Power Delivery adapter for the board power supply.
MicroHDMI CableUsed to connect the HDMI Monitor and the board.
+ RZ/V2L EVK has microHDMI port.
RZ/V2HRZ/V2H EVKEvaluation Board Kit for RZ/V2H.
AC AdapterUSB Power Delivery adapter for the board power supply.
+ 100W is required.
HDMI CableUsed to connect the HDMI Monitor and the board.
+ RZ/V2H EVK has HDMI port.
USB CameraUsed as a camera input source.
CommonUSB Cable Type-CConnect AC adapter and the board.
HDMI MonitorUsed to display the graphics of the board.
microSD cardUsed as the filesystem.
+ Must have over 4GB capacity of blank space.
+ Operating Environment: Transcend UHS-I microSD 300S 16GB
Linux PCUsed to build application and setup microSD card.
+ Operating Environment: Ubuntu 20.04
SD card readerUsed for setting up microSD card.
USB HubUsed to connect USB Keyboard and USB Mouse to the board.
USB KeyboardUsed to type strings on the terminal of board.
USB MouseUsed to operate the mouse on the screen of board.
+ +>**Note:** All external devices will be attached to the board and does not require any driver installation (Plug n Play Type) + +Connect the hardware as shown below. + +|RZ/V2L EVK | RZ/V2H EVK | +|:---|:---| +|| | + +>**Note 1:** When using the keyboard connected to RZ/V Evaluation Board, the keyboard layout and language are fixed to English. +**Note 2:** For RZ/V2H EVK, there are USB 2.0 and USB 3.0 ports. +USB camera needs to be connected to appropriate port based on its requirement. + + +## Application: Build Stage + +>**Note:** User can skip to the [next stage (deploy)](#application-deploy-stage) if they do not want to build the application. +All pre-built binaries are provided. + +### Prerequisites +This section expects the user to have completed Step 5 of [Getting Started Guide](https://renesas-rz.github.io/rzv_ai_sdk/latest/getting_started.html) provided by Renesas. + +After completion of the guide, the user is expected of following things. +- AI SDK setup is done. +- Following docker container is running on the host machine. + |Board | Docker container | + |:---|:---| + |RZ/V2L EVK|`rzv2l_ai_sdk_container` | + |RZ/V2H EVK|`rzv2h_ai_sdk_container` | + + >**Note:** Docker environment is required for building the sample application. + + +### Application File Generation +1. On your host machine, copy the repository from the GitHub to the desired location. + 1. It is recommended to copy/clone the repository on the `data` folder, which is mounted on the Docker container. + ```sh + cd /data + git clone https://github.com/renesas-rz/rzv_ai_sdk.git + ``` + >Note: This command will download the whole repository, which include all other applications. + If you have already downloaded the repository of the same version, you may not need to run this command. + +2. Run (or start) the docker container and open the bash terminal on the container. +E.g., for RZ/V2L, use the `rzv2l_ai_sdk_container` as the name of container created from `rzv2l_ai_sdk_image` docker image. + > Note that all the build steps/commands listed below are executed on the docker container bash terminal. + +3. Set your clone directory to the environment variable. + ```sh + export PROJECT_PATH=/drp-ai_tvm/data/rzv_ai_sdk + ``` +3. Go to the application source code directory. + ```sh + cd ${PROJECT_PATH}/Q08_object_counter/ + ``` + |Board | `SRC_DIR` | + |:---|:---| + |RZ/V2L EVK|`src` | + |RZ/V2H EVK|`src_v2h` | + +4. Create and move to the `build` directory. + ```sh + mkdir -p build && cd build + `````` +5. Build the application by following the commands below. + ```sh + cmake -DCMAKE_TOOLCHAIN_FILE=./toolchain/runtime.cmake .. + make -j$(nproc) + ``` +6. The following application file would be generated in the `${PROJECT_PATH}/Q08_object_counter//build` directory + - object_counter + + +## Application: Deploy Stage +### Prerequisites +This section expects the user to have completed Step 7-1 of [Getting Started Guide](https://renesas-rz.github.io/rzv_ai_sdk/latest/getting_started.html#step7) provided by Renesas. + +After completion of the guide, the user is expected of following things. +- microSD card setup is done. + +### File Configuration +For the ease of deployment all the deployable files and folders are provided in following folders. +|Board | `EXE_DIR` | +|:---|:---| +|RZ/V2L EVK|[exe_v2l](./exe_v2l) | +|RZ/V2H EVK|[exe_v2h](./exe_v2h) | + +Each folder contains following items. +|File | Details | +|:---|:---| +|coco/tinyyolov3_onnx | **[RZ/V2L only]** Model object files for Coco Detection | +|coco/yolov3_onnx | **[RZ/V2H only]** Model object files for Coco Detection | +|coco/coco_class.txt | Label list for Coco Detection | +|coco/config.ini | User input model config object | +|animal/animal_onnx | Model object files for Animal Detection | +|animal/animal_class.txt | Label list for Animal Detection | +|animal/config.ini | User input model config object | +|vehicle/vehicle_onnx | Model object files for Vehicle Detection | +|vehicle/vehicle_class.txt | Label list for Vehicle Detection | +|vehicle/config.ini | User input model config object | +|app_conf.ini | User input application config object | +|object_counter | Application file | + + +### Instruction +1. **[For RZ/V2H only]** Run following commands to download the necessary file. +Replace each variable according to your board. + ```sh + cd /data/Q08_object_counter/ + wget / + ``` + | Target | `EXE_PATH` |`SO_FILE` |`URL` | + |:---|:---|:---|:---| + |Animal|[exe_v2h/animal/animal_onnx](./exe_v2h/animal/animal_onnx) |`Q08_object_counter_animal_deploy_tvm_v2h-v210.so` |[Release v3.00](https://github.com/renesas-rz/rzv_ai_sdk/releases/tag/v3.00/) | + |Vehicle|[exe_v2h/vehicle/vehicle_onnx](./exe_v2h/vehicle/vehicle_onnx) |`Q08_object_counter_vehicle_deploy_tvm_v2h-v210.so` |[Release v3.00](https://github.com/renesas-rz/rzv_ai_sdk/releases/tag/v3.00/) | + |COCO|[exe_v2h/coco/yolov3_onnx](./exe_v2h/coco/yolov3_onnx) |`Q08_object_counter_coco_deploy_tvm_v2h-v210.so` |[Release v3.00](https://github.com/renesas-rz/rzv_ai_sdk/releases/tag/v3.00/) | + + - E.g., for Animal counting, use following commands. + ```sh + cd /data/Q08_object_counter/exe_v2h/animal/animal_onnx + wget https://github.com/renesas-rz/rzv_ai_sdk/releases/download/v3.00/Q08_object_counter_animal_deploy_tvm_v2h-v210.so + ``` +2. **[For RZ/V2H only]** Rename the `Q08_object_counter_deploy_*.so` to `deploy.so`. + ```sh + mv deploy.so + ``` +3. Copy the following files to the `/home/root/tvm` directory of the rootfs (SD Card) for the board. + |File | Details | + |:---|:---| + |All files in `EXE_DIR` directory | Including `deploy.so` file. | + |`object_counter` application file | Generated the file according to [Application File Generation](#application-file-generation) | + +4. Check if `libtvm_runtime.so` exists under `/usr/lib64` directory of the rootfs (SD card) on the board. + +5. Folder structure in the rootfs (SD Card) would look like: + ```sh + ├── usr/ + │ └── lib64/ + │ └── libtvm_runtime.so + └── home/ + └── root/ + └── tvm/ + ├── coco/ + │ ├── tinyyolov3_onnx/ #RZ/V2L only + │ │ ├── deploy.json #RZ/V2L only + │ │ ├── deploy.params #RZ/V2L only + │ │ ├── deploy.so #RZ/V2L only + │ │ └── preprocess/ #RZ/V2L only + │ │ + │ ├── yolov3_onnx/ #RZ/V2H only + │ │ ├── deploy.json #RZ/V2H only + │ │ ├── deploy.params #RZ/V2H only + │ │ └── deploy.so #RZ/V2H only + │ │ + │ ├── coco_class.txt + │ └── config.ini + ├── animal/ + │ ├── animal_onnx/ + │ │ ├── deploy.json + │ │ ├── deploy.params + │ │ └── deploy.so + │ │ + │ ├── animal_class.txt + │ └── config.ini + ├── vehicle/ + │ ├── vehicle_onnx/ + │ │ ├── deploy.json + │ │ ├── deploy.params + │ │ └── deploy.so + │ │ + │ ├── vehicle_class.txt + │ └── config.ini + │ + ├── app_conf.ini + └── object_counter + ``` +>**Note:** The directory name could be anything instead of `tvm`. If you copy the whole `EXE_DIR` folder on the board, you are not required to rename it `tvm`. + +## Application: Run Stage + +### Prerequisites +This section expects the user to have completed Step 7-3 of [Getting Started Guide](https://renesas-rz.github.io/rzv_ai_sdk/latest/getting_started.html#step7-3) provided by Renesas. + +After completion of the guide, the user is expected of following things. +- The board setup is done. +- The board is booted with microSD card, which contains the application file. + +### Instruction +1. On Board terminal, go to the `tvm` directory of the rootfs. + ```sh + cd /home/root/tvm/ + ``` + +2. Change the values in `app_conf.ini` as per the requirements. Detailed explanation of the `app_conf.ini` file is given at [below section](#explanation-of-the-app_confini-file). + + ```sh + vi app_conf.ini + ``` + +3. Run the application. + ```sh + ./object_counter + ``` + - mode options + |Value |Description | + |-------|-------------------------------------| + |COCO | Detects coco objects listed | + |animal | Detects animals listed | + |vehicle| Detects automobiles listed | + + >**Note:** The mode will be the section name in app_conf.ini file. + + - camera options + |Value|Description | + |-----|---------------------------------------| + | | MIPI camera as input **[RZ/V2L only]**| + |USB | USB camera as input | + + For example, to run in "animal" mode with a USB camera, write the following command. + ```sh + ./object_counter animal USB + ``` + +3. Following window shows up on HDMI screen. + + |RZ/V2L EVK (COCO) | RZ/V2H EVK (COCO) | + |:---|:---| + || | + + On application window, following information is displayed. + - Camera capture + - AI result + - Processing time + + +4. To terminate the application, switch the application window to the terminal by using `Super(windows key)+Tab` and press ENTER key on the terminal of the board. + + +## Application: Configuration +### AI Model +#### RZ/V2L EVK +- TinyYOLOv3: [Darknet](https://pjreddie.com/darknet/yolo/) +Dataset: [COCO](https://cocodataset.org/#home) +Input size: 1x3x416x416 +Output1 size: 1x13x13x255 +Output2 size: 1x26x26x255 + +#### RZ/V2H EVK +- YOLOv3: [Darknet](https://pjreddie.com/darknet/yolo/) +Dataset: [COCO](https://cocodataset.org/#home) +Input size: 1x3x416x416 +Output1 size: 1x13x13x255 +Output2 size: 1x26x26x255 +Output3 size: 1x52x52x255 + +### Dataset + +| Model | Dataset | Description | +|---|---|---| +| coco | [Dataset Link](https://cocodataset.org/#download) | Dataset used is the same as mentioned in the research paper | +| animal | [Dataset Link](https://huggingface.co/datasets/myyyyw/NTLNP) | Dataset of wildlife in the mixed coniferous broad-leaved forest | +| vehicle | [Site](https://universe.roboflow.com/) | Combined multiple sources for different classes from the given site. Sources used are listed in below table | + +| Class | Dataset for RZ/V2L EVK | Dataset for RZ/V2H EVK | +|---|---|---| +| motorcycle | [Dataset](https://universe.roboflow.com/vehicle-mscoco/vehicles-coco) | [Dataset](https://universe.roboflow.com/vehicle-mscoco/vehicles-coco) | +| bus | [Dataset 1](https://universe.roboflow.com/titu/bus-jm7t3), [Dataset 2](https://universe.roboflow.com/final-year-project-shhpl/bus-detection-2wlyo), [Dataset 3](https://universe.roboflow.com/fyp-object-detection-tc8af/sya-bus) | [Dataset 1](https://universe.roboflow.com/titu/bus-jm7t3), [Dataset 2](https://universe.roboflow.com/final-year-project-shhpl/bus-detection-2wlyo), [Dataset 3](https://universe.roboflow.com/fyp-object-detection-tc8af/sya-bus)| +| car | [Dataset 1](https://universe.roboflow.com/hungdk-t8jb0/nhandienxeoto-udgcp), [Dataset 2](https://universe.roboflow.com/project-fjp7n/car-detection-vwdhg) | [Dataset 1](https://universe.roboflow.com/hungdk-t8jb0/nhandienxeoto-udgcp), [Dataset 2](https://universe.roboflow.com/project-fjp7n/car-detection-vwdhg) | +| policecar | [Dataset 1](https://universe.roboflow.com/fyp-tc-idn2o/police-cars-sumfm), [Dataset 2](https://universe.roboflow.com/maryam-mahmood-6hoeq/pol-tslhg) | [Dataset 1](https://universe.roboflow.com/fyp-tc-idn2o/police-cars-sumfm), [Dataset 2](https://universe.roboflow.com/maryam-mahmood-6hoeq/pol-tslhg)| +| ambulance | [Dataset 1](https://universe.roboflow.com/ambulance-k0z3x/ambulance-detection-azspv), [Dataset 2](https://universe.roboflow.com/school-87zwx/emegency-vehicle-detection) | [Dataset 1](https://universe.roboflow.com/ambulance-k0z3x/ambulance-detection-azspv), [Dataset 2](https://universe.roboflow.com/school-87zwx/emegency-vehicle-detection)| +| truck | [Dataset 1](https://universe.roboflow.com/project-school-ulsua/truck-detection-g88di), [Dataset 2](https://universe.roboflow.com/kmec/truck-detection-vka5s) |[Dataset 1](https://universe.roboflow.com/project-school-ulsua/truck-detection-g88di), [Dataset 2](https://universe.roboflow.com/kmec/truck-detection-vka5s) | +| bicycle | [Dataset 1](https://universe.roboflow.com/vtc-ywqwf/tt-aio6y), [Dataset 2](https://universe.roboflow.com/north-south-university-faox7/bicycle-bdti6) |[Dataset 1](https://universe.roboflow.com/vtc-ywqwf/tt-aio6y), [Dataset 2](https://universe.roboflow.com/north-south-university-faox7/bicycle-bdti6), [Dataset 3](https://cocodataset.org/#download) | +| bike | [Dataset 1](https://universe.roboflow.com/subham-bhansali-fedah/bike-detection-tzvlj), [Dataset 2](https://universe.roboflow.com/fyp-object-detection-tc8af/sya-bike) | [Dataset 1](https://universe.roboflow.com/subham-bhansali-fedah/bike-detection-tzvlj), [Dataset 2](https://universe.roboflow.com/fyp-object-detection-tc8af/sya-bike)| +| Auto | [Dataset 1](https://universe.roboflow.com/rutviknirma/smart-traffic-management-system), [Dataset 2](https://universe.roboflow.com/graduation-project-rtgrc/tuk-tuk-labelling) | [Dataset 1](https://universe.roboflow.com/rutviknirma/smart-traffic-management-system), [Dataset 2](https://universe.roboflow.com/graduation-project-rtgrc/tuk-tuk-labelling) | +| LCV | [Dataset 1](https://universe.roboflow.com/project-final-ltl6m/vehicle-detection-inlat), [Dataset 2](https://universe.roboflow.com/pooja-shri-v/lcvs-zqmsu) | [Dataset 1](https://universe.roboflow.com/project-final-ltl6m/vehicle-detection-inlat), [Dataset 2](https://universe.roboflow.com/pooja-shri-v/lcvs-zqmsu) | +| Fire engine | [Dataset 1](https://universe.roboflow.com/grad-project-tjt2u/fire-truck-xumw3), [Dataset 2](https://universe.roboflow.com/pouria-maleki/firetruck) | [Dataset 1](https://universe.roboflow.com/grad-project-tjt2u/fire-truck-xumw3), [Dataset 2](https://universe.roboflow.com/pouria-maleki/firetruck) | + +>Note: Link for motorcycle dateset has additional classes bus, car and truck which is also used for training + + +### AI inference time +|Board | AI model | AI inference time| +|:---|:---|:---| +|RZ/V2L EVK|Tiny YOLOv3| Approximately 170ms | +|RZ/V2H EVK |YOLOv3 | Approximately 40ms | + +### Processing + +|Processing | RZ/V2L EVK | RZ/V2H EVK | +|:---|:---|:---| +|Pre-processing | Processed by DRP-AI.
| Processed by CPU.
| +|Inference | Processed by DRP-AI and CPU. | Processed by DRP-AI and CPU. | +|Post-processing | Processed by CPU. |Processed by CPU. | + + +### Image buffer size + +|Board | Camera capture buffer size|HDMI output buffer size| +|:---|:---|:---| +|RZ/V2L EVK| VGA (640x480) in YUYV format | FHD (1920x1080) in BGRA format | +|RZ/V2H EVK | VGA (640x480) in YUYV format | FHD (1920x1080) in BGRA format | + +### Explanation of the `app_conf.ini` file + +- The section name can be of your choice. And it will be the mode name. + +- The section should contains three variables - `model_path`, `label_path` & `config_path`. + + - The `model_path` value is the path to the folder containing compiled model. The folder should also contains also contain preprocess folder. + + - The `label_path` value is the path to the label list the model supports. + + - The `config_path` value is the path to the model configuration ini file. Detailed explanation of the `config.ini` file is given at below section. + +### Explanation of the `config.ini` file + +- The [**detect**] section contains three variables - `conf`, `anchors` & `objects`. + + - The `conf` value is the confidence threshold used for object detection, + - The `anchors` are the yolo anchors for the object detection. + - The `objects` represents class to be identified and it can be changed to other classes present on the class label list. + +- To modify the configuration settings, edit the values in this file using VI Editor. + +```sh +vi config.ini +``` + +## Reference +- For RZ/V2H EVK, this application supports USB camera only with 640x480 resolution. +FHD resolution is supported by e-CAM22_CURZH camera (MIPI). +Please refer to following URL for how to change camera input to MIPI camera. +[https://renesas-rz.github.io/rzv_ai_sdk/latest/about-applications](https://renesas-rz.github.io/rzv_ai_sdk/latest/about-applications#mipi). diff --git a/Q08_object_counter/exe_v2h/animal/animal_class.txt b/Q08_object_counter/exe_v2h/animal/animal_class.txt new file mode 100755 index 0000000..9980a29 --- /dev/null +++ b/Q08_object_counter/exe_v2h/animal/animal_class.txt @@ -0,0 +1,13 @@ +boar +bear +bird +cat +cow +dog +horse +sheep +rabbit +raccoon +monkey +fox +deer diff --git a/Q08_object_counter/exe_v2h/animal/animal_onnx/deploy.json b/Q08_object_counter/exe_v2h/animal/animal_onnx/deploy.json new file mode 100755 index 0000000..2c13800 --- /dev/null +++ b/Q08_object_counter/exe_v2h/animal/animal_onnx/deploy.json @@ -0,0 +1,76 @@ +{ + "nodes": [ + { + "op": "null", + "name": "input1", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "hash": "e050f4a9fc692759", + "num_outputs": "3", + "num_inputs": "1", + "global_symbol": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp" + }, + "inputs": [ + [ + 0, + 0, + 0 + ] + ] + } + ], + "arg_nodes": [0], + "heads": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 1, + 2, + 0 + ] + ], + "attrs": { + "dltype": [ + "list_str", + [ + "float32", + "float16", + "float16", + "float16" + ] + ], + "device_index": [ + "list_int", + [1, 1, 1, 1] + ], + "storage_id": [ + "list_int", + [0, 1, 2, 3] + ], + "shape": [ + "list_shape", + [ + [1, 3, 416, 416], + [1, 54, 13, 13], + [1, 54, 26, 26], + [1, 54, 52, 52] + ] + ] + }, + "node_row_ptr": [0, 1, 4] +} \ No newline at end of file diff --git a/Q08_object_counter/exe_v2h/animal/animal_onnx/deploy.params b/Q08_object_counter/exe_v2h/animal/animal_onnx/deploy.params new file mode 100755 index 0000000..1011def Binary files /dev/null and b/Q08_object_counter/exe_v2h/animal/animal_onnx/deploy.params differ diff --git a/Q08_object_counter/exe_v2h/animal/config.ini b/Q08_object_counter/exe_v2h/animal/config.ini new file mode 100755 index 0000000..a42bdd0 --- /dev/null +++ b/Q08_object_counter/exe_v2h/animal/config.ini @@ -0,0 +1,6 @@ +; Configuration File + +[detect] +conf=0.1; +anchors=10,13,16,30,33,23,30,61,62,45,59,119,116,90,156,198,373,326; +objects=bear,boar,bird,cat,cow,deer,dog,fox,horse,rabbit,raccoon,monkey,sheep; diff --git a/Q08_object_counter/exe_v2h/app_conf.ini b/Q08_object_counter/exe_v2h/app_conf.ini new file mode 100755 index 0000000..94cc1a5 --- /dev/null +++ b/Q08_object_counter/exe_v2h/app_conf.ini @@ -0,0 +1,19 @@ +; Configuration File + +[COCO] + +model_path=./coco/yolov3_onnx; +label_path=./coco/coco_class.txt; +config_path=./coco/config.ini; + +[animal] + +model_path=./animal/animal_onnx; +label_path=./animal/animal_class.txt; +config_path=./animal/config.ini; + +[vehicle] + +model_path=./vehicle/vehicle_onnx; +config_path=./vehicle/config.ini; +label_path=./vehicle/vehicle_class.txt; \ No newline at end of file diff --git a/Q08_object_counter/exe_v2h/coco/coco_class.txt b/Q08_object_counter/exe_v2h/coco/coco_class.txt new file mode 100755 index 0000000..1f42c8e --- /dev/null +++ b/Q08_object_counter/exe_v2h/coco/coco_class.txt @@ -0,0 +1,80 @@ +person +bicycle +car +motorcycle +airplane +bus +train +truck +boat +traffic light +fire hydrant +stop sign +parking meter +bench +bird +cat +dog +horse +sheep +cow +elephant +bear +zebra +giraffe +backpack +umbrella +handbag +tie +suitcase +frisbee +skis +snowboard +sports ball +kite +baseball bat +baseball glove +skateboard +surfboard +tennis racket +bottle +wine glass +cup +fork +knife +spoon +bowl +banana +apple +sandwich +orange +broccoli +carrot +hot dog +pizza +donut +cake +chair +couch +potted plant +bed +dining table +toilet +tv +laptop +mouse +remote +keyboard +cell phone +microwave +oven +toaster +sink +refrigerator +book +clock +vase +scissors +teddy bear +hair drier +toothbrush \ No newline at end of file diff --git a/Q08_object_counter/exe_v2h/coco/config.ini b/Q08_object_counter/exe_v2h/coco/config.ini new file mode 100755 index 0000000..f520276 --- /dev/null +++ b/Q08_object_counter/exe_v2h/coco/config.ini @@ -0,0 +1,8 @@ +; Configuration File + + +[detect] + +conf=0.5; +anchors=10,13,16,30,33,23,30,61,62,45,59,119,116,90,156,198,373,326; +objects=person,car,backpack,umbrella,fork,kite,tie,tennis racket,person,dog,laptop,mouse,remote,keyboard,cell phone; diff --git a/Q08_object_counter/exe_v2h/coco/yolov3_onnx/deploy.json b/Q08_object_counter/exe_v2h/coco/yolov3_onnx/deploy.json new file mode 100755 index 0000000..62b6854 --- /dev/null +++ b/Q08_object_counter/exe_v2h/coco/yolov3_onnx/deploy.json @@ -0,0 +1,76 @@ +{ + "nodes": [ + { + "op": "null", + "name": "input1", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "hash": "036f4701453dc3f3", + "num_outputs": "3", + "num_inputs": "1", + "global_symbol": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp" + }, + "inputs": [ + [ + 0, + 0, + 0 + ] + ] + } + ], + "arg_nodes": [0], + "heads": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 1, + 2, + 0 + ] + ], + "attrs": { + "dltype": [ + "list_str", + [ + "float32", + "float16", + "float16", + "float16" + ] + ], + "device_index": [ + "list_int", + [1, 1, 1, 1] + ], + "storage_id": [ + "list_int", + [0, 1, 2, 3] + ], + "shape": [ + "list_shape", + [ + [1, 3, 416, 416], + [1, 255, 13, 13], + [1, 255, 26, 26], + [1, 255, 52, 52] + ] + ] + }, + "node_row_ptr": [0, 1, 4] +} \ No newline at end of file diff --git a/Q08_object_counter/exe_v2h/coco/yolov3_onnx/deploy.params b/Q08_object_counter/exe_v2h/coco/yolov3_onnx/deploy.params new file mode 100755 index 0000000..1011def Binary files /dev/null and b/Q08_object_counter/exe_v2h/coco/yolov3_onnx/deploy.params differ diff --git a/Q08_object_counter/exe_v2h/object_counter b/Q08_object_counter/exe_v2h/object_counter new file mode 100755 index 0000000..dd39d1f Binary files /dev/null and b/Q08_object_counter/exe_v2h/object_counter differ diff --git a/Q08_object_counter/exe_v2h/vehicle/config.ini b/Q08_object_counter/exe_v2h/vehicle/config.ini new file mode 100755 index 0000000..6a142fa --- /dev/null +++ b/Q08_object_counter/exe_v2h/vehicle/config.ini @@ -0,0 +1,6 @@ +; Configuration File +[detect] + +conf=0.5; +anchors=10,13,16,30,33,23,30,61,62,45,59,119,116,90,156,198,373,326; +objects=bus,car,motorcycle,truck,ambulance,Fire truck,LCV,Policecar,bicycle,automobile; diff --git a/Q08_object_counter/exe_v2h/vehicle/vehicle_class.txt b/Q08_object_counter/exe_v2h/vehicle/vehicle_class.txt new file mode 100755 index 0000000..151936e --- /dev/null +++ b/Q08_object_counter/exe_v2h/vehicle/vehicle_class.txt @@ -0,0 +1,10 @@ +bus +car +motorcycle +truck +ambulance +Fire truck +LCV +Policecar +bicycle +automobile diff --git a/Q08_object_counter/exe_v2h/vehicle/vehicle_onnx/deploy.json b/Q08_object_counter/exe_v2h/vehicle/vehicle_onnx/deploy.json new file mode 100755 index 0000000..bbff683 --- /dev/null +++ b/Q08_object_counter/exe_v2h/vehicle/vehicle_onnx/deploy.json @@ -0,0 +1,76 @@ +{ + "nodes": [ + { + "op": "null", + "name": "input1", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "hash": "b4b8d690567be805", + "num_outputs": "3", + "num_inputs": "1", + "global_symbol": "tvmgen_default_tvmgen_default_tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp" + }, + "inputs": [ + [ + 0, + 0, + 0 + ] + ] + } + ], + "arg_nodes": [0], + "heads": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 1, + 2, + 0 + ] + ], + "attrs": { + "dltype": [ + "list_str", + [ + "float32", + "float16", + "float16", + "float16" + ] + ], + "device_index": [ + "list_int", + [1, 1, 1, 1] + ], + "storage_id": [ + "list_int", + [0, 1, 2, 3] + ], + "shape": [ + "list_shape", + [ + [1, 3, 416, 416], + [1, 45, 13, 13], + [1, 45, 26, 26], + [1, 45, 52, 52] + ] + ] + }, + "node_row_ptr": [0, 1, 4] +} \ No newline at end of file diff --git a/Q08_object_counter/exe_v2h/vehicle/vehicle_onnx/deploy.params b/Q08_object_counter/exe_v2h/vehicle/vehicle_onnx/deploy.params new file mode 100755 index 0000000..1011def Binary files /dev/null and b/Q08_object_counter/exe_v2h/vehicle/vehicle_onnx/deploy.params differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_class.txt b/Q08_object_counter/exe_v2l/animal/animal_class.txt new file mode 100644 index 0000000..9980a29 --- /dev/null +++ b/Q08_object_counter/exe_v2l/animal/animal_class.txt @@ -0,0 +1,13 @@ +boar +bear +bird +cat +cow +dog +horse +sheep +rabbit +raccoon +monkey +fox +deer diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.json b/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.json new file mode 100644 index 0000000..ca23de3 --- /dev/null +++ b/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.json @@ -0,0 +1,273 @@ +{ + "nodes": [ + { + "op": "null", + "name": "input1", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "1", + "global_symbol": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp", + "hash": "e79342ded48b2a85" + }, + "inputs": [ + [ + 0, + 0, + 0 + ] + ] + }, + { + "op": "null", + "name": "p26", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_nn_pad", + "attrs": { + "num_outputs": "1", + "num_inputs": "3", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_nn_pad", + "hash": "7aa0da36fbb870d2" + }, + "inputs": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 2, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_0", + "layout": "NCHW", + "hash": "6ab69954d7af9960", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "1", + "global_symbol": "tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp" + }, + "inputs": [ + [ + 3, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_concatenate", + "attrs": { + "num_outputs": "1", + "num_inputs": "2", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_concatenate", + "hash": "4969df77426d290f" + }, + "inputs": [ + [ + 4, + 0, + 0 + ], + [ + 4, + 1, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_10", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_10", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "1", + "num_inputs": "1", + "global_symbol": "tvmgen_default_mera_drp_main_10", + "Compiler": "mera_drp", + "hash": "43541740dc5cf5f3" + }, + "inputs": [ + [ + 5, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_concatenate_1", + "attrs": { + "num_outputs": "1", + "num_inputs": "3", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_concatenate_1", + "hash": "2cafd345ebba88e0" + }, + "inputs": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 6, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_9", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_9", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "2", + "global_symbol": "tvmgen_default_mera_drp_main_9", + "Compiler": "mera_drp", + "hash": "92411cd379fdb381" + }, + "inputs": [ + [ + 4, + 0, + 0 + ], + [ + 7, + 0, + 0 + ] + ] + } + ], + "arg_nodes": [0, 2], + "heads": [ + [ + 8, + 0, + 0 + ], + [ + 8, + 1, + 0 + ] + ], + "attrs": { + "dltype": [ + "list_str", + [ + "float32", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16" + ] + ], + "device_index": [ + "list_int", + [ + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1 + ] + ], + "storage_id": [ + "list_int", + [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 4, + 6, + 4, + 1, + 6 + ] + ], + "shape": [ + "list_shape", + [ + [1, 3, 416, 416], + [1, 512, 13, 13], + [1, 256, 26, 26], + [], + [1, 512, 14, 14], + [1, 54, 13, 13], + [1, 256, 13, 13], + [1, 256, 13, 13], + [1, 128, 26, 26], + [1, 384, 26, 26], + [1, 54, 13, 13], + [1, 54, 26, 26] + ] + ] + }, + "node_row_ptr": [0, 1, 3, 4, 5, 7, 8, 9, 10, 12] +} \ No newline at end of file diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.params b/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.params new file mode 100644 index 0000000..095989c Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.params differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.so b/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.so new file mode 100644 index 0000000..fdc81f8 Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/deploy.so differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/aimac_desc.bin b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/aimac_desc.bin new file mode 100644 index 0000000..6a64c90 Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/aimac_desc.bin differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_desc.bin b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_desc.bin new file mode 100644 index 0000000..6575127 Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_desc.bin differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_param.bin b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_param.bin new file mode 100644 index 0000000..d6a74d0 Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_param.bin differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_param_info.txt b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_param_info.txt new file mode 100644 index 0000000..bade26b --- /dev/null +++ b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/drp_param_info.txt @@ -0,0 +1,137 @@ +OFFSET_ADD:0,layer_name:post_0_0_conv_yuv2rgb,drp_lib:conv_yuv2rgb,prop:input,node:in, + Param:raddr, Value:0, offset:0, size:4, + Param:waddr, Value:614400, offset:4, size:4, + Param:IMG_IWIDHT, Value:640, offset:8, size:2, + Param:IMG_IHEIGHT, Value:480, offset:10,size:2, + Param:RFU1, Value:0, offset:12,size:2, + Param:RFU2, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:640, offset:16,size:2, + Param:IMG_OHEIGHT, Value:480, offset:18,size:2, + Param:RFU3, Value:0, offset:20,size:2, + Param:RFU4, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU5, Value:0, offset:42,size:2, + Param:RFU6, Value:0, offset:44,size:2, + Param:RFU7, Value:0, offset:46,size:2, + Param:INPUT_YUV_FORMAT, Value:0, offset:48,size:2, + Param:RFU8, Value:0, offset:50,size:2, + Param:DOUT_RGB_FORMAT, Value:0, offset:52,size:2, + Param:RFU10, Value:0, offset:54,size:2, +OFFSET_ADD:56,layer_name:post_0_1_resize_hwc,drp_lib:resize_hwc,prop:intermediate_data,node:, + Param:raddr, Value:614400, offset:0, size:4, + Param:waddr, Value:1536000, offset:4, size:4, + Param:IMG_IWIDHT, Value:640, offset:8, size:2, + Param:IMG_IHEIGHT, Value:480, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:RFU1, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:RFU2, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU3, Value:0, offset:42,size:2, + Param:RFU4, Value:0, offset:44,size:2, + Param:RFU5, Value:0, offset:46,size:2, + Param:RESIZE_ALG, Value:1, offset:48,size:2, + Param:DATA_TYPE, Value:0, offset:50,size:2, + Param:RFU7, Value:0, offset:52,size:2, + Param:RFU8, Value:0, offset:54,size:2, +OFFSET_ADD:112,layer_name:post_0_2_imagescaler,drp_lib:imagescaler,prop:intermediate_data,node:, + Param:raddr, Value:1536000, offset:0, size:4, + Param:waddr, Value:2055168, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:IMG_ICH_ORG, Value:3, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:IMG_OCH_ORG, Value:3, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU3, Value:0, offset:42,size:2, + Param:RFU4, Value:0, offset:44,size:2, + Param:RFU5, Value:0, offset:46,size:2, + Param:ADD_ADDR, Value:6208512, offset:48,size:4, + Param:MUL_ADDR, Value:6208520, offset:52,size:4, + Param:DIN_FORMAT, Value:0, offset:56,size:2, + Param:DOUT_RGB_ORDER, Value:0, offset:58,size:2, + Param:RFU6, Value:0, offset:60,size:2, + Param:RFU7, Value:0, offset:62,size:2, +OFFSET_ADD:176,layer_name:post_0_4_transpose,drp_lib:transpose,prop:intermediate_data,node:, + Param:raddr, Value:2055168, offset:0, size:4, + Param:waddr, Value:3093504, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:IMG_ICH_ORG, Value:3, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:IMG_OCH_ORG, Value:3, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU1, Value:0, offset:42,size:2, + Param:RFU2, Value:0, offset:44,size:2, + Param:RFU3, Value:0, offset:46,size:2, + Param:WORD_SIZE, Value:1, offset:48,size:2, + Param:IS_CHW2HWC, Value:0, offset:50,size:2, + Param:RFU4, Value:0, offset:52,size:2, + Param:RFU5, Value:0, offset:54,size:2, +OFFSET_ADD:232,layer_name:post_0_5_cast_fp16_fp32,drp_lib:cast_fp16_fp32,prop:output,node:out, + Param:raddr, Value:3093504, offset:0, size:4, + Param:waddr, Value:4131840, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:RFU1, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:RFU2, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU1, Value:0, offset:42,size:2, + Param:RFU2, Value:0, offset:44,size:2, + Param:RFU3, Value:0, offset:46,size:2, + Param:CAST_MODE, Value:0, offset:48,size:2, + Param:RFU1, Value:0, offset:50,size:2, + Param:RFU2, Value:0, offset:52,size:2, + Param:RFU3, Value:0, offset:54,size:2, diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_addrmap_intm.txt b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_addrmap_intm.txt new file mode 100644 index 0000000..4883d37 --- /dev/null +++ b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_addrmap_intm.txt @@ -0,0 +1,9 @@ +data_in 0 96000 +data 96000 35ac00 +data_out 3f0c00 1fb000 +work 5ebc00 0 +weight 5ebc00 e +drp_config 5ebc40 1869a0 +drp_param 772600 120 +desc_aimac 772740 f0 +desc_drp 772840 1a0 diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_drpcfg.mem b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_drpcfg.mem new file mode 100644 index 0000000..f4fb291 Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_drpcfg.mem differ diff --git a/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_weight.dat b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_weight.dat new file mode 100644 index 0000000..9455fb8 Binary files /dev/null and b/Q08_object_counter/exe_v2l/animal/animal_onnx/preprocess/pp_weight.dat differ diff --git a/Q08_object_counter/exe_v2l/animal/config.ini b/Q08_object_counter/exe_v2l/animal/config.ini new file mode 100644 index 0000000..199ba44 --- /dev/null +++ b/Q08_object_counter/exe_v2l/animal/config.ini @@ -0,0 +1,8 @@ +; Configuration File + + +[detect] + +conf=0.5; +anchors=14,17,46,60,93,130,213,169,142,265,294,326; +objects=bear,boar,bird,cat,cow,deer,dog,fox,horse,rabbit,racoon,monkey,sheep; diff --git a/Q08_object_counter/exe_v2l/app_conf.ini b/Q08_object_counter/exe_v2l/app_conf.ini new file mode 100644 index 0000000..5379950 --- /dev/null +++ b/Q08_object_counter/exe_v2l/app_conf.ini @@ -0,0 +1,19 @@ +; Configuration File + +[COCO] + +model_path=./coco/tinyyolov3_onnx; +label_path=./coco/coco_class.txt; +config_path=./coco/config.ini; + +[animal] + +model_path=./animal/animal_onnx; +label_path=./animal/animal_class.txt; +config_path=./animal/config.ini; + +[vehicle] + +model_path=./vehicle/vehicle_onnx; +label_path=./vehicle/vehicle_class.txt; +config_path=./vehicle/config.ini; \ No newline at end of file diff --git a/Q08_object_counter/exe_v2l/coco/coco_class.txt b/Q08_object_counter/exe_v2l/coco/coco_class.txt new file mode 100644 index 0000000..1f42c8e --- /dev/null +++ b/Q08_object_counter/exe_v2l/coco/coco_class.txt @@ -0,0 +1,80 @@ +person +bicycle +car +motorcycle +airplane +bus +train +truck +boat +traffic light +fire hydrant +stop sign +parking meter +bench +bird +cat +dog +horse +sheep +cow +elephant +bear +zebra +giraffe +backpack +umbrella +handbag +tie +suitcase +frisbee +skis +snowboard +sports ball +kite +baseball bat +baseball glove +skateboard +surfboard +tennis racket +bottle +wine glass +cup +fork +knife +spoon +bowl +banana +apple +sandwich +orange +broccoli +carrot +hot dog +pizza +donut +cake +chair +couch +potted plant +bed +dining table +toilet +tv +laptop +mouse +remote +keyboard +cell phone +microwave +oven +toaster +sink +refrigerator +book +clock +vase +scissors +teddy bear +hair drier +toothbrush \ No newline at end of file diff --git a/Q08_object_counter/exe_v2l/coco/config.ini b/Q08_object_counter/exe_v2l/coco/config.ini new file mode 100644 index 0000000..dc773b1 --- /dev/null +++ b/Q08_object_counter/exe_v2l/coco/config.ini @@ -0,0 +1,8 @@ +; Configuration File + + +[detect] + +conf=0.1; +anchors=10,14,23,27,37,58,81,82,135,169,344,319; +objects=backpack,umbrella,fork,kite,tie,tennis racket,person,dog; diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.json b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.json new file mode 100644 index 0000000..7086859 --- /dev/null +++ b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.json @@ -0,0 +1,273 @@ +{ + "nodes": [ + { + "op": "null", + "name": "input1", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "1", + "global_symbol": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp", + "hash": "78397ceef949a69d" + }, + "inputs": [ + [ + 0, + 0, + 0 + ] + ] + }, + { + "op": "null", + "name": "p26", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_nn_pad", + "attrs": { + "num_outputs": "1", + "num_inputs": "3", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_nn_pad", + "hash": "7aa0da36fbb870d2" + }, + "inputs": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 2, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_0", + "layout": "NCHW", + "hash": "1131936441a198e1", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "1", + "global_symbol": "tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp" + }, + "inputs": [ + [ + 3, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_concatenate", + "attrs": { + "num_outputs": "1", + "num_inputs": "2", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_concatenate", + "hash": "a049934a28c8210c" + }, + "inputs": [ + [ + 4, + 0, + 0 + ], + [ + 4, + 1, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_10", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_10", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "1", + "num_inputs": "1", + "global_symbol": "tvmgen_default_mera_drp_main_10", + "Compiler": "mera_drp", + "hash": "5fab61a5913489de" + }, + "inputs": [ + [ + 5, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_concatenate_1", + "attrs": { + "num_outputs": "1", + "num_inputs": "3", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_concatenate_1", + "hash": "2cafd345ebba88e0" + }, + "inputs": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 6, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_9", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_9", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "2", + "global_symbol": "tvmgen_default_mera_drp_main_9", + "Compiler": "mera_drp", + "hash": "432914bdc7064d9c" + }, + "inputs": [ + [ + 4, + 0, + 0 + ], + [ + 7, + 0, + 0 + ] + ] + } + ], + "arg_nodes": [0, 2], + "heads": [ + [ + 8, + 0, + 0 + ], + [ + 8, + 1, + 0 + ] + ], + "attrs": { + "dltype": [ + "list_str", + [ + "float32", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16" + ] + ], + "device_index": [ + "list_int", + [ + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1 + ] + ], + "storage_id": [ + "list_int", + [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 4, + 6, + 4, + 1, + 2 + ] + ], + "shape": [ + "list_shape", + [ + [1, 3, 416, 416], + [1, 512, 13, 13], + [1, 256, 26, 26], + [], + [1, 512, 14, 14], + [1, 255, 13, 13], + [1, 256, 13, 13], + [1, 256, 13, 13], + [1, 128, 26, 26], + [1, 384, 26, 26], + [1, 255, 13, 13], + [1, 255, 26, 26] + ] + ] + }, + "node_row_ptr": [0, 1, 3, 4, 5, 7, 8, 9, 10, 12] +} \ No newline at end of file diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.params b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.params new file mode 100644 index 0000000..095989c Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.params differ diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.so b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.so new file mode 100644 index 0000000..85a1bb6 Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/deploy.so differ diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/aimac_desc.bin b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/aimac_desc.bin new file mode 100644 index 0000000..6a64c90 Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/aimac_desc.bin differ diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_desc.bin b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_desc.bin new file mode 100644 index 0000000..6575127 Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_desc.bin differ diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_param.bin b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_param.bin new file mode 100644 index 0000000..d6a74d0 Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_param.bin differ diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_param_info.txt b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_param_info.txt new file mode 100644 index 0000000..bade26b --- /dev/null +++ b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/drp_param_info.txt @@ -0,0 +1,137 @@ +OFFSET_ADD:0,layer_name:post_0_0_conv_yuv2rgb,drp_lib:conv_yuv2rgb,prop:input,node:in, + Param:raddr, Value:0, offset:0, size:4, + Param:waddr, Value:614400, offset:4, size:4, + Param:IMG_IWIDHT, Value:640, offset:8, size:2, + Param:IMG_IHEIGHT, Value:480, offset:10,size:2, + Param:RFU1, Value:0, offset:12,size:2, + Param:RFU2, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:640, offset:16,size:2, + Param:IMG_OHEIGHT, Value:480, offset:18,size:2, + Param:RFU3, Value:0, offset:20,size:2, + Param:RFU4, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU5, Value:0, offset:42,size:2, + Param:RFU6, Value:0, offset:44,size:2, + Param:RFU7, Value:0, offset:46,size:2, + Param:INPUT_YUV_FORMAT, Value:0, offset:48,size:2, + Param:RFU8, Value:0, offset:50,size:2, + Param:DOUT_RGB_FORMAT, Value:0, offset:52,size:2, + Param:RFU10, Value:0, offset:54,size:2, +OFFSET_ADD:56,layer_name:post_0_1_resize_hwc,drp_lib:resize_hwc,prop:intermediate_data,node:, + Param:raddr, Value:614400, offset:0, size:4, + Param:waddr, Value:1536000, offset:4, size:4, + Param:IMG_IWIDHT, Value:640, offset:8, size:2, + Param:IMG_IHEIGHT, Value:480, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:RFU1, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:RFU2, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU3, Value:0, offset:42,size:2, + Param:RFU4, Value:0, offset:44,size:2, + Param:RFU5, Value:0, offset:46,size:2, + Param:RESIZE_ALG, Value:1, offset:48,size:2, + Param:DATA_TYPE, Value:0, offset:50,size:2, + Param:RFU7, Value:0, offset:52,size:2, + Param:RFU8, Value:0, offset:54,size:2, +OFFSET_ADD:112,layer_name:post_0_2_imagescaler,drp_lib:imagescaler,prop:intermediate_data,node:, + Param:raddr, Value:1536000, offset:0, size:4, + Param:waddr, Value:2055168, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:IMG_ICH_ORG, Value:3, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:IMG_OCH_ORG, Value:3, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU3, Value:0, offset:42,size:2, + Param:RFU4, Value:0, offset:44,size:2, + Param:RFU5, Value:0, offset:46,size:2, + Param:ADD_ADDR, Value:6208512, offset:48,size:4, + Param:MUL_ADDR, Value:6208520, offset:52,size:4, + Param:DIN_FORMAT, Value:0, offset:56,size:2, + Param:DOUT_RGB_ORDER, Value:0, offset:58,size:2, + Param:RFU6, Value:0, offset:60,size:2, + Param:RFU7, Value:0, offset:62,size:2, +OFFSET_ADD:176,layer_name:post_0_4_transpose,drp_lib:transpose,prop:intermediate_data,node:, + Param:raddr, Value:2055168, offset:0, size:4, + Param:waddr, Value:3093504, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:IMG_ICH_ORG, Value:3, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:IMG_OCH_ORG, Value:3, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU1, Value:0, offset:42,size:2, + Param:RFU2, Value:0, offset:44,size:2, + Param:RFU3, Value:0, offset:46,size:2, + Param:WORD_SIZE, Value:1, offset:48,size:2, + Param:IS_CHW2HWC, Value:0, offset:50,size:2, + Param:RFU4, Value:0, offset:52,size:2, + Param:RFU5, Value:0, offset:54,size:2, +OFFSET_ADD:232,layer_name:post_0_5_cast_fp16_fp32,drp_lib:cast_fp16_fp32,prop:output,node:out, + Param:raddr, Value:3093504, offset:0, size:4, + Param:waddr, Value:4131840, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:RFU1, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:RFU2, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU1, Value:0, offset:42,size:2, + Param:RFU2, Value:0, offset:44,size:2, + Param:RFU3, Value:0, offset:46,size:2, + Param:CAST_MODE, Value:0, offset:48,size:2, + Param:RFU1, Value:0, offset:50,size:2, + Param:RFU2, Value:0, offset:52,size:2, + Param:RFU3, Value:0, offset:54,size:2, diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_addrmap_intm.txt b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_addrmap_intm.txt new file mode 100644 index 0000000..4883d37 --- /dev/null +++ b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_addrmap_intm.txt @@ -0,0 +1,9 @@ +data_in 0 96000 +data 96000 35ac00 +data_out 3f0c00 1fb000 +work 5ebc00 0 +weight 5ebc00 e +drp_config 5ebc40 1869a0 +drp_param 772600 120 +desc_aimac 772740 f0 +desc_drp 772840 1a0 diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_drpcfg.mem b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_drpcfg.mem new file mode 100644 index 0000000..f4fb291 Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_drpcfg.mem differ diff --git a/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_weight.dat b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_weight.dat new file mode 100644 index 0000000..9455fb8 Binary files /dev/null and b/Q08_object_counter/exe_v2l/coco/tinyyolov3_onnx/preprocess/pp_weight.dat differ diff --git a/Q08_object_counter/exe_v2l/object_counter b/Q08_object_counter/exe_v2l/object_counter new file mode 100644 index 0000000..d39a52d Binary files /dev/null and b/Q08_object_counter/exe_v2l/object_counter differ diff --git a/Q08_object_counter/exe_v2l/vehicle/config.ini b/Q08_object_counter/exe_v2l/vehicle/config.ini new file mode 100644 index 0000000..60a6d0e --- /dev/null +++ b/Q08_object_counter/exe_v2l/vehicle/config.ini @@ -0,0 +1,8 @@ +; Configuration File + + +[detect] + +conf=0.5; +anchors=14,17,46,60,93,130,213,169,142,265,294,326; +objects=bus,car,motorcycle,truck,ambulance,Fire truck,LCV,Policecar,bicycle,automobile; diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_class.txt b/Q08_object_counter/exe_v2l/vehicle/vehicle_class.txt new file mode 100644 index 0000000..151936e --- /dev/null +++ b/Q08_object_counter/exe_v2l/vehicle/vehicle_class.txt @@ -0,0 +1,10 @@ +bus +car +motorcycle +truck +ambulance +Fire truck +LCV +Policecar +bicycle +automobile diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.json b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.json new file mode 100644 index 0000000..2345109 --- /dev/null +++ b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.json @@ -0,0 +1,273 @@ +{ + "nodes": [ + { + "op": "null", + "name": "input1", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "1", + "global_symbol": "tvmgen_default_tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp", + "hash": "b8f856d2a8ffc66d" + }, + "inputs": [ + [ + 0, + 0, + 0 + ] + ] + }, + { + "op": "null", + "name": "p26", + "inputs": [] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_nn_pad", + "attrs": { + "num_outputs": "1", + "num_inputs": "3", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_nn_pad", + "hash": "7aa0da36fbb870d2" + }, + "inputs": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 2, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_0", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_0", + "layout": "NCHW", + "hash": "c1379755750a491c", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "1", + "global_symbol": "tvmgen_default_mera_drp_main_0", + "Compiler": "mera_drp" + }, + "inputs": [ + [ + 3, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_concatenate", + "attrs": { + "num_outputs": "1", + "num_inputs": "2", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_concatenate", + "hash": "626e54466454ec9a" + }, + "inputs": [ + [ + 4, + 0, + 0 + ], + [ + 4, + 1, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_10", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_10", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "1", + "num_inputs": "1", + "global_symbol": "tvmgen_default_mera_drp_main_10", + "Compiler": "mera_drp", + "hash": "9bc961f5b73adbf2" + }, + "inputs": [ + [ + 5, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_fused_concatenate_1", + "attrs": { + "num_outputs": "1", + "num_inputs": "3", + "flatten_data": "0", + "func_name": "tvmgen_default_fused_concatenate_1", + "hash": "2cafd345ebba88e0" + }, + "inputs": [ + [ + 1, + 0, + 0 + ], + [ + 1, + 1, + 0 + ], + [ + 6, + 0, + 0 + ] + ] + }, + { + "op": "tvm_op", + "name": "tvmgen_default_mera_drp_main_9", + "attrs": { + "flatten_data": "0", + "func_name": "tvmgen_default_mera_drp_main_9", + "kernel_layout": "OIHW", + "data_layout": "NCHW", + "out_layout": "", + "num_outputs": "2", + "num_inputs": "2", + "global_symbol": "tvmgen_default_mera_drp_main_9", + "Compiler": "mera_drp", + "hash": "21c07af3e27f4192" + }, + "inputs": [ + [ + 4, + 0, + 0 + ], + [ + 7, + 0, + 0 + ] + ] + } + ], + "arg_nodes": [0, 2], + "heads": [ + [ + 8, + 0, + 0 + ], + [ + 8, + 1, + 0 + ] + ], + "attrs": { + "dltype": [ + "list_str", + [ + "float32", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16", + "float16" + ] + ], + "device_index": [ + "list_int", + [ + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1 + ] + ], + "storage_id": [ + "list_int", + [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 4, + 6, + 4, + 1, + 6 + ] + ], + "shape": [ + "list_shape", + [ + [1, 3, 416, 416], + [1, 512, 13, 13], + [1, 256, 26, 26], + [], + [1, 512, 14, 14], + [1, 45, 13, 13], + [1, 256, 13, 13], + [1, 256, 13, 13], + [1, 128, 26, 26], + [1, 384, 26, 26], + [1, 45, 13, 13], + [1, 45, 26, 26] + ] + ] + }, + "node_row_ptr": [0, 1, 3, 4, 5, 7, 8, 9, 10, 12] +} \ No newline at end of file diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.params b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.params new file mode 100644 index 0000000..095989c Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.params differ diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.so b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.so new file mode 100644 index 0000000..a0bc258 Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/deploy.so differ diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/aimac_desc.bin b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/aimac_desc.bin new file mode 100644 index 0000000..6a64c90 Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/aimac_desc.bin differ diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_desc.bin b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_desc.bin new file mode 100644 index 0000000..6575127 Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_desc.bin differ diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_param.bin b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_param.bin new file mode 100644 index 0000000..d6a74d0 Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_param.bin differ diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_param_info.txt b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_param_info.txt new file mode 100644 index 0000000..bade26b --- /dev/null +++ b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/drp_param_info.txt @@ -0,0 +1,137 @@ +OFFSET_ADD:0,layer_name:post_0_0_conv_yuv2rgb,drp_lib:conv_yuv2rgb,prop:input,node:in, + Param:raddr, Value:0, offset:0, size:4, + Param:waddr, Value:614400, offset:4, size:4, + Param:IMG_IWIDHT, Value:640, offset:8, size:2, + Param:IMG_IHEIGHT, Value:480, offset:10,size:2, + Param:RFU1, Value:0, offset:12,size:2, + Param:RFU2, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:640, offset:16,size:2, + Param:IMG_OHEIGHT, Value:480, offset:18,size:2, + Param:RFU3, Value:0, offset:20,size:2, + Param:RFU4, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU5, Value:0, offset:42,size:2, + Param:RFU6, Value:0, offset:44,size:2, + Param:RFU7, Value:0, offset:46,size:2, + Param:INPUT_YUV_FORMAT, Value:0, offset:48,size:2, + Param:RFU8, Value:0, offset:50,size:2, + Param:DOUT_RGB_FORMAT, Value:0, offset:52,size:2, + Param:RFU10, Value:0, offset:54,size:2, +OFFSET_ADD:56,layer_name:post_0_1_resize_hwc,drp_lib:resize_hwc,prop:intermediate_data,node:, + Param:raddr, Value:614400, offset:0, size:4, + Param:waddr, Value:1536000, offset:4, size:4, + Param:IMG_IWIDHT, Value:640, offset:8, size:2, + Param:IMG_IHEIGHT, Value:480, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:RFU1, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:RFU2, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU3, Value:0, offset:42,size:2, + Param:RFU4, Value:0, offset:44,size:2, + Param:RFU5, Value:0, offset:46,size:2, + Param:RESIZE_ALG, Value:1, offset:48,size:2, + Param:DATA_TYPE, Value:0, offset:50,size:2, + Param:RFU7, Value:0, offset:52,size:2, + Param:RFU8, Value:0, offset:54,size:2, +OFFSET_ADD:112,layer_name:post_0_2_imagescaler,drp_lib:imagescaler,prop:intermediate_data,node:, + Param:raddr, Value:1536000, offset:0, size:4, + Param:waddr, Value:2055168, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:IMG_ICH_ORG, Value:3, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:IMG_OCH_ORG, Value:3, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU3, Value:0, offset:42,size:2, + Param:RFU4, Value:0, offset:44,size:2, + Param:RFU5, Value:0, offset:46,size:2, + Param:ADD_ADDR, Value:6208512, offset:48,size:4, + Param:MUL_ADDR, Value:6208520, offset:52,size:4, + Param:DIN_FORMAT, Value:0, offset:56,size:2, + Param:DOUT_RGB_ORDER, Value:0, offset:58,size:2, + Param:RFU6, Value:0, offset:60,size:2, + Param:RFU7, Value:0, offset:62,size:2, +OFFSET_ADD:176,layer_name:post_0_4_transpose,drp_lib:transpose,prop:intermediate_data,node:, + Param:raddr, Value:2055168, offset:0, size:4, + Param:waddr, Value:3093504, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:IMG_ICH_ORG, Value:3, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:IMG_OCH_ORG, Value:3, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU1, Value:0, offset:42,size:2, + Param:RFU2, Value:0, offset:44,size:2, + Param:RFU3, Value:0, offset:46,size:2, + Param:WORD_SIZE, Value:1, offset:48,size:2, + Param:IS_CHW2HWC, Value:0, offset:50,size:2, + Param:RFU4, Value:0, offset:52,size:2, + Param:RFU5, Value:0, offset:54,size:2, +OFFSET_ADD:232,layer_name:post_0_5_cast_fp16_fp32,drp_lib:cast_fp16_fp32,prop:output,node:out, + Param:raddr, Value:3093504, offset:0, size:4, + Param:waddr, Value:4131840, offset:4, size:4, + Param:IMG_IWIDHT, Value:416, offset:8, size:2, + Param:IMG_IHEIGHT, Value:416, offset:10,size:2, + Param:IMG_ICH, Value:3, offset:12,size:2, + Param:RFU1, Value:0, offset:14,size:2, + Param:IMG_OWIDTH, Value:416, offset:16,size:2, + Param:IMG_OHEIGHT, Value:416, offset:18,size:2, + Param:IMG_OCH, Value:3, offset:20,size:2, + Param:RFU2, Value:0, offset:22,size:2, + Param:OCH0_SYNCSET_DT, Value:1, offset:24,size:2, + Param:OCH0_SYNCSET_ID, Value:12, offset:26,size:2, + Param:OCH1_SYNCSET_DT, Value:2, offset:28,size:2, + Param:OCH1_SYNCSET_ID, Value:12, offset:30,size:2, + Param:OCH2_SYNCSET_DT, Value:4, offset:32,size:2, + Param:OCH2_SYNCSET_ID, Value:12, offset:34,size:2, + Param:OCH3_SYNCSET_DT, Value:24, offset:36,size:2, + Param:OCH3_SYNCSET_ID, Value:12, offset:38,size:2, + Param:INT_DISABLE, Value:1, offset:40,size:2, + Param:RFU1, Value:0, offset:42,size:2, + Param:RFU2, Value:0, offset:44,size:2, + Param:RFU3, Value:0, offset:46,size:2, + Param:CAST_MODE, Value:0, offset:48,size:2, + Param:RFU1, Value:0, offset:50,size:2, + Param:RFU2, Value:0, offset:52,size:2, + Param:RFU3, Value:0, offset:54,size:2, diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_addrmap_intm.txt b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_addrmap_intm.txt new file mode 100644 index 0000000..4883d37 --- /dev/null +++ b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_addrmap_intm.txt @@ -0,0 +1,9 @@ +data_in 0 96000 +data 96000 35ac00 +data_out 3f0c00 1fb000 +work 5ebc00 0 +weight 5ebc00 e +drp_config 5ebc40 1869a0 +drp_param 772600 120 +desc_aimac 772740 f0 +desc_drp 772840 1a0 diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_drpcfg.mem b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_drpcfg.mem new file mode 100644 index 0000000..f4fb291 Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_drpcfg.mem differ diff --git a/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_weight.dat b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_weight.dat new file mode 100644 index 0000000..9455fb8 Binary files /dev/null and b/Q08_object_counter/exe_v2l/vehicle/vehicle_onnx/preprocess/pp_weight.dat differ diff --git a/Q08_object_counter/images/COCO_v2h.png b/Q08_object_counter/images/COCO_v2h.png new file mode 100755 index 0000000..e68ae66 Binary files /dev/null and b/Q08_object_counter/images/COCO_v2h.png differ diff --git a/Q08_object_counter/images/Q08_animal.png b/Q08_object_counter/images/Q08_animal.png new file mode 100755 index 0000000..98d6bbd Binary files /dev/null and b/Q08_object_counter/images/Q08_animal.png differ diff --git a/Q08_object_counter/images/Q08_coco.png b/Q08_object_counter/images/Q08_coco.png new file mode 100755 index 0000000..e0ad8d3 Binary files /dev/null and b/Q08_object_counter/images/Q08_coco.png differ diff --git a/Q08_object_counter/images/Q08_vehicle.png b/Q08_object_counter/images/Q08_vehicle.png new file mode 100755 index 0000000..bcf0360 Binary files /dev/null and b/Q08_object_counter/images/Q08_vehicle.png differ diff --git a/Q08_object_counter/images/buffer_flow.png b/Q08_object_counter/images/buffer_flow.png new file mode 100755 index 0000000..a773c2a Binary files /dev/null and b/Q08_object_counter/images/buffer_flow.png differ diff --git a/Q08_object_counter/images/hw_conf_v2h.png b/Q08_object_counter/images/hw_conf_v2h.png new file mode 100755 index 0000000..7b3d546 Binary files /dev/null and b/Q08_object_counter/images/hw_conf_v2h.png differ diff --git a/Q08_object_counter/images/hw_conf_v2l.png b/Q08_object_counter/images/hw_conf_v2l.png new file mode 100755 index 0000000..0005ea3 Binary files /dev/null and b/Q08_object_counter/images/hw_conf_v2l.png differ diff --git a/Q08_object_counter/src_v2h/CMakeLists.txt b/Q08_object_counter/src_v2h/CMakeLists.txt new file mode 100755 index 0000000..429c0bf --- /dev/null +++ b/Q08_object_counter/src_v2h/CMakeLists.txt @@ -0,0 +1,38 @@ +cmake_minimum_required(VERSION 3.12) +project(object_counter_cpp) + +set(CMAKE_CXX_STANDARD 17) + +set(TVM_ROOT $ENV{TVM_HOME}) +include_directories(${TVM_ROOT}/include) +include_directories(${TVM_ROOT}/3rdparty/dlpack/include) +include_directories(${TVM_ROOT}/3rdparty/dmlc-core/include) +include_directories(${TVM_ROOT}/3rdparty/compiler-rt) + +set(TVM_RUNTIME_LIB ${TVM_ROOT}/build_runtime/libtvm_runtime.so) +set(EXE_NAME object_counter) + +file(GLOB SOURCE *.cpp *.h) +add_executable (${EXE_NAME} +${SOURCE} +) +TARGET_LINK_LIBRARIES(${EXE_NAME} pthread) +TARGET_LINK_LIBRARIES(${EXE_NAME} jpeg) +target_link_libraries(${EXE_NAME} + pthread glib-2.0 mmngr gobject-2.0 gstreamer-1.0 gstbase-1.0 gstapp-1.0 + opencv_imgcodecs opencv_imgproc opencv_core opencv_highgui + jpeg webp tiff z tbb gtk-3 png16 gdk-3 cairo + lzma rt cairo-gobject + xkbcommon wayland-cursor wayland-egl wayland-client epoxy + fribidi harfbuzz fontconfig + glib-2.0 gobject-2.0 gdk_pixbuf-2.0 gmodule-2.0 pangocairo-1.0 + atk-1.0 gio-2.0 pango-1.0 freetype pixman-1 uuid pcre + mount resolv expat pangoft2-1.0 blkid + EGL GLESv2 mmngrbuf) +find_package(OpenCV REQUIRED) +if(OpenCV_FOUND) + target_include_directories(${EXE_NAME} PUBLIC ${OpenCV_INCLUDE_DIRS}) + target_link_libraries(${EXE_NAME} ${OpenCV_LIBS}) +endif() +target_link_libraries(${EXE_NAME} ${TVM_RUNTIME_LIB}) +target_compile_definitions(${EXE_NAME} PRIVATE V2H) diff --git a/Q08_object_counter/src_v2h/MeraDrpRuntimeWrapper.cpp b/Q08_object_counter/src_v2h/MeraDrpRuntimeWrapper.cpp new file mode 100755 index 0000000..ff005a8 --- /dev/null +++ b/Q08_object_counter/src_v2h/MeraDrpRuntimeWrapper.cpp @@ -0,0 +1,208 @@ +/* + * Original Code (C) Copyright Edgecortix, Inc. 2022 + * Modified Code (C) Copyright Renesas Electronics Corporation 2023 + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * + */ +#include +#include +#include +#include + +#include +#include +#include +#include "MeraDrpRuntimeWrapper.h" + +template +static std::vector LoadBinary(const std::string& bin_file) { + std::ifstream file(bin_file.c_str(), std::ios::in | std::ios::binary); + if (!file.is_open()) { + LOG(FATAL) << "unable to open file " + bin_file; + } + + file.seekg(0, file.end); + const uint32_t file_size = static_cast(file.tellg()); + file.seekg(0, file.beg); + + const auto file_buffer = std::unique_ptr(new char[file_size]); + file.read(file_buffer.get(), file_size); + + if (file.bad() || file.fail()) { + LOG(FATAL) << "error occured while reading the file"; + } + + file.close(); + + auto ptr = reinterpret_cast(file_buffer.get()); + const auto num_elements = file_size / sizeof(T); + return std::vector(ptr, ptr + num_elements); +} + +MeraDrpRuntimeWrapper::MeraDrpRuntimeWrapper() { + //device_type = kDLCPU; + device_type = kDLDrpAi; + device_id = 0; +}; + +MeraDrpRuntimeWrapper::~MeraDrpRuntimeWrapper() = default; + +bool MeraDrpRuntimeWrapper::LoadModel(const std::string& model_dir, uint32_t start_address){ + device_type = kDLCPU; + + return LoadModel(model_dir, (uint64_t)start_address); +} + +bool MeraDrpRuntimeWrapper::LoadModel(const std::string& model_dir, uint64_t start_address = 0x00) { + LOG(INFO) << "Loading json data..."; + const std::string json_file(model_dir + "/deploy.json"); + std::ifstream json_in(json_file.c_str(), std::ios::in); + std::string json_data((std::istreambuf_iterator(json_in)), std::istreambuf_iterator()); + json_in.close(); + + #if 0 + if(json_data.find("drp") == json_data.npos && device_type != kDLCPU){ + LOG(INFO) <<"Break! this model is Not for DRP-AI retry as CPU Only"; + return false; + } + #else + if(json_data.find("drp") == json_data.npos && device_type != kDLCPU){ + LOG(INFO) <<"try as CPU Only"; + device_type = kDLCPU; + } + #endif + + LOG(INFO) << "Loading runtime module..."; + tvm::runtime::Module mod_syslib = tvm::runtime::Module::LoadFromFile(model_dir + "/deploy.so"); + mod = (*tvm::runtime::Registry::Get("tvm.graph_executor_debug.create"))( + json_data, mod_syslib, device_type, device_id); + + LOG(INFO) << "Loading parameters..."; + tvm::runtime::PackedFunc load_params = mod.GetFunction("load_params"); + auto params_data = LoadBinary(model_dir + "/deploy.params"); + TVMByteArray params_arr; + params_arr.data = params_data.data(); + params_arr.size = params_data.size(); + load_params(params_arr); + tvm::runtime::PackedFunc set_start_address = mod.GetFunction("set_start_address"); + if(set_start_address != nullptr){ + set_start_address(start_address); + } + return true; +} + +template +void MeraDrpRuntimeWrapper::SetInput(int input_index, const T* data_ptr) { + LOG(INFO) << "Loading input..."; + + tvm::runtime::PackedFunc get_input = mod.GetFunction("get_input"); + tvm::runtime::NDArray xx = get_input(input_index); + auto in_shape = xx.Shape(); + int64_t in_size = 1; + for (unsigned long i = 0; i < in_shape.size(); ++i) { + in_size *= in_shape[i]; + } + + DLDevice ctx; + ctx.device_id = device_id; + ctx.device_type = DLDeviceType(device_type); + + auto input_array = tvm::runtime::NDArray::Empty(in_shape, xx.DataType(), ctx); + auto input_data = (T*)(input_array->data); + std::memcpy(input_data, data_ptr, sizeof(T) * in_size); + tvm::runtime::PackedFunc set_input = mod.GetFunction("set_input"); + set_input(input_index, input_array); +} +template void MeraDrpRuntimeWrapper::SetInput(int input_index, const float*); +template void MeraDrpRuntimeWrapper::SetInput(int input_index, const unsigned short*); + +void MeraDrpRuntimeWrapper::Run() { + mod.GetFunction("run")(); +} + +void MeraDrpRuntimeWrapper::ProfileRun(const std::string& profile_table, const std::string& profile_csv) { + tvm::runtime::PackedFunc profile = mod.GetFunction("profile"); + tvm::runtime::Array collectors; + tvm::runtime::profiling::Report report = profile(collectors); + + std::string rep_table = report->AsTable(); + std::ofstream ofs_table (profile_table, std::ofstream::out); + ofs_table << rep_table << std::endl; + ofs_table.close(); + + std::string rep_csv = report->AsCSV(); + std::ofstream ofs_csv (profile_csv, std::ofstream::out); + ofs_csv << rep_csv << std::endl; + ofs_csv.close(); +} + +int MeraDrpRuntimeWrapper::GetNumInput(std::string model_dir) { + // TVM does not support api to get number input of model. + // This function calculate input number base on convention + // of input data name (input_xyz.bin) + DIR *dir; + dirent *diread; + int num_input = 0; + if ((dir = opendir(model_dir.c_str())) != nullptr) { + while ((diread = readdir(dir)) != nullptr) { + std::string file_name(diread->d_name); + if (std::regex_match(file_name, std::regex("(input_)(.*)(bin)") )) { + num_input++; + } + } + closedir(dir); + } else { + LOG(FATAL) << "Can not open model dir : " << model_dir; + } + + return num_input; +} + +InOutDataType MeraDrpRuntimeWrapper::GetInputDataType(int index) { + tvm::runtime::PackedFunc get_input = mod.GetFunction("get_input"); + tvm::runtime::NDArray input_info = get_input(index); + InOutDataType data_type = InOutDataType::OTHER; + if (input_info.DataType().is_float() && input_info.DataType().bits() == 32) { + data_type = InOutDataType::FLOAT32; + } else if (input_info.DataType().is_float() && input_info.DataType().bits() == 16) { + data_type = InOutDataType::FLOAT16; + } + return data_type; + } + +int MeraDrpRuntimeWrapper::GetNumOutput() { + return mod.GetFunction("get_num_outputs")(); + } + +std::tuple MeraDrpRuntimeWrapper::GetOutput(int index) { + tvm::runtime::PackedFunc get_output = mod.GetFunction("get_output"); + tvm::runtime::NDArray out = get_output(index); + int64_t out_size = 1; + for ( unsigned long i = 0; i < out.Shape().size(); ++i) { + out_size *= out.Shape()[i]; + } + + InOutDataType data_type = InOutDataType::OTHER; + if (out.DataType().is_float() && out.DataType().bits() == 32) { + data_type = InOutDataType::FLOAT32; + } else if (out.DataType().is_float() && out.DataType().bits() == 16) { + data_type = InOutDataType::FLOAT16; + } + return std::make_tuple(data_type, reinterpret_cast(out->data), out_size); +} diff --git a/Q08_object_counter/src_v2h/MeraDrpRuntimeWrapper.h b/Q08_object_counter/src_v2h/MeraDrpRuntimeWrapper.h new file mode 100755 index 0000000..5f1b2be --- /dev/null +++ b/Q08_object_counter/src_v2h/MeraDrpRuntimeWrapper.h @@ -0,0 +1,52 @@ +/* + * Original Code (C) Copyright Edgecortix, Inc. 2022 + * Modified Code (C) Copyright Renesas Electronics Corporation 2023 + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * +*/ +#include + +enum class InOutDataType { + FLOAT32, + FLOAT16, + OTHER +}; + +class MeraDrpRuntimeWrapper { + public: + MeraDrpRuntimeWrapper(); + ~MeraDrpRuntimeWrapper(); + + bool LoadModel(const std::string& model_dir, uint32_t start_address); + bool LoadModel(const std::string& model_dir, uint64_t start_address); + template + void SetInput(int input_index, const T* data_ptr); + void Run(); + void ProfileRun(const std::string& profile_table, const std::string& profile_csv); + int GetNumInput(std::string model_dir); + InOutDataType GetInputDataType(int index); + int GetNumOutput(); + + std::tuple GetOutput(int index); + + private: + int device_type; + int device_id; + tvm::runtime::Module mod; +}; diff --git a/Q08_object_counter/src_v2h/box.cpp b/Q08_object_counter/src_v2h/box.cpp new file mode 100755 index 0000000..1d8eeb9 --- /dev/null +++ b/Q08_object_counter/src_v2h/box.cpp @@ -0,0 +1,140 @@ +/*********************************************************************************************************************** +* DISCLAIMER +* This software is supplied by Renesas Electronics Corporation and is only intended for use with Renesas products. No +* other uses are authorized. This software is owned by Renesas Electronics Corporation and is protected under all +* applicable laws, including copyright laws. +* THIS SOFTWARE IS PROVIDED "AS IS" AND RENESAS MAKES NO WARRANTIES REGARDING +* THIS SOFTWARE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. ALL SUCH WARRANTIES ARE EXPRESSLY DISCLAIMED. TO THE MAXIMUM +* EXTENT PERMITTED NOT PROHIBITED BY LAW, NEITHER RENESAS ELECTRONICS CORPORATION NOR ANY OF ITS AFFILIATED COMPANIES +* SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES FOR ANY REASON RELATED TO THIS +* SOFTWARE, EVEN IF RENESAS OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +* Renesas reserves the right, without notice, to make changes to this software and to discontinue the availability of +* this software. By using this software, you agree to the additional terms and conditions found by accessing the +* following link: +* http://www.renesas.com/disclaimer +* +* Copyright (C) 2023 Renesas Electronics Corporation. All rights reserved. +***********************************************************************************************************************/ +/*********************************************************************************************************************** +* File Name : box.cpp +* Version : v1.00 +* Description : RZ/V2H AI SDK Sample Application for Object Detection +***********************************************************************************************************************/ + +/***************************************** +* Includes +******************************************/ +#include "box.h" + +/***************************************** +* Function Name : overlap +* Description : Function to compute the overlapped data between coordinate x with size w +* Arguments : x1 = 1-dimensional coordinate of first line +* w1 = size of fist line +* x2 = 1-dimensional coordinate of second line +* w2 = size of second line +* Return value : overlapped line size +******************************************/ +float overlap(float x1, float w1, float x2, float w2) +{ + float l1 = x1 - w1/2; + float l2 = x2 - w2/2; + float left = l1 > l2 ? l1 : l2; + float r1 = x1 + w1/2; + float r2 = x2 + w2/2; + float right = r1 < r2 ? r1 : r2; + return right - left; +} + +/***************************************** +* Function Name : box_intersection +* Description : Function to compute the area of intersection of Box a and b +* Arguments : a = Box 1 +* b = Box 2 +* Return value : area of intersection +******************************************/ +float box_intersection(Box a, Box b) +{ + float w = overlap(a.x, a.w, b.x, b.w); + float h = overlap(a.y, a.h, b.y, b.h); + if(w < 0 || h < 0) + { + return 0; + } + float area = w*h; + return area; +} + +/***************************************** +* Function Name : box_union +* Description : Function to compute the area of union of Box a and b +* Arguments : a = Box 1 +* b = Box 2 +* Return value : area of union +******************************************/ +float box_union(Box a, Box b) +{ + float i = box_intersection(a, b); + float u = a.w*a.h + b.w*b.h - i; + return u; +} + +/***************************************** +* Function Name : box_iou +* Description : Function to compute the Intersection over Union (IoU) of Box a and b +* Arguments : a = Box 1 +* b = Box 2 +* Return value : IoU +******************************************/ +float box_iou(Box a, Box b) +{ + return box_intersection(a, b)/box_union(a, b); +} + +/***************************************** +* Function Name : filter_boxes_nms +* Description : Apply Non-Maximum Suppression (NMS) to get rid of overlapped rectangles. +* Arguments : det= detected rectangles +* size = number of detections stored in det +* th_nms = threshold for nms +* Return value : - +******************************************/ +void filter_boxes_nms(std::vector &det, int32_t size, float th_nms) +{ + int32_t count = size; + int32_t i = 0; + int32_t j = 0; + Box a; + Box b; + float b_intersection = 0; + for (i = 0; i < count; i++) + { + a = det[i].bbox; + for (j = 0; j < count; j++) + { + if (i == j) + { + continue; + } + if (det[i].c != det[j].c) + { + continue; + } + b = det[j].bbox; + b_intersection = box_intersection(a, b); + if ((box_iou(a, b)>th_nms) || (b_intersection >= a.h * a.w - 1) || (b_intersection >= b.h * b.w - 1)) + { + if (det[i].prob > det[j].prob) + { + det[j].prob= 0; + } + else + { + det[i].prob= 0; + } + } + } + } + return; +} diff --git a/Q08_object_counter/src_v2h/box.h b/Q08_object_counter/src_v2h/box.h new file mode 100755 index 0000000..872ba23 --- /dev/null +++ b/Q08_object_counter/src_v2h/box.h @@ -0,0 +1,73 @@ +/*********************************************************************************************************************** +* DISCLAIMER +* This software is supplied by Renesas Electronics Corporation and is only intended for use with Renesas products. No +* other uses are authorized. This software is owned by Renesas Electronics Corporation and is protected under all +* applicable laws, including copyright laws. +* THIS SOFTWARE IS PROVIDED "AS IS" AND RENESAS MAKES NO WARRANTIES REGARDING +* THIS SOFTWARE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. ALL SUCH WARRANTIES ARE EXPRESSLY DISCLAIMED. TO THE MAXIMUM +* EXTENT PERMITTED NOT PROHIBITED BY LAW, NEITHER RENESAS ELECTRONICS CORPORATION NOR ANY OF ITS AFFILIATED COMPANIES +* SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES FOR ANY REASON RELATED TO THIS +* SOFTWARE, EVEN IF RENESAS OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +* Renesas reserves the right, without notice, to make changes to this software and to discontinue the availability of +* this software. By using this software, you agree to the additional terms and conditions found by accessing the +* following link: +* http://www.renesas.com/disclaimer +* +* Copyright (C) 2023 Renesas Electronics Corporation. All rights reserved. +***********************************************************************************************************************/ +/*********************************************************************************************************************** +* File Name : box.h +* Version : v1.00 +* Description : RZ/V2H AI SDK Sample Application for Object Detection +***********************************************************************************************************************/ + +#ifndef BOX_H +#define BOX_H + +#include +#include +#include +#include +#include + +/***************************************** +* Box : Bounding box coordinates and its size +******************************************/ +typedef struct +{ + float x, y, w, h; +} Box; + +/***************************************** +* detection : Detected result +******************************************/ +typedef struct detection +{ + Box bbox; + int32_t c; + float prob; +} detection; + +/***************************************** +* detection : Detected result +******************************************/ +struct bbox_t +{ + std::string name; + int32_t X; + int32_t Y; + int32_t W; + int32_t H; + float pred; +}; +/***************************************** +* Functions +******************************************/ +float box_iou(Box a, Box b); +float overlap(float x1, float w1, float x2, float w2); +float box_intersection(Box a, Box b); +float box_union(Box a, Box b); +void filter_boxes_nms(std::vector &det, int32_t size, float th_nms); + +#endif diff --git a/Q08_object_counter/src_v2h/define.h b/Q08_object_counter/src_v2h/define.h new file mode 100755 index 0000000..786e797 --- /dev/null +++ b/Q08_object_counter/src_v2h/define.h @@ -0,0 +1,182 @@ +/* + * Original Code (C) Copyright Edgecortix, Inc. 2022 + * Modified Code (C) Copyright Renesas Electronics Corporation 2023 + *　 + * *1 DRP-AI TVM is powered by EdgeCortix MERA(TM) Compiler Framework. + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * + */ +/*********************************************************************************************************************** +* DISCLAIMER +* This software is supplied by Renesas Electronics Corporation and is only intended for use with Renesas products. No +* other uses are authorized. This software is owned by Renesas Electronics Corporation and is protected under all +* applicable laws, including copyright laws. +* THIS SOFTWARE IS PROVIDED "AS IS" AND RENESAS MAKES NO WARRANTIES REGARDING +* THIS SOFTWARE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. ALL SUCH WARRANTIES ARE EXPRESSLY DISCLAIMED. TO THE MAXIMUM +* EXTENT PERMITTED NOT PROHIBITED BY LAW, NEITHER RENESAS ELECTRONICS CORPORATION NOR ANY OF ITS AFFILIATED COMPANIES +* SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES FOR ANY REASON RELATED TO THIS +* SOFTWARE, EVEN IF RENESAS OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +* Renesas reserves the right, without notice, to make changes to this software and to discontinue the availability of +* this software. By using this software, you agree to the additional terms and conditions found by accessing the +* following link: +* http://www.renesas.com/disclaimer +* +* Copyright (C) 2023 Renesas Electronics Corporation. All rights reserved. +***********************************************************************************************************************/ +/*********************************************************************************************************************** +* File Name : define.h +* Version : v1.00 +* Description : RZ/V2H AI SDK Sample Application for Object Detection +***********************************************************************************************************************/ + +#ifndef DEFINE_MACRO_H +#define DEFINE_MACRO_H + +/***************************************** +* includes +******************************************/ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "opencv2/core.hpp" +#include "opencv2/imgproc.hpp" +#include "opencv2/highgui.hpp" +#include "wayland.h" + + +static Wayland wayland; +/* Empty since labels will be loaded from label_list file */ +static std::vector label_file_map = {}; + +/***************************************** +* Macro for YOLOv3 +******************************************/ + +/* Number of class to be detected */ +int32_t NUM_CLASS; +/* Number for [region] layer num parameter */ +#define NUM_BB (3) +#define NUM_INF_OUT_LAYER (3) +/* Thresholds */ +#define TH_PROB (0.5f) +#define TH_NMS (0.5f) +// /* Size of input image to the model */ +#define MODEL_IN_W (416) +#define MODEL_IN_H (416) + +/* Number of grids in the image. The length of this array MUST match with the NUM_INF_OUT_LAYER */ +const static uint8_t num_grids[] = { 13, 26, 52 }; + +/***************************************** +* Macro for Application +******************************************/ +#define IMAGE_CHANNEL_BGRA (4) +/*Waiting Time*/ +#define WAIT_TIME (1000) /* microseconds */ + +/*Timer Related*/ +#define CAPTURE_TIMEOUT (20) /* seconds */ +#define AI_THREAD_TIMEOUT (20) /* seconds */ +/*DRP-AI memory area offset for model objects*/ +/*Offset value depends on the size of memory area used by DRP-AI Pre-processing Runtime Object files*/ +#define DRPAI_MEM_OFFSET (0) + +/*Maximum DRP-AI Timeout threshold*/ +#define DRPAI_TIMEOUT (5) + +/*Frame threshold to execute inference in every loop + *This value must be determined by DRP-AI processing time and capture processing time. + *For your information YOLOv3 takes around 50 msec and capture takes around 50 msec. */ +#define INF_FRAME_NUM (2) + +/*Camera Capture Image Information*/ +#define CAM_IMAGE_WIDTH (640) +#define CAM_IMAGE_HEIGHT (480) +#define CAM_IMAGE_CHANNEL_YUY2 (2) + +#define CAP_BUF_NUM (6) + +/*DRP-AI Input image information*/ +#define DRPAI_IN_WIDTH (CAM_IMAGE_WIDTH) +#define DRPAI_IN_HEIGHT (CAM_IMAGE_HEIGHT) +#define DRPAI_IN_CHANNEL_YUY2 (CAM_IMAGE_CHANNEL_YUY2) + +/* Display Image Information*/ +#define DISP_OUTPUT_WIDTH (1920) +#define DISP_OUTPUT_HEIGHT (1080) +#define DISP_IMAGE_OUTPUT_WIDTH (1500) +#define DISP_IMAGE_OUTPUT_HEIGHT (1080) + + +/*Waiting Time*/ +#define WAIT_TIME (1000) /* microseconds */ + +/*Timer Related*/ +#define CAPTURE_TIMEOUT (20) /* seconds */ +#define AI_THREAD_TIMEOUT (20) /* seconds */ +#define KEY_THREAD_TIMEOUT (5) /* seconds */ +/* DRP_MAX_FREQ and DRPAI_FREQ are the */ +/* frequency settings for DRP-AI. */ +/*Basicallyuse the default values */ + +#define DRP_MAX_FREQ (2) +/* DRP_MAX_FREQ can be set from 2 to 127 */ +/* 2: 420MHz */ +/* 3: 315MHz */ +/* ... */ +/* 127: 9.84MHz */ +/* Calculation Formula: */ +/* 1260MHz /(DRP_MAX_FREQ + 1) */ + +#define DRPAI_FREQ (2) +/* DRPAI_FREQ can be set from 1 to 127 */ +/* 1,2: 1GHz */ +/* 3: 630MHz */ +/* 4: 420MHz */ +/* 5: 315MHz */ +/* ... */ +/* 127: 10MHz */ +/* Calculation Formula: */ +/* 1260MHz /(DRPAI_FREQ - 1) */ +/* (When DRPAI_FREQ = 3 or more.) */ + + +#endif diff --git a/Q08_object_counter/src_v2h/main.cpp b/Q08_object_counter/src_v2h/main.cpp new file mode 100755 index 0000000..64fa495 --- /dev/null +++ b/Q08_object_counter/src_v2h/main.cpp @@ -0,0 +1,1415 @@ +/* + * Original Code (C) Copyright Edgecortix, Inc. 2022 + * Modified Code (C) Copyright Renesas Electronics Corporation 2023 + *　 + * *1 DRP-AI TVM is powered by EdgeCortix MERA(TM) Compiler Framework. + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * + */ +/*********************************************************************************************************************** +* DISCLAIMER +* This software is supplied by Renesas Electronics Corporation and is only intended for use with Renesas products. No +* other uses are authorized. This software is owned by Renesas Electronics Corporation and is protected under all +* applicable laws, including copyright laws. +* THIS SOFTWARE IS PROVIDED "AS IS" AND RENESAS MAKES NO WARRANTIES REGARDING +* THIS SOFTWARE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. ALL SUCH WARRANTIES ARE EXPRESSLY DISCLAIMED. TO THE MAXIMUM +* EXTENT PERMITTED NOT PROHIBITED BY LAW, NEITHER RENESAS ELECTRONICS CORPORATION NOR ANY OF ITS AFFILIATED COMPANIES +* SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES FOR ANY REASON RELATED TO THIS +* SOFTWARE, EVEN IF RENESAS OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +* Renesas reserves the right, without notice, to make changes to this software and to discontinue the availability of +* this software. By using this software, you agree to the additional terms and conditions found by accessing the +* following link: +* http://www.renesas.com/disclaimer +* +* Copyright (C) 2023 Renesas Electronics Corporation. All rights reserved. +***********************************************************************************************************************/ +/*********************************************************************************************************************** +* File Name : main.cpp +* Version : v1.00 +* Description : RZ/V2H AI SDK Application for Object Detection +***********************************************************************************************************************/ + +/***************************************** +* Includes +******************************************/ +/*DRP-AI TVM[*1] Runtime*/ +#include "MeraDrpRuntimeWrapper.h" +/*Definition of Macros & other variables*/ +#include "define.h" +/*box drawing*/ +#include "box.h" +#include "utils.h" + + + +/*Multithreading*/ +static sem_t terminate_req_sem; +static pthread_t ai_inf_thread; +static pthread_t capture_thread; +static pthread_t exit_thread; +static pthread_t kbhit_thread; +static std::mutex mtx; + +/*Flags*/ +static std::atomic inference_start (0); +static std::atomic img_obj_ready (0); + +/*Global Variables*/ +float * drpai_output_buf; +static uint64_t udmabuf_address = 0; + +/*AI Inference for DRPAI*/ +/* DRP-AI TVM[*1] Runtime object */ +MeraDrpRuntimeWrapper runtime; + +static float pre_time = 0; +static float post_time = 0; +static float ai_time = 0; +static float total_time = 0; +static std::vector det; +cv::Mat yuyv_image; +cv::Mat input_image; +std::unordered_map ini_values; +std::vector anchors; +bool doubleClick = false; +static int32_t drp_max_freq; +static int32_t drpai_freq; +using INI_FORMAT = std::unordered_map>; +/***************************************** +* Function Name : float16_to_float32 +* Description : Function by Edgecortex. Cast uint16_t a into float value. +* Arguments : a = uint16_t number +* Return value : float = float32 number +******************************************/ +float float16_to_float32(uint16_t a) +{ + return __extendXfYf2__(a); +} + +/***************************************** +* Function Name : timedifference_msec +* Description : compute the time differences in ms between two moments +* Arguments : t0 = start time +* t1 = stop time +* Return value : the time difference in ms +******************************************/ +static double timedifference_msec(struct timespec t0, struct timespec t1) +{ + return (t1.tv_sec - t0.tv_sec) * 1000.0 + (t1.tv_nsec - t0.tv_nsec) / 1000000.0; +} + +/***************************************** +* Function Name : wait_join +* Description : waits for a fixed amount of time for the thread to exit +* Arguments : p_join_thread = thread that the function waits for to Exit +* join_time = the timeout time for the thread for exiting +* Return value : 0 if successful +* not 0 otherwise +******************************************/ +static int8_t wait_join(pthread_t *p_join_thread, uint32_t join_time) +{ + int8_t ret_err; + struct timespec join_timeout; + ret_err = clock_gettime(CLOCK_REALTIME, &join_timeout); + if ( 0 == ret_err ) + { + join_timeout.tv_sec += join_time; + ret_err = pthread_timedjoin_np(*p_join_thread, NULL, &join_timeout); + } + return ret_err; +} + +/***************************************** +* Function Name : config_read +* Description : Read configuration from the config.ini file. +******************************************/ +INI_FORMAT config_read(std::string file) +{ + INI_FORMAT ini_values; + /*Open config.ini file*/ + std::ifstream ini_file(file); + std::string line; + std::string current_section; + /*parsing ini file*/ + while (std::getline(ini_file, line)) + { + std::string::size_type comment_pos = line.find(";"); + if (comment_pos != std::string::npos) + { + line.erase(comment_pos); + } + line.erase(0, line.find_first_not_of(" \t\r\n")); + line.erase(line.find_last_not_of(" \t\r\n") + 1); + if (line.empty()) + { + continue; + } + else if (line[0] == '[') + { + current_section = line.substr(1, line.size() - 2); + } + else + { + std::string::size_type delimiter_pos = line.find("="); + std::string key = line.substr(0, delimiter_pos); + std::string value = line.substr(delimiter_pos + 1); + ini_values[current_section][key] = value; + } + } + return ini_values; +} +/***************************************** +* Function Name : load_label_file +* Description : Load label list text file and return the label list that contains the label. +* Arguments : label_file_name = filename of label list. must be in txt format +* Return value : vector list = list contains labels +* empty if error occurred +******************************************/ +std::vector load_label_file(std::string label_file_name) +{ + std::vector list = {}; + std::vector empty = {}; + std::ifstream infile(label_file_name); + + if (!infile.is_open()) + { + return list; + } + + std::string line = ""; + while (getline(infile,line)) + { + list.push_back(line); + if (infile.fail()) + { + return empty; + } + } + + return list; +} + +/***************************************** +* Function Name : get_result +* Description : Get DRP-AI Output from memory via DRP-AI Driver +* Arguments : drpai_fd = file descriptor of DRP-AI Driver +* output_addr = memory start address of DRP-AI output +* output_size = output data size +* Return value : 0 if succeeded +* not 0 otherwise +******************************************/ +int8_t get_result() +{ + int8_t ret = 0; + int32_t i = 0; + + int32_t output_num = 0; + std::tuple output_buffer; + int64_t output_size; + uint32_t size_count = 0; + + /* Get the number of output of the target model. */ + output_num = runtime.GetNumOutput(); + size_count = 0; + /*GetOutput loop*/ + for (i = 0;i(output_buffer). */ + output_size = std::get<2>(output_buffer); + + /*Output Data Type = std::get<0>(output_buffer)*/ + if (InOutDataType::FLOAT16 == std::get<0>(output_buffer)) + { + /*Output Data = std::get<1>(output_buffer)*/ + uint16_t* data_ptr = reinterpret_cast(std::get<1>(output_buffer)); + for (int j = 0; j(output_buffer)) + { + /*Output Data = std::get<1>(output_buffer)*/ + float* data_ptr = reinterpret_cast(std::get<1>(output_buffer)); + for (int j = 0; j rgb_images; + cv::split(image, rgb_images); + cv::Mat m_flat_r = rgb_images[0].reshape(1, 1); + cv::Mat m_flat_g = rgb_images[1].reshape(1, 1); + cv::Mat m_flat_b = rgb_images[2].reshape(1, 1); + cv::Mat matArray[] = {m_flat_r, m_flat_g, m_flat_b}; + cv::Mat flat_image; + cv::hconcat(matArray, 3, flat_image); + return flat_image; +} + +/***************************************** +* Function Name : R_Post_Proc +* Description : Process CPU post-processing for YOLOv3 +* Arguments : floatarr = drpai output address +* Return value : - +******************************************/ +void R_Post_Proc(float* floatarr) +{ + /* Following variables are required for correct_region_boxes in Darknet implementation*/ + /* Note: This implementation refers to the "darknet detector test" */ + mtx.lock(); + float new_w, new_h; + float correct_w = 1.; + float correct_h = 1.; + if ((float) (MODEL_IN_W / correct_w) < (float) (MODEL_IN_H/correct_h) ) + { + new_w = (float) MODEL_IN_W; + new_h = correct_h * MODEL_IN_W / correct_w; + } + else + { + new_w = correct_w * MODEL_IN_H / correct_h; + new_h = MODEL_IN_H; + } + + int32_t n = 0; + int32_t b = 0; + int32_t y = 0; + int32_t x = 0; + int32_t offs = 0; + int32_t i = 0; + float tx = 0; + float ty = 0; + float tw = 0; + float th = 0; + float tc = 0; + float center_x = 0; + float center_y = 0; + float box_w = 0; + float box_h = 0; + float objectness = 0; + uint8_t num_grid = 0; + uint8_t anchor_offset = 0; + float classes[NUM_CLASS]; + float max_pred = 0; + int32_t pred_class = -1; + float probability = 0; + detection d; + /* Clear the detected result list */ + det.clear(); + + /*Post Processing Start*/ + for (n = 0; n < NUM_INF_OUT_LAYER; n++) + { + num_grid = num_grids[n]; + anchor_offset = 2 * NUM_BB * (NUM_INF_OUT_LAYER - (n + 1)); + + for(b = 0; b < NUM_BB; b++) + { + for(y = 0; y < num_grid; y++) + { + for(x = 0; x < num_grid; x++) + { + offs = yolo_offset(n, b, y, x); + tx = floatarr[offs]; + ty = floatarr[yolo_index(n, offs, 1)]; + tw = floatarr[yolo_index(n, offs, 2)]; + th = floatarr[yolo_index(n, offs, 3)]; + tc = floatarr[yolo_index(n, offs, 4)]; + /* Compute the bounding box */ + /*get_region_box*/ + center_x = ((float) x + sigmoid(tx)) / (float) num_grid; + center_y = ((float) y + sigmoid(ty)) / (float) num_grid; + box_w = (float) exp(tw) * anchors[anchor_offset+2*b+0] / (float) MODEL_IN_W; + box_h = (float) exp(th) * anchors[anchor_offset+2*b+1] / (float) MODEL_IN_W; + /* Adjustment for VGA size */ + /* correct_region_boxes */ + center_x = (center_x - (MODEL_IN_W - new_w) / 2. / MODEL_IN_W) / ((float) new_w / MODEL_IN_W); + center_y = (center_y - (MODEL_IN_H - new_h) / 2. / MODEL_IN_H) / ((float) new_h / MODEL_IN_H); + box_w *= (float) (MODEL_IN_W / new_w); + box_h *= (float) (MODEL_IN_H / new_h); + center_x = round(center_x * DRPAI_IN_WIDTH); + center_y = round(center_y * DRPAI_IN_HEIGHT); + box_w = round(box_w * DRPAI_IN_WIDTH); + box_h = round(box_h * DRPAI_IN_HEIGHT); + objectness = sigmoid(tc); + Box bb = {center_x, center_y, box_w, box_h}; + /* Get the class prediction */ + for (i = 0; i < NUM_CLASS; i++) + { + classes[i] = sigmoid(floatarr[yolo_index(n, offs, 5+i)]); + } + max_pred = 0; + pred_class = -1; + for (i = 0; i < NUM_CLASS; i++) + { + if (classes[i] > max_pred) + { + pred_class = i; + max_pred = classes[i]; + } + } + /* Store the result into the list if the probability is more than the threshold */ + probability = max_pred * objectness; + if (probability > TH_PROB) + { + d = {bb, pred_class, probability}; + det.push_back(d); + } + } + } + } + } + /* Non-Maximum Supression filter */ + filter_boxes_nms(det, det.size(), TH_NMS); + mtx.unlock(); + return ; +} + + + + + +/***************************************** + * Function Name : R_exit_Thread + * Description : Executes the double click exit thread + * Arguments : threadid = thread identification + * Return value : - + ******************************************/ +void *R_exit_Thread(void *threadid) +{ + /*Semaphore Variable*/ + int32_t kh_sem_check = 0; + + /*Variable for checking return value*/ + int8_t ret = 0; + devices dev; + + /*Set Standard Input to Non Blocking*/ + errno = 0; + ret = fcntl(0, F_SETFL, O_NONBLOCK); + if (-1 == ret) + { + fprintf(stderr, "[ERROR] Failed to run fctnl(): errno=%d\n", errno); + goto err; + } + + while (1) + { + /*Gets the Termination request semaphore value. If different then 1 Termination was requested*/ + /*Checks if sem_getvalue is executed wihtout issue*/ + errno = 0; + ret = sem_getvalue(&terminate_req_sem, &kh_sem_check); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to get Semaphore Value: errno=%d\n", errno); + goto err; + } + /*Checks the semaphore value*/ + if (1 != kh_sem_check) + { + goto exit_end; + } + + dev.detect_mouse_click(); + if (doubleClick) + { + goto err; + } + } + +/*Error Processing*/ +err: + /*Set Termination Request Semaphore to 0*/ + sem_trywait(&terminate_req_sem); + goto exit_end; + +exit_end: + printf("Exit Thread Terminated\n"); + pthread_exit(NULL); +} + + + +/***************************************** +* Function Name : R_Inf_Thread +* Description : Executes the DRP-AI inference thread +* Arguments : threadid = thread identification +* Return value : - +******************************************/ +void *R_Inf_Thread(void *threadid) +{ + /*Semaphore Variable*/ + int32_t inf_sem_check = 0; + + /*Inference Variables*/ + fd_set rfds; + struct timespec tv; + int8_t inf_status = 0; + /*Variable for checking return value*/ + int8_t ret = 0; + /*Variable for Performance Measurement*/ + static struct timespec start_time; + static struct timespec inf_end_time; + static struct timespec pre_start_time; + static struct timespec pre_end_time; + static struct timespec post_start_time; + static struct timespec post_end_time; + + std::cout << "[INFO] Inference Thread Starting\n"; + + /*Inference Loop Start*/ + while(1) + { + while(1) + { + /*Gets the Termination request semaphore value. If different then 1 Termination was requested*/ + /*Checks if sem_getvalue is executed wihtout issue*/ + errno = 0; + ret = sem_getvalue(&terminate_req_sem, &inf_sem_check); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to get Semaphore Value: errno=%d\n", errno); + goto err; + } + /*Checks the semaphore value*/ + if (1 != inf_sem_check) + { + goto ai_inf_end; + } + /*Checks if image frame from Capture Thread is ready.*/ + if (inference_start.load()) + { + break; + } + usleep(WAIT_TIME); + } + + /*Gets Pre-process starting time*/ + ret = timespec_get(&pre_start_time, TIME_UTC); + if (0 == ret) + { + fprintf(stderr, "[ERROR] Failed to get Pre-process Start Time\n"); + goto err; + } + + cv::Size size(MODEL_IN_H, MODEL_IN_W); + cv::resize(input_image, input_image, size); + cv::cvtColor(input_image, input_image, cv::COLOR_BGR2RGB); + input_image = hwc2chw(input_image); + input_image.convertTo(input_image, CV_32FC3,1.0 / 255.0, 0); + if (!input_image.isContinuous()) + input_image = input_image.clone(); + + ret = timespec_get(&pre_end_time, TIME_UTC); + if ( 0 == ret) + { + fprintf(stderr, "[ERROR] Failed to Get Pre-process End Time\n"); + goto err; + } + + /*Set Pre-processing output to be inference input. */ + runtime.SetInput(0, input_image.ptr()); + + /*Pre-process Time Result*/ + pre_time = (float)((timedifference_msec(pre_start_time, pre_end_time))); + + /*Gets inference starting time*/ + ret = timespec_get(&start_time, TIME_UTC); + if (0 == ret) + { + fprintf(stderr, "[ERROR] Failed to get Inference Start Time\n"); + goto err; + } + + runtime.Run(); + + /*Gets AI Inference End Time*/ + ret = timespec_get(&inf_end_time, TIME_UTC); + if ( 0 == ret) + { + fprintf(stderr, "[ERROR] Failed to Get Inference End Time\n"); + goto err; + } + /*Inference Time Result*/ + ai_time = (float)((timedifference_msec(start_time, inf_end_time))); + + /*Gets Post-process starting time*/ + ret = timespec_get(&post_start_time, TIME_UTC); + if (0 == ret) + { + fprintf(stderr, "[ERROR] Failed to get Post-process Start Time\n"); + goto err; + } + /*Process to read the DRPAI output data.*/ + ret = get_result(); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to get result from memory.\n"); + goto err; + } + + /*CPU Post-Processing For YOLOv3*/ + R_Post_Proc(drpai_output_buf); + /*Gets Post-process End Time*/ + ret = timespec_get(&post_end_time, TIME_UTC); + if ( 0 == ret) + { + fprintf(stderr, "[ERROR] Failed to Get Post-process End Time\n"); + goto err; + } + /*Post-process Time Result*/ + post_time = (float)((timedifference_msec(post_start_time, post_end_time))); + total_time = pre_time + ai_time + post_time; + inference_start.store(0); + } + /*End of Inference Loop*/ + +/*Error Processing*/ +err: + /*Set Termination Request Semaphore to 0*/ + sem_trywait(&terminate_req_sem); + goto ai_inf_end; +/*AI Thread Termination*/ +ai_inf_end: + /*To terminate the loop in Capture Thread.*/ + printf("AI Inference Thread Terminated\n"); + pthread_exit(NULL); +} + +/***************************************** +* Function Name : R_Capture_Thread +* Description : Executes the V4L2 capture with Capture thread. +* Arguments : cap_pipeline = gstreamer pipeline +* Return value : - +******************************************/ +void *R_Capture_Thread(void *cap_pipeline) +{ + std::string &gstream = *(static_cast(cap_pipeline)); + std::cout << gstream << std::endl; + /*Semaphore Variable*/ + int32_t capture_sem_check = 0; + int8_t ret = 0; + cv::Mat g_frame; + cv::Mat raw_frame; + cv::VideoCapture g_cap; + + printf("[INFO] Capture Thread Starting\n"); + + g_cap.open(gstream, cv::CAP_GSTREAMER); + if (!g_cap.isOpened()) + { + std::cout << "[ERROR] Error opening video stream or camera !\n" + << std::endl; + goto err; + } + + while(1) + { + /*Gets the Termination request semaphore value. If different then 1 Termination was requested*/ + /*Checks if sem_getvalue is executed wihtout issue*/ + errno = 0; + ret = sem_getvalue(&terminate_req_sem, &capture_sem_check); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to get Semaphore Value: errno=%d\n", errno); + goto err; + } + /*Checks the semaphore value*/ + if (1 != capture_sem_check) + { + goto capture_end; + } + + g_cap >> g_frame; + /* Breaking the loop if no video frame is detected */ + if (g_frame.empty()) + { + std::cout << "[INFO] Video ended or corrupted frame !\n"; + goto capture_end; + } + else + { + //cv::resize(g_frame, g_frame, cv::Size(CAM_IMAGE_WIDTH, CAM_IMAGE_HEIGHT)); + if (!inference_start.load()) + { + + input_image = g_frame.clone(); + inference_start.store(1); /* Flag for AI Inference Thread. */ + } + + if (!img_obj_ready.load()) + { + yuyv_image = g_frame.clone(); + img_obj_ready.store(1); /* Flag for Main Thread. */ + } + } + } /*End of Loop*/ + +/*Error Processing*/ +err: + sem_trywait(&terminate_req_sem); + goto capture_end; + +capture_end: + /*To terminate the loop in AI Inference Thread.*/ + inference_start.store(1); + + printf("Capture Thread Terminated\n"); + pthread_exit(NULL); +} + + +/***************************************** + * Function Name : create_output_frame + * Description : create the output frame with space for displaying inference details + * Arguments : cv::Mat frame_g, input frame to be displayed in the background + * Return value : cv::Mat background, final display frame to be written to gstreamer pipeline + *****************************************/ +cv::Mat create_output_frame(cv::Mat frame_g) +{ + /* Create a black background image of size 1080x720 */ + cv::Mat background(DISP_OUTPUT_HEIGHT, DISP_OUTPUT_WIDTH, frame_g.type(), cv::Scalar(0, 0, 0)); + /* Resize the original image to fit within 960x720 */ + cv::Mat resizedImage; + cv::resize(frame_g, resizedImage, cv::Size(DISP_IMAGE_OUTPUT_WIDTH, DISP_IMAGE_OUTPUT_HEIGHT)); + /* Copy the resized image to the left side of the background (0 to 960) */ + cv::Rect roi(cv::Rect(0, 0, resizedImage.cols, resizedImage.rows)); + resizedImage.copyTo(background(roi)); + return background; +} + + +/***************************************** +* Function Name : R_Main_Process +* Description : Runs the main process loop +* Arguments : - +* Return value : 0 if succeeded +* not 0 otherwise +******************************************/ +int8_t R_Main_Process() +{ + /*Main Process Variables*/ + int8_t main_ret = 0; + /*Semaphore Related*/ + int32_t sem_check = 0; + /*Variable for checking return value*/ + int8_t ret = 0; + /*Variable for image buffer id*/ + uint8_t img_buf_id; + /*Variable for detected object time ids*/ + std::map id_time; + std::set unique_ids = {}; + std::map location_history; + long int infer_time_ms; + cv::Mat bgra_image; + + std::set detection_object_set; + std::string objects_available = ""; + std::string objects_not_available = ""; + /* detection count */ + std::map detection_count; + std::string config_file_path = ini_values["config_path"]; + INI_FORMAT config_values = config_read(config_file_path); + float conf = std::stof(config_values["detect"]["conf"]); + std::string get_anchor = config_values["detect"]["anchors"]; + std::string detection_object_string = config_values["detect"]["objects"]; + std::stringstream detection_anchor_ss(get_anchor); + std::string anch_value; + while (std::getline(detection_anchor_ss, anch_value, ',')) + { + double conv_anch_value =std::stod(anch_value); + anchors.push_back(conv_anch_value); + } + std::stringstream detection_object_ss(detection_object_string); + std::string item; + while (std::getline(detection_object_ss, item, ',')) + { + if (count(label_file_map.begin(), label_file_map.end(), item) > 0) + { + detection_object_set.insert(item); + objects_available += item + "\n"; + } + else + { + objects_not_available += item + "\n"; + } + } + std::cout << "[INFO] *******************Detection Parameters*******************" << std::endl; + if (!objects_not_available.empty()) + { + std::cout << "[INFO] Selected objects in config.ini which is not found in the label list\n" + << objects_not_available << "\n"; + } + if (!objects_available.empty()) + { + std::cout << "[INFO] Selected objects to detect\n" + << objects_available << "\n"; + } + else + { + std::cerr << "[ERROR] No matching objects in label list from the config.ini file" << std::endl; + exit(0); + } + /*Display font parameter values*/ + float font_size = .85; + float font_weight = 1; + float font_size_dt = 0.65; + float font_size_bb = 0.5; + float font_weight_bb = 1; + + printf("Main Loop Starts\n"); + while(1) + { + /*Gets the Termination request semaphore value. If different then 1 Termination was requested*/ + errno = 0; + ret = sem_getvalue(&terminate_req_sem, &sem_check); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to get Semaphore Value: errno=%d\n", errno); + goto err; + } + /*Checks the semaphore value*/ + if (1 != sem_check) + { + goto main_proc_end; + } + /* Check img_obj_ready flag which is set in Capture Thread. */ + if (img_obj_ready.load()) + { + /*key : object, value:count*/ + detection_count.clear(); + bgra_image = yuyv_image; + infer_time_ms = total_time; + int total_count = 0; + int items = 0; + + mtx.lock(); + /*filter detection based on confidence score and objects selected*/ + for (detection detect : det) + { + bbox_t dat; + + /*ignore detection based on the threshold from the config.ini file*/ + if (detect.prob < conf) + { + continue; + } + /*get the label from label file map*/ + dat.name = label_file_map[detect.c].c_str(); + + /*check if the detected object is in the list of objects to be detected(from the config.ini file)*/ + if (count(detection_object_set.begin(), detection_object_set.end(), dat.name) <= 0) + { + continue; + } + + /*map for storing the count of detected objects*/ + if (detection_count.count(dat.name) > 0) + { + detection_count[dat.name]++; + } + else + { + detection_count[dat.name] = 1; + } + + dat.X = (int32_t)(detect.bbox.x - (detect.bbox.w / 2)); + dat.Y = (int32_t)(detect.bbox.y - (detect.bbox.h / 2)); + dat.W = (int32_t)detect.bbox.w; + dat.H = (int32_t)detect.bbox.h; + dat.pred = detect.prob * 100.0; + + cv::Size text_size = cv::getTextSize(dat.name, cv::FONT_HERSHEY_SIMPLEX, font_size_bb, 2, 0); + /*adjust the font size based on the detection text size*/ + if (text_size.width > dat.W) + { + font_weight_bb = .75; + font_size_bb = 0.3; + } + else + { + font_size_dt = 0.65; + font_size_bb = 0.55; + } + + cv::Rect rect(dat.X, dat.Y, dat.W, dat.H); + cv::Rect rect_text_box(dat.X, dat.Y - 20, dat.W, 20); + /*draw the rectangle for detected object*/ + cv::rectangle(bgra_image, rect, cv::Scalar(0, 255, 0), 1.5); + /*draw text box for holding the class label*/ + cv::rectangle(bgra_image, rect_text_box, cv::Scalar(0, 255, 0), cv::FILLED); + /*writing class label to the display frame */ + cv::putText(bgra_image, dat.name, cv::Point(dat.X + 5, dat.Y - 8), + cv::FONT_HERSHEY_SIMPLEX, font_size_bb, cv::Scalar(0, 0, 0), font_weight_bb); + } + mtx.unlock(); + bgra_image = create_output_frame(bgra_image); + cv::putText(bgra_image, "Preprocess Time: " + std::to_string(int(pre_time)), cv::Point(1520, 60), + cv::FONT_HERSHEY_DUPLEX, font_size, cv::Scalar(255, 255, 255), font_weight); + cv::putText(bgra_image, "AI Inference Time: " + std::to_string(int(ai_time)), cv::Point(1520, 90), + cv::FONT_HERSHEY_DUPLEX, font_size, cv::Scalar(255, 255, 255), font_weight); + cv::putText(bgra_image, "Postprocess Time: " + std::to_string(int(post_time)), cv::Point(1520, 120), + cv::FONT_HERSHEY_DUPLEX, font_size, cv::Scalar(255, 255, 255), font_weight); + // cv::putText(bgra_image, "Double Click to exit the Application!!", cv::Point(1540, 1020), + // cv::FONT_HERSHEY_SIMPLEX, 0.60, cv::Scalar(255, 255, 255), font_weight, cv::LINE_AA); + for (std::map::iterator it = detection_count.begin(); it != detection_count.end(); ++it) + { + cv::putText(bgra_image, std::string(it->first) + ": " + std::to_string(it->second), + cv::Point(1530, 180 + 30*items), cv::FONT_HERSHEY_DUPLEX, font_size, + cv::Scalar(255, 255, 255), font_weight); + total_count += (int)it->second; + items++; + } + cv::putText(bgra_image, "Total Objects: " + std::to_string(total_count), cv::Point(1520, 150), + cv::FONT_HERSHEY_DUPLEX, font_size, cv::Scalar(255, 255, 255), font_weight); + cv::cvtColor(bgra_image, bgra_image, cv::COLOR_BGR2BGRA); + wayland.commit(bgra_image.data, NULL); + img_obj_ready.store(0); + } + + /*Wait for 1 TICK.*/ + usleep(WAIT_TIME); + } + +/*Error Processing*/ +err: + sem_trywait(&terminate_req_sem); + main_ret = 1; + goto main_proc_end; +/*Main Processing Termination*/ +main_proc_end: + /*To terminate the loop in Capture Thread.*/ + img_obj_ready.store(0); + printf("Main Process Terminated\n"); + return main_ret; +} + + +/***************************************** +* Function Name : get_drpai_start_addr +* Description : Get DRP-AI Memory Area Address via DRP-AI Driver +* Arguments : - +* Return value : drpai start address +******************************************/ +uint64_t get_drpai_start_addr(int drpai_fd) +{ + int fd = 0; + int ret = 0; + drpai_data_t drpai_data; + + errno = 0; + + fd = open("/dev/drpai0", O_RDWR); + if (0 > fd ) + { + LOG(FATAL) << "[ERROR] Failed to open DRP-AI Driver : errno=" << errno; + return (uint32_t)NULL; + } + + /* Get DRP-AI Memory Area Address via DRP-AI Driver */ + ret = ioctl(fd , DRPAI_GET_DRPAI_AREA, &drpai_data); + if (-1 == ret) + { + LOG(FATAL) << "[ERROR] Failed to get DRP-AI Memory Area : errno=" << errno ; + return (uint32_t)NULL; + } + + return drpai_data.address; +} + + +/***************************************** + * Function Name : query_device_status + * Description : function to check USB device is connectod. + * Return value : media_port, media port that device is connectod. + ******************************************/ +std::string query_device_status(std::string device_type) +{ + std::string media_port = ""; + /* Linux command to be executed */ + const char* command = "v4l2-ctl --list-devices"; + /* Open a pipe to the command and execute it */ + FILE* pipe = popen(command, "r"); + if (!pipe) + { + std::cerr << "[ERROR] Unable to open the pipe." << std::endl; + return media_port; + } + /* Read the command output line by line */ + char buffer[128]; + size_t found; + while (fgets(buffer, sizeof(buffer), pipe) != nullptr) + { + std::string response = std::string(buffer); + found = response.find(device_type); + if (found != std::string::npos) + { + fgets(buffer, sizeof(buffer), pipe); + media_port = std::string(buffer); + pclose(pipe); + /* return media port*/ + return media_port; + } + } + pclose(pipe); + /* return media port*/ + return media_port; +} + +/***************************************** +* Function Name : set_drpai_freq +* Description : Function to set the DRP and DRP-AI frequency. +* Arguments : drpai_fd: DRP-AI file descriptor +* Return value : 0 if succeeded +* not 0 otherwise +******************************************/ +int set_drpai_freq(int drpai_fd) +{ + int ret = 0; + uint32_t data; + + errno = 0; + data = drp_max_freq; + ret = ioctl(drpai_fd , DRPAI_SET_DRP_MAX_FREQ, &data); + if (-1 == ret) + { + std::cerr << "[ERROR] Failed to set DRP Max Frequency : errno=" << errno << std::endl; + return -1; + } + + errno = 0; + data = drpai_freq; + ret = ioctl(drpai_fd , DRPAI_SET_DRPAI_FREQ, &data); + if (-1 == ret) + { + std::cerr << "[ERROR] Failed to set DRP-AI Frequency : errno=" << errno << std::endl; + return -1; + } + + return 0; +} + +/***************************************** +* Function Name : init_drpai +* Description : Function to initialize DRP-AI. +* Arguments : drpai_fd: DRP-AI file descriptor +* Return value : If non-zero, DRP-AI memory start address. +* 0 is failure. +******************************************/ +uint64_t init_drpai(int drpai_fd) +{ + int ret = 0; + uint64_t drpai_addr = 0; + + /*Get DRP-AI memory start address*/ + drpai_addr = get_drpai_start_addr(drpai_fd); + if (drpai_addr == 0) + { + return 0; + } + + /*Set DRP-AI frequency*/ + ret = set_drpai_freq(drpai_fd); + if (ret != 0) + { + return 0; + } + + return drpai_addr; +} +/***************************************** + * Function Name : R_Kbhit_Thread + * Description : Executes the Keyboard hit thread (checks if enter key is hit) + * Arguments : threadid = thread identification + * Return value : - + ******************************************/ +void *R_Kbhit_Thread(void *threadid) +{ + /*Semaphore Variable*/ + int32_t kh_sem_check = 0; + /*Variable to store the getchar() value*/ + int32_t c = 0; + /*Variable for checking return value*/ + int8_t ret = 0; + + printf("[INFO] Key Hit Thread Starting\n"); + + printf("************************************************\n"); + printf("* Press ENTER key to quit. *\n"); + printf("************************************************\n"); + + /*Set Standard Input to Non Blocking*/ + errno = 0; + ret = fcntl(0, F_SETFL, O_NONBLOCK); + if (-1 == ret) + { + fprintf(stderr, "[ERROR] Failed to run fctnl(): errno=%d\n", errno); + goto err; + } + + while (1) + { + /*Gets the Termination request semaphore value. If different then 1 Termination was requested*/ + /*Checks if sem_getvalue is executed wihtout issue*/ + errno = 0; + ret = sem_getvalue(&terminate_req_sem, &kh_sem_check); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to get Semaphore Value: errno=%d\n", errno); + goto err; + } + /*Checks the semaphore value*/ + if (1 != kh_sem_check) + { + goto key_hit_end; + } + + c = getchar(); + if (EOF != c) + { + /* When key is pressed. */ + printf("key Detected.\n"); + goto err; + } + else + { + /* When nothing is pressed. */ + usleep(WAIT_TIME); + } + } + +/*Error Processing*/ +err: + /*Set Termination Request Semaphore to 0*/ + sem_trywait(&terminate_req_sem); + goto key_hit_end; + +key_hit_end: + printf("Key Hit Thread Terminated\n"); + pthread_exit(NULL); +} + +int32_t main(int32_t argc, char * argv[]) +{ + int8_t main_proc = 0; + int8_t ret = 0; + int8_t ret_w = 0; + int8_t ret_main = 0; + /*Multithreading Variables*/ + int32_t create_thread_ai = -1; + int32_t create_thread_capture = -1; + int32_t create_thread_exit = -1; + int32_t create_thread_key = -1; + int32_t sem_create = -1; + InOutDataType input_data_type; + bool runtime_status = false; + std::string gstreamer_pipeline; + + /*Disable OpenCV Accelerator due to the use of multithreading */ + unsigned long OCA_list[16]; + for (int i=0; i < 16; i++) OCA_list[i] = 0; + OCA_Activate( &OCA_list[0] ); + + if (argc<3) + { + std::cout << "[ERROR] Please specify Input Option/Source\n"; + std::cout << "[INFO] usage: ./object_counter COCO|animal|vehicle USB\n"; + std::cout << "[INFO] End Application.\n"; + return -1; + + } + + std::string mode = argv[1]; + std::string input_source = argv[2]; + + /* DRP-AI Frequency Setting */ + if (4 <= argc) + drp_max_freq = atoi(argv[3]); + else + drp_max_freq = DRP_MAX_FREQ; + if (5 <= argc) + drpai_freq = atoi(argv[4]); + else + drpai_freq = DRPAI_FREQ; + std::cout<<"\n[INFO] DRP MAX FREQUENCY : "< drpai_fd) + { + std::cerr << "[ERROR] Failed to open DRP-AI Driver : errno=" << errno << std::endl; + return -1; + } + + uint64_t drpaimem_addr_start = 0; + // uint64_t drpaimem_addr_start = 1073741824; + + /*Load Label from label_list file*/ + label_file_map = load_label_file(ini_values["label_path"]); + if (label_file_map.empty()) + { + fprintf(stderr,"[ERROR] Failed to load label file: %s\n", ini_values["label_path"].c_str()); + ret = -1; + goto end_main; + } + uint32_t INF_OUT_SIZE; + + /*Load number of class from label_list file*/ + NUM_CLASS = label_file_map.size(); + INF_OUT_SIZE = (NUM_CLASS + 5) * NUM_BB * num_grids[0] * num_grids[0] + + (NUM_CLASS + 5) * NUM_BB * num_grids[1] * num_grids[1] + + (NUM_CLASS + 5) * NUM_BB * num_grids[2] * num_grids[2]; + + drpai_output_buf =new float[INF_OUT_SIZE]; + + /*Load model_dir structure and its weight to runtime object */ + drpaimem_addr_start = init_drpai(drpai_fd); + + if ((uint32_t)NULL == drpaimem_addr_start) + { + fprintf(stderr, "[ERROR] Failed to get DRP-AI memory area start address.\n"); + goto end_main; + } + + runtime_status = runtime.LoadModel(ini_values["model_path"], drpaimem_addr_start+DRPAI_MEM_OFFSET); + if(!runtime_status) + { + fprintf(stderr, "[ERROR] Failed to load model.\n"); + goto end_main; + } + + /*Get input data */ + input_data_type = runtime.GetInputDataType(0); + if (InOutDataType::FLOAT32 == input_data_type) + { + /*Do nothing*/ + } + else if (InOutDataType::FLOAT16 == input_data_type) + { + fprintf(stderr, "[ERROR] Input data type : FP16.\n"); + /*If your model input data type is FP16, use std::vector for reading input data. */ + goto end_main; + } + else + { + fprintf(stderr, "[ERROR] Input data type : neither FP32 nor FP16.\n"); + goto end_main; + } + + /*Termination Request Semaphore Initialization*/ + /*Initialized value at 1.*/ + sem_create = sem_init(&terminate_req_sem, 0, 1); + if (0 != sem_create) + { + fprintf(stderr, "[ERROR] Failed to Initialize Termination Request Semaphore.\n"); + ret_main = -1; + goto end_threads; + } + /*Create exit Thread*/ + create_thread_exit = pthread_create(&exit_thread, NULL, R_exit_Thread, NULL); + if (0 != create_thread_exit) + { + fprintf(stderr, "[ERROR] Failed to create exit Thread.\n"); + ret_main = -1; + goto end_threads; + } + + /*Create Key Hit Thread*/ + create_thread_key = pthread_create(&kbhit_thread, NULL, R_Kbhit_Thread, NULL); + if (0 != create_thread_key) + { + fprintf(stderr, "[ERROR] Failed to create Key Hit Thread.\n"); + ret_main = -1; + goto end_threads; + } + + /*Create Inference Thread*/ + create_thread_ai = pthread_create(&ai_inf_thread, NULL, R_Inf_Thread, NULL); + if (0 != create_thread_ai) + { + sem_trywait(&terminate_req_sem); + fprintf(stderr, "[ERROR] Failed to create AI Inference Thread.\n"); + ret_main = -1; + goto end_threads; + } + + /*Create Capture Thread*/ + create_thread_capture = pthread_create(&capture_thread, NULL, R_Capture_Thread, (void *) &gstreamer_pipeline); + if (0 != create_thread_capture) + { + sem_trywait(&terminate_req_sem); + fprintf(stderr, "[ERROR] Failed to create Capture Thread.\n"); + ret_main = -1; + goto end_threads; + } + + /*Main Processing*/ + main_proc = R_Main_Process(); + if (0 != main_proc) + { + fprintf(stderr, "[ERROR] Error during Main Process\n"); + ret_main = -1; + } + goto end_threads; + +end_threads: + if(0 == create_thread_capture) + { + ret = wait_join(&capture_thread, CAPTURE_TIMEOUT); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to exit Capture Thread on time.\n"); + ret_main = -1; + } + } + if (0 == create_thread_ai) + { + ret = wait_join(&ai_inf_thread, AI_THREAD_TIMEOUT); + if (0 != ret) + { + fprintf(stderr, "[ERROR] Failed to exit AI Inference Thread on time.\n"); + ret_main = -1; + } + } + + + /*Delete Terminate Request Semaphore.*/ + if (0 == sem_create) + { + sem_destroy(&terminate_req_sem); + } + goto end_main; + +end_main: + printf("Application End\n"); + return ret_main; +} \ No newline at end of file diff --git a/Q08_object_counter/src_v2h/toolchain/runtime.cmake b/Q08_object_counter/src_v2h/toolchain/runtime.cmake new file mode 100755 index 0000000..0ad3cfc --- /dev/null +++ b/Q08_object_counter/src_v2h/toolchain/runtime.cmake @@ -0,0 +1,14 @@ +set(CMAKE_SYSTEM_NAME Linux) +set(CMAKE_SYSTEM_PROCESSOR aarch64) +set(MERA_DRP_RUNTIME ON) +set(DCMAKE_SYSTEM_VERSION 1) + +set(CMAKE_SYSROOT $ENV{SDK}/sysroots/aarch64-poky-linux) +set(CMAKE_FIND_ROOT_PATH $ENV{SDK}/sysroots/aarch64-poky-linux/usr/include/gnu) +set(CMAKE_CXX_COMPILER $ENV{SDK}/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++) +set(CMAKE_C_COMPILER $ENV{SDK}/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc) + +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/Q08_object_counter/src_v2h/utils.cpp b/Q08_object_counter/src_v2h/utils.cpp new file mode 100755 index 0000000..e414593 --- /dev/null +++ b/Q08_object_counter/src_v2h/utils.cpp @@ -0,0 +1,125 @@ +#include "utils.h" + +extern bool doubleClick; + +int devices::detect_mouse_click(void) +{ + /* Gets the full path to the mouse event device. */ + std::string full_path = find_mouse_event(); + /* Converts the full path to a const char pointer. */ + const char* device_path = full_path.c_str(); + /* Opens the mouse event device file. */ + int fd = open(device_path, O_RDONLY); + if (fd < 0) + { + std::cerr << "Error opening input device" << std::endl; + return 1; + } + /* Enters a loop to read mouse events. */ + while (true) + { + struct input_event ev; + /* Reads a mouse event from the device file. */ + ssize_t bytesRead = read(fd, &ev, sizeof(struct input_event)); + /* Checks if the event is a key event and if the key code is for the left or right mouse button. */ + if (bytesRead == sizeof(struct input_event) && ev.type == EV_KEY && + (ev.code == BTN_LEFT || ev.code == BTN_RIGHT)) + { + /* Checks if the key is being pressed (value == 1) or released (value == 0). */ + if (ev.value == 0) + { + if(first_click == true) + { + t2 = std::chrono::high_resolution_clock::now(); + duration = std::chrono::duration_cast(t2 - t1).count(); + first_click = false; + } + if(second_click == true) + { + t4 = std::chrono::high_resolution_clock::now(); + duration_sc = std::chrono::duration_cast(t4 - t3).count(); + second_click = false; + } + /* If the second click has occurred and the duration between clicks is less than 110 milliseconds, + then a double click has been detected. */ + if(duration_cd < tm_clk_difference && duration_cd != 0 && duration < tm_difference + && duration != 0 && duration_sc < tm_difference && duration_sc != 0) + { + std::cout<<"[INFO] Double click detected\n"; + duration = 0; + duration_cd = 0; + duration_sc = 0; + doubleClick = true; + break; + } + } + else if (ev.value == 1) + { + t1 = std::chrono::high_resolution_clock::now(); + if(first_click == false && duration < tm_difference && duration != 0) + { + t3 = std::chrono::high_resolution_clock::now(); + duration = 0; + second_click = true; + } + if(second_click == true) + duration_cd = std::chrono::duration_cast(t3 - t2).count(); + first_click = true; + } + } + } + /* Closes the mouse event device file. */ + close(fd); + /* Returns 0 if no double click was detected. */ + return 0; +} + +std::string devices::find_mouse_event(void) +{ + /* Path to the directory containing all input devices. */ + const char* inputDirPath = "/dev/input/"; + /* Opens the input directory.*/ + DIR* dir = opendir(inputDirPath); + if (!dir) + { + std::cerr << "Error opening input directory" << std::endl; + exit(1); + } + /* Iterates over all entries in the input directory. */ + dirent* entry; + while ((entry = readdir(dir)) != nullptr) + { + /* Skips entries that start with a dot (i.e., hidden files). */ + if (entry->d_name[0] == '.') + { + continue; + } + /* Constructs the full path to the input device. */ + std::string fullPath = inputDirPath + std::string(entry->d_name); + int fd = open(fullPath.c_str(), O_RDONLY | O_NONBLOCK); + if (fd < 0) + { + continue; + } + /* Checks if the input device has relative axis (e.g., a mouse). */ + if (ioctl(fd, EVIOCGBIT(0, EV_MAX), sizeof(long)*EV_MAX)) + { + unsigned long evBits[EV_MAX]; + ioctl(fd, EVIOCGBIT(0, EV_MAX), evBits); + /* Returns the path to the device if the EV_REL bit is set (i.e., the device has relative axis). */ + if (evBits[EV_REL / BITS_PER_LONG] & (1 << (EV_REL % BITS_PER_LONG))) + { + close(fd); + closedir(dir); + return fullPath; + } + } + /* Closes the input device file. */ + close(fd); + } + /* Closes the input directory. */ + closedir(dir); + /* Prints an error message and exits the program if no mouse device is found. */ + std::cerr << "No mouse device found" << std::endl; + exit(1); +} diff --git a/Q08_object_counter/src_v2h/utils.h b/Q08_object_counter/src_v2h/utils.h new file mode 100755 index 0000000..787683b --- /dev/null +++ b/Q08_object_counter/src_v2h/utils.h @@ -0,0 +1,55 @@ +/* Prevents the header file from being included multiple times. */ +#ifndef UTILS +#define UTILS + +/* Includes necessary header files. */ +#include +#include +#include +#include +#include +#include +#include + +/* Defines constants that are used in the code. */ +#ifndef BITS_PER_LONG +#define BITS_PER_LONG (sizeof(long) * 8) +#endif + +#ifndef EV_REL +#define EV_REL 0x02 +#endif + +#ifndef EV_MAX +#define EV_MAX 0x1F +#endif + +/* Declares a class called devices. */ +class devices +{ + private: + /* Initializes variables to track the duration between mouse clicks. */ + std::chrono::high_resolution_clock::time_point t1; + std::chrono::high_resolution_clock::time_point t2; + std::chrono::high_resolution_clock::time_point t3; + std::chrono::high_resolution_clock::time_point t4; + + int duration = 0; + int duration_sc = 0; + int duration_cd = 0; + + const int tm_difference = 110; + const int tm_clk_difference = 190; + + bool last_click = false; + bool first_click = false; + bool second_click = false; + + /* Finds the path to the mouse event device. */ + std::string find_mouse_event(void); + public: + /* Detects mouse clicks. */ + int detect_mouse_click(void); +}; + +#endif \ No newline at end of file diff --git a/Q08_object_counter/src_v2h/wayland.cpp b/Q08_object_counter/src_v2h/wayland.cpp new file mode 100755 index 0000000..b34f20b --- /dev/null +++ b/Q08_object_counter/src_v2h/wayland.cpp @@ -0,0 +1,494 @@ +/*********************************************************************************************************************** + * Copyright (C) 2023 Renesas Electronics Corporation. All rights reserved. + ***********************************************************************************************************************/ +/*********************************************************************************************************************** + * File Name : wayland.cpp + * Version : 0.90 + * Description : RZ/V2H DRP-AI Sample Application for Megvii-Base Detection YOLOX with MIPI/USB Camera + ***********************************************************************************************************************/ + +/***************************************** + * Includes + ******************************************/ +// #include "define.h" +#include "wayland.h" +#include +#include +#include +#include +#include +#include +#include +#include + + +struct WaylandGlobals { + struct wl_compositor* compositor; + struct wl_shell* shell; +}; + +/***************************************** + * Function Name : registry_global + * Description : wl_registry_listener callback + * wayland func bind. + * Arguments : data = The third argument of wl_registry_add_listener() is notified. + * regisry = The first argument of wl_registry_add_listener() is notified. + * name = global object ID is notified. + * interface = interface name is notifed. + * version = interface version is notified. + * Return value : - + ******************************************/ +static void registry_global(void *data, + struct wl_registry *registry, uint32_t id, + const char *interface, uint32_t version) +{ + struct WaylandGlobals* globals = (struct WaylandGlobals*)data; + if (strcmp(interface, "wl_compositor") == 0) { + globals->compositor = (struct wl_compositor*)wl_registry_bind(registry, id, &wl_compositor_interface, 1); + } + else if (strcmp(interface, "wl_shell") == 0) { + globals->shell = (struct wl_shell*)wl_registry_bind(registry, id, &wl_shell_interface, 1); + } +} + +/* registry callback for listener */ +static const struct wl_registry_listener registry_listener = { registry_global, NULL }; + +/***************************************** + * Function Name : shell_surface_ping + * Description : wl_shell_surface_listener callback + * compositer check hungup + * Arguments : data = The third argument of wl_shell_surface_add_listener() is notified. + * shell_surface = The first argument of wl_shell_surface_add_listener() is notified. + * serial = Identification ID is notified. + * Return value : - + ******************************************/ +static void shell_surface_ping(void *data, + struct wl_shell_surface *shell_surface, + uint32_t serial) +{ + wl_shell_surface_pong(shell_surface, serial); +} + +static const struct wl_shell_surface_listener shell_surface_listener = +{ + .ping = shell_surface_ping, +}; + +Wayland::Wayland() +{ +} + +Wayland::~Wayland() +{ +} + +/***************************************** + * Function Name : LoadShader + * Description : Return the loaded and compiled shader + * Arguments : type + * shaderSrc + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +GLuint Wayland::LoadShader(GLenum type, const char* shaderSrc) +{ + GLuint shader = glCreateShader(type); + assert(shader); + + glShaderSource(shader, 1, &shaderSrc, NULL); + glCompileShader(shader); + + GLint compiled; + glGetShaderiv(shader, GL_COMPILE_STATUS, &compiled); + assert(compiled); + + return shader; +} + +/***************************************** + * Function Name : initProgramObject + * Description : Initialize the shaders and return the program object + * Arguments : pShader + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +GLuint Wayland::initProgramObject(SShader* pShader) +{ + const char* vshader = R"( + attribute vec4 position; + attribute vec2 texcoord; + varying vec2 texcoordVarying; + void main() { + gl_Position = position; + texcoordVarying = texcoord; + } + )"; + + const char* fshader = R"( + precision mediump float; + uniform sampler2D texture; + varying vec2 texcoordVarying; + void main() { + highp float r = texture2D(texture, texcoordVarying).b; + highp float g = texture2D(texture, texcoordVarying).g; + highp float b = texture2D(texture, texcoordVarying).r; + highp float a = texture2D(texture, texcoordVarying).a; + gl_FragColor = vec4(r,g,b,a); + } + + )"; + + GLuint vertexShader = LoadShader(GL_VERTEX_SHADER, vshader); + GLuint fragmentShader = LoadShader(GL_FRAGMENT_SHADER, fshader); + + GLuint programObject = glCreateProgram(); + assert(programObject); + + glAttachShader(programObject, vertexShader); + glAttachShader(programObject, fragmentShader); + + glLinkProgram(programObject); + + GLint linked; + glGetProgramiv(programObject, GL_LINK_STATUS, &linked); + assert(linked); + + glDeleteShader(fragmentShader); + glDeleteShader(vertexShader); + + pShader->unProgram = programObject; + pShader->nAttrPos = glGetAttribLocation(pShader->unProgram, "position"); + pShader->nAttrColor = glGetAttribLocation(pShader->unProgram, "texcoord"); + return programObject; +} + +/***************************************** + * Function Name : initEGLDisplay + * Description : Configure EGL and return necessary resources + * Arguments : nativeDisplay + * nativeWindow + * eglDisplay + * eglSurface + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +static int8_t initEGLDisplay(EGLNativeDisplayType nativeDisplay, EGLNativeWindowType nativeWindow, EGLDisplay* eglDisplay, EGLSurface* eglSurface) +{ +// int8_t ret = 0; + + EGLint number_of_config; + EGLint config_attribs[] = { + EGL_SURFACE_TYPE, EGL_WINDOW_BIT, + EGL_RED_SIZE, 8, + EGL_GREEN_SIZE, 8, + EGL_BLUE_SIZE, 8, + EGL_ALPHA_SIZE, 8, + EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT, + EGL_NONE + }; + + static const EGLint context_attribs[] = { + EGL_CONTEXT_CLIENT_VERSION, 2, + EGL_NONE + }; + + *eglDisplay = eglGetDisplay(nativeDisplay); + if (*eglDisplay == EGL_NO_DISPLAY) + { + return -1; + } + + EGLBoolean initialized = eglInitialize(*eglDisplay, NULL, NULL); + if (initialized != EGL_TRUE) + { + return -1; + } + + EGLConfig configs[1]; + + EGLBoolean config = eglChooseConfig(*eglDisplay, config_attribs, configs, 1, &number_of_config); + if (config != EGL_TRUE) + { + return -1; + } + + EGLContext eglContext = eglCreateContext(*eglDisplay, configs[0], EGL_NO_CONTEXT, context_attribs); + + *eglSurface = eglCreateWindowSurface(*eglDisplay, configs[0], nativeWindow, NULL); + if (*eglSurface == EGL_NO_SURFACE) + { + return -1; + } + + EGLBoolean makeCurrent = eglMakeCurrent(*eglDisplay, *eglSurface, *eglSurface, eglContext); + if (makeCurrent != EGL_TRUE) + { + return -1; + } + return 0; +} + + +/***************************************** + * Function Name : initWaylandDisplay + * Description : Connect to the Wayland display and return the display and the surface + * Arguments : wlDisplay + * wlSurface + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +static int8_t initWaylandDisplay(struct wl_display** wlDisplay, struct wl_surface** wlSurface) +{ +// int8_t ret = 0; + struct WaylandGlobals globals = { 0 }; + + *wlDisplay = wl_display_connect(NULL); + if(*wlDisplay == NULL) + { + return -1; + } + + struct wl_registry* registry = wl_display_get_registry(*wlDisplay); + wl_registry_add_listener(registry, ®istry_listener, (void*)&globals); + + wl_display_dispatch(*wlDisplay); + wl_display_roundtrip(*wlDisplay); + if (globals.compositor == NULL || globals.shell == NULL) + { + return -1; + } + + *wlSurface = wl_compositor_create_surface(globals.compositor); + if (*wlSurface == NULL) + { + return -1; + } + + struct wl_shell_surface* shellSurface = wl_shell_get_shell_surface(globals.shell, *wlSurface); + wl_shell_surface_set_toplevel(shellSurface); + return 0; +} + +/***************************************** + * Function Name : initWindow + * Description : Connect Wayland and make EGL + * Arguments : width + * height + * wlDisplay + * eglDisplay + * eglSurface + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +static int8_t initWindow(GLint width, GLint height, struct wl_display** wlDisplay, EGLDisplay* eglDisplay, EGLSurface* eglSurface) +{ + int8_t ret = 0; + struct wl_surface* wlSurface; + ret = initWaylandDisplay(wlDisplay, &wlSurface); + if (ret != 0) + { + return -1; + } + + struct wl_egl_window* wlEglWindow = wl_egl_window_create(wlSurface, width, height); + if (wlEglWindow == NULL) + { + return -1; + } + + ret = initEGLDisplay((EGLNativeDisplayType)*wlDisplay, (EGLNativeWindowType)wlEglWindow, eglDisplay, eglSurface); + if (ret != 0) + { + return -1; + } + return 0; +} + +/***************************************** + * Function Name : init + * Description : wayland client init + * create buffer. + * Arguments : idx = index of the display buffer + * w = width + * c = color + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +uint8_t Wayland::init(uint32_t idx, uint32_t w, uint32_t h, uint32_t c, bool overlay) +{ + int8_t ret = 0; + img_w = w; + img_h = h; + img_c = c; + img_overlay = overlay; + + // Connect Wayland and make EGL + ret = initWindow(w, h, &display, &eglDisplay, &eglSurface); + if (ret != 0) + { + return -1; + } + + //Initialize the shaders and return the program object + GLuint programObject = initProgramObject(&sShader); + if (programObject == 0) + { + return -1; + } + + // Apply program object + glUseProgram(sShader.unProgram); + glGenTextures(2, textures); + + glEnableVertexAttribArray(sShader.nAttrPos); + glEnableVertexAttribArray(sShader.nAttrColor); + + // enable Alpha Blending + if (img_overlay == true){ + glEnable(GL_BLEND); + glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); + } + + glUniform1i(glGetUniformLocation(sShader.unProgram, "texture"), 0); + + return 0; +} + +/***************************************** + * Function Name : exit + * Description : Exit Wayland + * Arguments : - + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +uint8_t Wayland::exit() +{ + SShader* pShader = &sShader; + if (pShader) { + glDeleteProgram(pShader->unProgram); + pShader->unProgram = 0; + pShader->nAttrPos = -1; + pShader->nAttrColor = -1; + } + wl_display_disconnect(display); + return 0; +} + + +/***************************************** + * Function Name : render + * Description : + * Arguments : pShader + * texID + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +uint8_t Wayland::render(SShader* pShader, GLuint texID) +{ + const float vertices[] = { + -1.0f, 1.0f, 0.0f, + -1.0f, -1.0f, 0.0f, + 1.0f, 1.0f, 0.0f, + 1.0f, -1.0f, 0.0f + }; + + const float texcoords[] = { + 0.0f, 0.0f, + 0.0f, 1.0f, + 1.0f, 0.0f, + 1.0f, 1.0f }; + + + glVertexAttribPointer(pShader->nAttrColor, 2, GL_FLOAT, GL_FALSE, 0, texcoords); + glVertexAttribPointer(pShader->nAttrPos, 3, GL_FLOAT, GL_FALSE, 0, vertices); + + // draw texture + glActiveTexture(GL_TEXTURE0); + glBindTexture(GL_TEXTURE_2D, texID); + //glUniform1i(uniID, texID); + glDrawArrays(GL_TRIANGLE_STRIP, 0, 4); + return 0; +} + + +/***************************************** + * Function Name : setupTexture + * Description : Bind Texture + * Arguments : texID + * src_pixels + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +uint8_t Wayland::setupTexture(GLuint texID, uint8_t* src_pixels) +{ + glPixelStorei(GL_UNPACK_ALIGNMENT, 1); + glBindTexture(GL_TEXTURE_2D, texID); + glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, img_w, img_h, 0, GL_RGBA, GL_UNSIGNED_BYTE, src_pixels); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); + return 0; +} + + +/***************************************** + * Function Name : commit + * Description : Commit to update the display image + * Arguments : buf_id = buffer id + * Return value : 0 if Success + * not 0 otherwise + ******************************************/ +uint8_t Wayland::commit(uint8_t* cam_buffer, uint8_t* ol_buffer) +{ + uint8_t ret = 0; +#ifdef DEBUG_TIME_FLG + using namespace std; + chrono::system_clock::time_point start, end; + double time = 0; + start = chrono::system_clock::now(); +#endif // DEBUG_TIME_FLG + + // setup texture + setupTexture(textures[0], cam_buffer); + if (ol_buffer != NULL && img_overlay == true) { + setupTexture(textures[1], ol_buffer); + } +#ifdef DEBUG_TIME_FLG + end = chrono::system_clock::now(); + time = static_cast(chrono::duration_cast(end - start).count() / 1000.0); + printf("Setup Image Time : %lf[ms]\n", time); +#endif // DEBUG_TIME_FLG + + // clear + glClearColor(0.0f, 0.0f, 0.0f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); + +#ifdef DEBUG_TIME_FLG + start = chrono::system_clock::now(); +#endif // DEBUG_TIME_FLG + + // render + render(&sShader, textures[0]); + if (ol_buffer != NULL && img_overlay == true) { + render(&sShader, textures[1]); + } +#ifdef DEBUG_TIME_FLG + end = chrono::system_clock::now(); + time = static_cast(chrono::duration_cast(end - start).count() / 1000.0); + printf("Specifies Render Time : %lf[ms]\n", time); + start = chrono::system_clock::now(); +#endif // DEBUG_TIME_FLG + + eglSwapBuffers(eglDisplay, eglSurface); + +#ifdef DEBUG_TIME_FLG + end = chrono::system_clock::now(); + time = static_cast(chrono::duration_cast(end - start).count() / 1000.0); + printf("Update Frame Time : %lf[ms]\n", time); +#endif // DEBUG_TIME_FLG + + return ret; +} + diff --git a/Q08_object_counter/src_v2h/wayland.h b/Q08_object_counter/src_v2h/wayland.h new file mode 100755 index 0000000..ec3ef13 --- /dev/null +++ b/Q08_object_counter/src_v2h/wayland.h @@ -0,0 +1,62 @@ +/*********************************************************************************************************************** +* Copyright (C) 2023 Renesas Electronics Corporation. All rights reserved. +***********************************************************************************************************************/ +/*********************************************************************************************************************** +* File Name : wayland.h +* Version : 0.90 +* Description : RZ/V2H DRP-AI Sample Application for Megvii-Base Detection YOLOX with MIPI/USB Camera +***********************************************************************************************************************/ + +#ifndef WAYLAND_H +#define WAYLAND_H + + +// #include "define.h" +#include +#include +#include +#include + +class Wayland +{ + /* structure of Shader settings */ + typedef struct _SShader { + GLuint unProgram; + GLint nAttrPos; + GLint nAttrColor; + } SShader; + + public: + Wayland(); + ~Wayland(); + + uint8_t init(uint32_t idx, uint32_t w, uint32_t h, uint32_t c, bool overlay = false); + uint8_t exit(); + uint8_t commit(uint8_t* cam_buffer, uint8_t* ol_buffer); + + struct wl_compositor *compositor = NULL; + struct wl_shm *shm = NULL; + struct wl_shell *shell = NULL; + + private: + uint32_t img_h; + uint32_t img_w; + uint32_t img_c; + bool img_overlay; + + struct wl_display *display = NULL; + struct wl_surface *surface; + struct wl_shell_surface *shell_surface; + struct wl_registry *registry = NULL; + EGLDisplay eglDisplay; + EGLSurface eglSurface; + SShader sShader; + GLuint textures[2]; + + GLuint LoadShader(GLenum type, const char* shaderSrc); + GLuint initProgramObject(SShader* pShader); + uint8_t render(SShader* pShader, GLuint texID); + uint8_t setupTexture(GLuint texID, uint8_t* src_pixels); +}; + +#endif