Code for generating the features

[atanas1054]

Hello,

Thank you very much for providing this repo. Do you plan to release any code for the feature generation (person appearance/keypoints, scene semantics, etc.)? I am interested in running this model on another dataset.

[JunweiLiang]

Hi,
I'm planning to release the inference code which includes the feature extraction. But you can start with using the individual pre-trained models that we use:
scene semantic segmentation: deeplabv3_xception_ade20k
person keypoints: https://github.com/MVIG-SJTU/AlphaPose
person appearance: My implementation of Faster RCNN. But you can use any other popular implementations, which should have similar results.

[JunweiLiang]

Hi,
We have updated our object detection and tracking code: https://github.com/JunweiLiang/Object_Detection_Tracking. You can use that code base to extract object bounding boxes and appearance features. You still need to use the aforementioned scene semantic segmentation model/person keypoint model and put them into the right format for training on a new dataset. We may not release this part of the code before CVPR.

[JunweiLiang]

Added notes for the prepared data here.

[jagmonroy]

Hi! I have a doubt about segmentation process.

I took zara2 and I did the segmentation with the deeplab model you comment above. The segmentations are close to yours according to normalized hamming distance, but are not exactly the same. I would like to know what I did diffrent. In the next lines I explain the process.

The original frame size is (720, 576). When the "run" function of deeplab is used the frame is resized in the next way (I did not change any parameter of deeplab):

width, height = image.size
resize_ratio = 1.0 * self.INPUT_SIZE / max(width, height) # self.INPUT_SIZE = 513
target_size = (int(resize_ratio * width), int(resize_ratio * height))

I saved the segmentations in a folder with that size and after that I loaded one by one to resize to (64, 51) (grid size). Something like this:

shape = (64, 51)
im = cv2.resize(frame.astype('uint8'), shape, interpolation = cv2.INTER_NEAREST)

I think that the diffrence could be for the interpolation method, but I'm not shure.

[JunweiLiang]

I have reproduced the feature extraction code here.

[jagmonroy]

thanks!