FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation

The official pytorch implementation of FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation (Yair Kittenplon, Yonina C. Eldar, Dan Raviv, CVPR 2021).

Citation

If you find this work useful, please cite our paper:

@article{Kittenplon2020FlowStep3DMU,
  title={FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation},
  author={Yair Kittenplon and Yonina C. Eldar and Dan Raviv},
  journal={ArXiv},
  year={2020},
  volume={abs/2011.10147}
}

Installation

This project implemented and tested on

• PyTorchLightning 0.9.0
• PyTorch 0.4.1
• Python3.6
• CUDA versions 10.1.243, 10.2.89
• Ubuntu 18.04

Create a virtual env and install requirements

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt  

Install PointNet2 CPP lib:

cd lib/pointnet2
python3 setup.py install
cd ../../

Datasets

We follow HPLFlowNet preprocessing methods:

Datasets path definition

SAVE_PATH=data/datasets
mkdir ${SAVE_PATH}

• for another SAVE_PATH, please change "train_data_root" in all configs/* yaml files accordingly

FlyingThings3D

Download and unzip the "Disparity", "Disparity Occlusions", "Disparity change", "Optical flow", "Flow Occlusions" for DispNet/FlowNet2.0 dataset subsets from the FlyingThings3D website (we used the paths from this file, now they added torrent downloads) . They will be upzipped into the same directory, RAW_DATA_PATH. Then run the following script for 3D reconstruction:

python data/preprocess/process_flyingthings3d_subset.py --raw_data_path RAW_DATA_PATH --save_path ${SAVE_PATH}/FlyingThings3D_subset_processed_35m --only_save_near_pts

KITTI

Download and unzip KITTI Scene Flow Evaluation 2015 to directory RAW_DATA_PATH. Run the following script for 3D reconstruction:

python data/preprocess/process_kitti.py RAW_DATA_PATH ${SAVE_PATH}/KITTI_processed_occ_final

Pretrained models

In order to download pre-trained models, please run:

bash scripts/download_models.sh

Training

Self-supervised training:

python run.py -c configs/train/flowstep3d_self_pre.yaml
# set pre_trained_weights_checkpoint in flowstep3d_self_ft.yaml according to checkpoint.
python run.py -c configs/train/flowstep3d_self_ft.yaml

Fully-supervised training:

python run.py -c configs/train/flowstep3d_sv_pre.yaml
# set pre_trained_weights_checkpoint in flowstep3d_sv_ft.yaml according to checkpoint.
python run.py -c configs/train/flowstep3d_sv_ft.yaml

Neptune logger

To activate logger, please create a user at neptune.ai and config "logging_params" at all config/train/*.yaml files accordingly.

Inference

Self-supervised model:

python run.py -c configs/test/flowstep3d_self.yaml

Fully-supervised model:

python run.py -c configs/test/flowstep3d_sv.yaml

Credits

In this project we use parts of the official implementations of the following libraries and repositories:

• PyTorchLightning
• Pointnet2_PyTorch
• flownet3d_Pytorch
• HPLFlowNet
• FLOT