ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Alexandre Boulch¹    Corentin Sautier^1,2    Björn Michele^1,3    Gilles Puy¹    Renaud Marlet^1,2

¹ Valeo.ai, Paris, France ² LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

³ CNRS, IRISA, Univ. Bretagne Sud, Vannes, France

ALSO has been accepted at CVPR 2023

This is the reference PyTorch implementation for training and testing self-supervision for automotive lidar point cloud using the method described in our paper ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Overview

• Citation
• Dependencies
• Installation
• Semantic segmentation
• Pre-trained models
• Performances
• Acknowledgments

---

Citation

Please acknowledge our work in your publications:

@InProceedings{ALSO,
author = {Alexandre Boulch and Corentin Sautier and Björn Michele and Gilles Puy and Renaud Marlet},
title = {{ALSO}: Automotive Lidar Self-supervision by Occupancy estimation},
booktitle = {International Conference on Computer Vision and Pattern Recognition (CVPR)},
year = 2023,
}

Dependencies

We give the library version(s), we used for experiments.

• Hydra (1.2.0)
• Pytorch-geometric (2.1.0)
• Pytorch-Lightning (1.7.2)

Semantic segmentation specific libraries

• torchsparse (1.4.0)

Detection specific libraries

• spconv (2.1.21)
• openpcdet (0.6.0)

Installation

At root of the folder run:

pip install -ve .

Then all scripts can be run from also_selfsup folder:

cd also_selfsup

Datasets

ONCE

We follow intruction from ONCE dataset to generate the information files of the raw_small dataset.

Semantic segmentation

Pre-training

python train_selfsupervised.py cfg=nuscenes

for a pre-training on nuScenes.

for other datasets, replace cfg=nuscenes by the desired dataset

• nuScenes: cfg=nuscenes
• SemanticKITTI: cfg=semantickitti
• SemanticPOSS: To be added
• LivoxSimuDataset: To be added

Parameters

We use Hydra for easy configuration of trainings. The pretraining parameters are set for pre-training with a NVidia-V100 16Gb. In order to pre-train on other configurations, one can modify parameters directly in the command line, e.g., setting the batch size to 8:

python train_selfsupervised.py ... cfg.training.batch_size=8

or the backbone to SPVCNN:

python train_selfsupervised.py ... cfg.network.backbone=SPVCNN

Downstream

From scratch

# 100%
python also_selfsup/train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=30 cfg.downstream.val_interval=5 cfg.downstream.skip_ratio=1

# 50%
python also_selfsup/train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=50 cfg.downstream.val_interval=5 cfg.downstream.skip_ratio=2

# 10%
python also_selfsup/train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=100 cfg.downstream.val_interval=10 cfg.downstream.skip_ratio=10

# 1%
python also_selfsup/train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=500 cfg.downstream.val_interval=50 cfg.downstream.skip_ratio=100

# 0.1%
python also_selfsup/train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=1000 cfg.downstream.val_interval=100 cfg.downstream.skip_ratio=1000

With pretrained models

First, convert the pre-trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, run downstream training

python also_selfsup/train_downstream_semseg.py cfg=nuscenes cfg.downstream.checkpoint_dir='path_to_checkpoint_directory' cfg.downstream.checkpoint_name='pretrained_backbone_XXX.ckpt'

Evaluation of downstream models

First, convert the trained model to be loaded

python convert_models.py --downstream --ckpt path_to_downstream_checkpoint

Second, run the evaluation script

python eval.py --split val --config path_to_downstream_model/config.yaml --ckpt path_to_downstream_checkpoint/trained_model_XXX.ckpt

Detection

Pre-training

Similar to semantic segmentation, pre-training for detection can simply be run using:

python train_selfsupervised.py cfg=kitti3d_second
python train_selfsupervised.py cfg=nuscenes_second_kitti

Downstream

First, convert the trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, use the pre-trained model with the official OpenPCDet code:

cd path_to_OpenPCDet/tools/
python train.py --cfg_file cfgs/kitti_models/second.yaml --extra_tag --pretrained_model path_to_pretrained_model/pretrained_backbone_XXX.ckpt

Pre-trained models

Task	Pre-training dataset	Model	Link	Notes
Semantic segmentation	nuScenes	MinkUNet34	download
		SPVCNN	download
	SemanticKITTI	MinkUNet18SC	download
		SPVCNN	download
Detection	KITTI3D	SECOND/PVRCNN	download	trained for KITTI3D
	nuScenes	SECOND/PVRCNN	download	trained for KITTI3D
	KITTI360	SECOND/PVRCNN	download	trained for KITTI3D
	ONCE Small	SECOND/PVRCNN	download	trained for ONCE

Here are links to pre-trained models

Acknowledgments

This project would not have been possible without many community resources and repositories. Among them:

• PointContrast
• SLidR
• SegContrast
• SPVNAS
• POCO

Please, consider acknowleding these projects.