Skip to content

Latest commit

 

History

History
146 lines (134 loc) · 12.3 KB

README.md

File metadata and controls

146 lines (134 loc) · 12.3 KB
Raw

NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search [slides][arXiv]

@inproceedings{
  lu2020nsganetv2,
  title={{NSGANetV2}: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search},
  author={Zhichao Lu and Kalyanmoy Deb and Erik Goodman and Wolfgang Banzhaf and Vishnu Naresh Boddeti},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2020}
}

Overview

NSGANetV2 is an efficient NAS algorithm for generating task-specific models that are competitive under multiple competing objectives. It comprises of two surrogates, one at the architecture level to improve sample efficiency and one at the weights level, through a supernet, to improve gradient descent training efficiency.

Datasets

Download the datasets from the links embedded in the names. Datasets with * can be automatically downloaded.

Dataset Type Train Size Test Size #Classes
ImageNet multi-class 1,281,167 50,000 1,000
CINIC-10 180,000 9,000 10
CIFAR-10* 50,000 10,000 10
CIFAR-100* 50,000 10,000 10
STL-10* 5,000 8,000 10
FGVC Aircraft* fine-grained 6,667 3,333 100
DTD 3,760 1,880 47
Oxford-IIIT Pets 3,680 3,369 37
Oxford Flowers102 2,040 6,149 102

How to evalute NSGANetV2 models

Download the models (net.config) and weights (net.init) from [Google Drive] or [Baidu Yun](提取码：4isq).

""" NSGANetV2 pretrained models
Syntax: python validation.py \
    --dataset [imagenet/cifar10/...] --data /path/to/data \
    --model /path/to/model/config/file --pretrained /path/to/model/weights 
"""
ImageNet CIFAR-10 CINIC10
FLOPs@225: [Google Drive]
FLOPs@312: [Google Drive]
FLOPs@400: [Google Drive]
FLOPs@593: [Google Drive]		 FLOPs@232: [Google Drive]
FLOPs@291: [Google Drive]
FLOPs@392: [Google Drive]
FLOPs@468: [Google Drive]		 FLOPs@317: [Google Drive]
FLOPs@411: [Google Drive]
FLOPs@501: [Google Drive]
FLOPs@710: [Google Drive]
Flowers102 Aircraft Oxford-IIIT Pets
FLOPs@151: [Google Drive]
FLOPs@218: [Google Drive]
FLOPs@249: [Google Drive]
FLOPs@317: [Google Drive]		 FLOPs@176: [Google Drive]
FLOPs@271: [Google Drive]
FLOPs@331: [Google Drive]
FLOPs@502: [Google Drive]		 FLOPs@137: [Google Drive]
FLOPs@189: [Google Drive]
FLOPs@284: [Google Drive]
FLOPs@391: [Google Drive]
CIFAR-100 DTD STL-10
FLOPs@261: [Google Drive]
FLOPs@398: [Google Drive]
FLOPs@492: [Google Drive]
FLOPs@796: [Google Drive]		 FLOPs@123: [Google Drive]
FLOPs@164: [Google Drive]
FLOPs@202: [Google Drive]
FLOPs@213: [Google Drive]		 FLOPs@240: [Google Drive]
FLOPs@303: [Google Drive]
FLOPs@436: [Google Drive]
FLOPs@573: [Google Drive]

How to use MSuNAS to search

""" Bi-objective search
Syntax: python msunas.py \
    --dataset [imagenet/cifar10/...] --data /path/to/dataset/images \
    --save search-xxx \ # dir to save search results
    --sec_obj [params/flops/cpu] \ # objective (in addition to top-1 acc)
    --n_gpus 8 \ # number of available gpus
    --supernet_path /path/to/supernet/weights \
    --vld_size [10000/5000/...] \ # number of subset images from training set to guide search
    --n_epochs [0/5]
"""
  • Download the pre-trained (on ImageNet) supernet from here.
  • It supports searching for FLOPs, Params, and Latency as the second objective.
    • To optimize latency on your own device, you need to first construct a look-up-table for your own device, like this.
  • Choose an appropriate --vld_size to guide the search, e.g. 10,000 for ImageNet, 5,000 for CIFAR-10/100.
  • Set --n_epochs to 0 for ImageNet and 5 for all other datasets.
  • See here for some examples.
  • Output file structure:
    • Every architecture sampled during search has net_x.subnet and net_x.stats stored in the corresponding iteration dir.
    • A stats file is generated by the end of each iteration, iter_x.stats; it stores every architectures evaluated so far in ["archive"], and iteration-wise statistics, e.g. hypervolume in ["hv"], accuracy predictor related in ["surrogate"].
    • In case any architectures failed to evaluate during search, you may re-visit them in failed sub-dir under experiment dir.
ImageNet CIFAR-10

How to choose architectures

Once the search is completed, you can choose suitable architectures by:

  • You have preferences, e.g. architectures with xx.x% top-1 acc. and xxxM FLOPs, etc.
""" Find architectures with objectives close to your preferences
Syntax: python post_search.py \
    -n 3 \ # number of desired architectures you want, the most accurate archecture will always be selected 
    --save search-imagenet/final \ # path to the dir to store the selected architectures 
    --expr search-imagenet/iter_30.stats \ # path to last iteration stats file in experiment dir 
    --prefer top1#80+flops#150 \ # your preferences, i.e. you want an architecture with 80% top-1 acc. and 150M FLOPs 
    --supernet_path /path/to/imagenet/supernet/weights \
"""
  • If you do not have preferences, pass None to argument --prefer, architectures will then be selected based on trade-offs.
  • All selected architectures should have three files created:
    • net.subnet: use to sample the architecture from the supernet
    • net.config: configuration file that defines the full architectural components
    • net.inherited: the inherited weights from supernet

How to validate architectures

To realize the full potential of the searched architectures, we further fine-tune from the inherited weights. Assuming that you have both net.config and net.inherited files.

""" Fine-tune on ImageNet from inherited weights
Syntax: sh scripts/distributed_train.sh 8 \ # of available gpus
    /path/to/imagenet/data/ \
    --model [nsganetv2_s/nsganetv2_m/...] \ # just for naming the output dir
    --model-config /path/to/model/.config/file \
    --initial-checkpoint /path/to/model/.inherited/file \
    --img-size [192, ..., 224, ..., 256] \ # image resolution, check "r" in net.subnet
    -b 128 --sched step --epochs 450 --decay-epochs 2.4 --decay-rate .97 \
    --opt rmsproptf --opt-eps .001 -j 6 --warmup-lr 1e-6 \
    --weight-decay 1e-5 --drop 0.2 --drop-path 0.2 --model-ema --model-ema-decay 0.9999 \
    --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp --lr .024 \
    --teacher /path/to/supernet/weights \
"""
  • Adjust learning rate as (batch_size_per_gpu * #GPUs / 256) * 0.006 depending on your system config.
""" Fine-tune on CIFAR-10 from inherited weights
Syntax: python train_cifar.py \
    --data /path/to/CIFAR-10/data/ \
    --model [nsganetv2_s/nsganetv2_m/...] \ # just for naming the output dir
    --model-config /path/to/model/.config/file \
    --img-size [192, ..., 224, ..., 256] \ # image resolution, check "r" in net.subnet
    --drop 0.2 --drop-path 0.2 \
    --cutout --autoaugment --save
"""

More Use Cases (coming soon)

  • With a different supernet (search space).
  • NASBench 101/201.
  • Architecture visualization.

Requirements

  • Python 3.7
  • Cython 0.29 (optional; makes non_dominated_sorting faster in pymoo)
  • PyTorch 1.5.1
  • pymoo 0.4.1
  • torchprofile 0.0.1 (for FLOPs calculation)
  • OnceForAll 0.0.4 (lower level supernet)
  • timm 0.1.30
  • pySOT 0.2.3 (RBF surrogate model)
  • pydacefit 1.0.1 (GP surrogate model)