From a1c59a4d689bbfab1a42e18012f09bf4ea7b7a5a Mon Sep 17 00:00:00 2001 From: Nikolay Falaleev Date: Fri, 6 Oct 2017 01:29:46 +0300 Subject: [PATCH] Update README.md docs: Add project description --- README.md | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 84 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 66d0b42..0f1c134 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,88 @@ # Semantic Segmentation -### Introduction -In this project, you'll label the pixels of a road in images using a Fully Convolutional Network (FCN). + +### Udacity Self-Driving Car Engineer Nanodegree. Project: Vehicle Detection and Tracking + +This Project is the twelfth task of the Udacity Self-Driving Car Nanodegree program. The main goal of the project is to train an artificial neural network for semantic segmentation of a video from a front-facing camera on a car in order to mark road pixels using Tensorflow. + +## Results + +KITTI Road segmentation (main task of the project): +![title.gif animation](readme_img/title.gif) + +Cityscapes multiclass segmentation (optional task): +![title_cityscapes.gif animation](readme_img/title_cityscapes.gif) + +## Content of this repo + +- `Segmentation.ipynb` - Jupyter notebook with the main code for the project +- `helper.py` - python program for images pre- and post- processing. +- `runs` - directory with processed images +- `cityscapes.ipynb` - Jupyter notebook with some visualization and preprocessing of the Cityscape dataset. Please, see the notebook for correct dataset directories placing. +- `Segmentation_cityscapes.ipynb` - Jupyter notebook with the main code for the Cityscape dataset. +- `helper_cityscapes.py` - python program for images pre- and post- processing for the Cityscape dataset. + +**Note:** The repository does not contain any training images. You have to download the image datasetsplace them in appropriate directories on your own. + +### Architecture + +A Fully Convolutional Network (FCN-8 Architecture developed at Berkeley, see [paper](https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf) ) was applied for the project. It uses VGG16 pretrained on ImageNet as a encoder. +Decoder is used to upsample features, extracted by the VGG16 model, to the original image size. The decoder is based on transposed convolution layers. + +The goal is to assign each pixel of the input image to the appropriate class (road, backgroung, etc). So, it is a classification problem, that is why, cross entropy loss was applied. + +### Setup + +Hyperparameters were chosen by try-and-error process. Adam optimizer was used as a well-established optimizer. Weights were initialized by a random normal initializer. Some benefits of L2 weights regularization were observed, therefore, it was applied in order to reduce grainy edges of masks. + +### Augmentation + +Resized input images were also treated by random contrast and brightness augmentation (as linear function of the input image). It helps to produce reasonable predictions in difficult light conditions. + +```Python +def bc_img(img, s = 1.0, m = 0.0): + img = img.astype(np.int) + img = img * s + m + img[img > 255] = 255 + img[img < 0] = 0 + img = img.astype(np.uint8) +return img +``` + +![1.png](readme_img/1.png) + +![2.png](readme_img/2.png) + +![3.png](readme_img/3.png) + +_Deep shadows and contrast variations are not a problem because of rich augmentation on the training stage._ + +## Cityscapes + +Two classes (roads and cars) were chosen from the Cityscapes dataset for the optional task. The classes are unbalanced (roads are prevalent), so, a weighted loss function was involved (see [Segmentation_cityscapes.ipynb](Segmentation_cityscapes.ipynb +) for details). Interestingly, RMSProp optimizer performed better for the imageset. + +Unfortunately, accord to the Cityscapes dataset licence I can not publish all produced images, however, there are some of them. + +![cityscapes_1.png](readme_img/cityscapes_1.png) + +_It correctly do not label a cyclist as a car, but mark small partly occluded cars._ + +![cityscapes_1.png](readme_img/cityscapes_2.png) + +_It does not have problems with recognizing a cobbled street as a road._ + +![cityscapes_1.png](readme_img/cityscapes_3.png) + +_And the ANN is able to mark cars in different projections._ + +References: + +1. [KITTI](http://www.cvlibs.net/datasets/kitti/) dataset +2. [Cityscapes](https://www.cityscapes-dataset.com/) dataset +3. [FCN](https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf) ANN + +_____________________ Udacity Readme.md ____________________ + ### Setup ##### Frameworks and Packages