With the advent of COVID-19 facial coverings like masks have become increasingly prevalent. Even though advanced facial recognition systems have been developed and produce satisfactory results, obstructions and covering on facial features is a major issue that render these systems useless. Since mask is one of the things that can be classified as obstruction to a face, we developed project that removes the mask and generates or reconstructs the hidden part of the face covered by mask.
It was developed by Arpita Nanda, Dawood Damda, Gagan V, Emaad Jaffer
We developed a model using two main modules:
- Map Module, a modified version of the U-Net, it creates a segmentation map of the mask.
- Editing Module, composed of GAN(one generator and two discriminator) and a Perceptual Network, it generates the hidden part of the face covered by mask.
This project will take all the required images the model needs (masked images) from the directory testing/test_mask. First, it will generate the maps from the masked images and will save them in testing/test_map. Then, it uses these maps for the editing module, and it generates the unmasked images, saved in testing/results.
- Download or clone this repository.
- Download the Dataset.
- Run
Unmask.ipynb - To skip the training process, you can download the pre-trained model from Model.
After the training process, accuracy and loss graphs were plotted.
Check out "Facial Reconstruction of Masked Face Using Generative Adversarial Networks.pdf" for more detailed explanation about the project.
Here's some sample outputs:
------------------Input--------------------------Mask Segmentation---------------------Generated Output---------
The ground truth images for above examples:
As you can see, even with the few resources we had, the outcomes are still more than adequate. Overfitting of the model was seen, which resulted in poor generalisation. The issue might have been resolved by making the dataset more diverse and heterogeneous.
We recommend you to use system with higher processor, higher RAM and a better GPU for smooth training and processing.
OR
Use Google Colab Pro to avoid "ResourceExhaustedError" error during the training process due to insufficient RAM.