Introduction
A generative adversarial network (GAN) is a class of machine learning frameworks that generate new fake data deceptively from real data. It was invented in just 2014, but its applications have increased rapidly. It has also been used successfully in lots of areas, including fashion, art, advertising, and science.
GANs can be used to achieve image-to-image translation, where the generation of the output image is conditional on an input. I think it would be interesting to use segmentation layouts or sketches as inputs to let the model generate pretty faces. Right, pretty or not is a matter of taste). Fortunately, I found one paper, Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation(Encoding in Style), which published recently has paved the road to this imaginary transformation.
Main Problems
- How to build a pretty-face dataset?
- How to generate face?
- How to translate sketch to face?
Solutions I applied
Use a GAN to create a dataset for another GAN
The benefit of GANs is that it can provide unlimited data. If I want to create a dataset with pretty-face, why not use a GAN pre-trained on celebrities. Why celebrities? Because it’s heuristic that there are more good-looking people in the group of stars than in the group of ordinary people. Since a GAN learn the distribution of its training data, it masks sense to select stars as training data. Luckily, when I was searching for a proper dataset, I found someone has already trained a StyleGAN2 on an ideal dataset that has 95600 512*512 faces from 500 Chinese stars.
Since not all the stars in the training dataset have beautiful faces, the GAN images are not all pretty. A model’s problem can be solved using another model. Here, I used the model to generate about 3000 pictures and manually selected those that look not bad. Using not bad images and the same number of other images, I prepared a dataset to train a binary classifier using MixNet-S as the backbone. This model can filter qualified faces from the GANs’ products. Next, StyleGAN2 generated another 30,000 images, and the classifier selected 3318 not-bad from them. Finally, these images are collected as the dataset for the following training.
Generate Sketch Data
Many studies mentioned how to generate sketch data from original data, especially in the community of GANs for line art coloration. Common methods include using sketchKeras, Sketch Simplification etc. Here, I applied the sketchGAN from Sketch Simplification as the authors of Encoding in Style did to make sketches from the selected faces.
Training
The authors of Encoding in Style did a fantastic job of sharing their training procedure. Just follow their simple instructions, and it’s done.
- Change the settings in pixel2style2pixel/configs/data_configs.py
- Use the command lines
Plus: Segmentation Layout
I also tried to use segmentation layouts as inputs to generate images. Here another model is created to create images from segmentation maps (also just following their instruction). The faces’ segmentation layouts are created by using face-parsing
- Change the settings in pixel2style2pixel/configs/data_configs.py
- Use the command lines
Results
Pretty face dataset
- Faces
- sketches
- segmentation
The dataset has been published on Kaggle.
sketch to faces
segmentation to faces
