MExTS

multi example-based texture synthesis

A package that allows to generate textures of arbitrary size with the ability to edit stylistic attributes in a meaningful way

Installation

pip install mexts

Demo

https://colab.research.google.com/drive/1b9YMnm2q9rjrZFX7aG75veUflcPwdFLX?usp=sharing

Features

1. Simple generation

Generate arbitrary-sized textures from a single example.

The image below shows a comparison between real and generated textures. Top row - real; bottom row - generated.

The texture synthesis algorithm is hybrid: it combines iterative optimization [1] with MSInit [2] and AdaIN-autoencoder [3]. The algorithm allows to reach a compromise between quality and time, so it is possible to generate good-quality textures as well as rough-quality previews of final results.

Initialization of texture generator:

from mexts.feature_extractor import FeatureExtractor
from mexts.adain_autoencoder import AdaINAutoencoder
from mexts.gen import TextureGen
import torch

device = torch.device("cuda" if (torch.cuda.is_available()) else 'cpu')
FE = FeatureExtractor().to(device)
AA = AdaINAutoencoder().to(device)
TG = TextureGen(FE, AA, device)

Loading real image and obtaining it's style reperesentation vector:

image = load_from_url("image_url").resize((256, 256))
t = FE.get_style_representation(image_to_tensor(image).to(device), K=2)

Generating a preview:

result = TG.run(
    style_tensor=t,
    size=(256, 256),
    n_iter=0,
    alpha=1
)
preview = tensor_to_image(result[0])

Generating final results:

result = TG.run(
    style_tensor=t,
    size=(256, 256),
    n_iter=10,
    alpha=0
)
final = tensor_to_image(result[0])

2. Style manipulation

Algebraic operations with style representation vectors can give predictable results.

2.1. Interpolation

It is possible to "interpolate" between two real textures.

Code:

l = 0.5
result_style_tensor = t1 * (1 - l) + t2 * l

2.2. Extraction of stylistic attributes and linear operations

This part was inspired by the papers "Deep Feature Interpolation for Image Content Changes" [4] and "Interpreting the Latent Space of GANs for Semantic Face Editing" [5].
To extract individual stylistic attributes, such as "grass between rocks" two methods are adopted. The first one, naïve, includes finding mean style representation vectors for two sets of images: for one that shows a particular attribute and for one that doesn't. Then the difference between these two vectors represents style difference.

Code:

from mexts.style_features_manipulation import style_attribute_extraction_svm
style_difference = style_attribute_extraction_means(style_tensor_set1, style_tensor_set2)

The second method is based on the assumption that for any binary stylistic attribute, there exists a hyperplane on the one side of which the attribute appears, and on the other doesn't. Then the normal to this hyperplane represents style difference. To find the hyperplane, I use SVM. It should be possible to use other algorithms.

Code:

from mexts.style_features_manipulation import style_attribute_extraction_svm
style_difference = style_attribute_extraction_svm(style_tensor_set1, style_tensor_set2)

The image below shows a comparison between these two methods.

3. GUI

To use GUI, you can clone this repository and run setup.py. After the installation is complete, launch the app by running gui/main.py. If you installed the package via pip, you may download only the folder gui/.

References

[1] Leon A. Gatys, Alexander S. Ecker, Matthias Bethge (2015). Texture synthesis and the controlled generation of natural stimuli using convolutional neural net-works. CoRR, abs/1505.07376.

[2] Gonthier, N., Gousseau, Y., & Ladjal, S. (2020). High resolution neural texture synthesis with long range constraints. arXiv. https://doi.org/10.48550/ARXIV.2008.01808

[3] Huang, X., & Belongie, S. (2017). Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. arXiv. https://doi.org/10.48550/ARXIV.1703.06868

[4] Upchurch, P., Gardner, J., Pleiss, G., Pless, R., Snavely, N., Bala, K., & Wein-berger, K. (2016). Deep Feature Interpolation for Image Content Changes. arXiv. https://doi.org/10.48550/ARXIV.1611.05507

[5] Shen, Y., Gu, J., Tang, X., & Zhou, B. (2019). Interpreting the Latent Space of GANs for Semantic Face Editing. arXiv. https://doi.org/10.48550/ARXIV.1907.10786

Also used some code and state-dicts from this implementation of [3].