---
title: 【DL輪読会】Toward Fast and Stabilized GAN Training for Highfidelity Few-shot Image Synthesis
tags:  #deep learning  
author: [Deep Learning JP](https://www.docswell.com/user/DeepLearning2023)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/L73534RPE5.jpg?width=480
description: 2021/01/08 Deep Learning JP: http://deeplearning.jp/seminar-2/
published: January 27, 21
canonical: https://www.docswell.com/s/DeepLearning2023/ZENWX6-dltoward-fast-and-stabilized-gan-training-for-highfidelity-fewshot-image-synthesis-241879156
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/L73534RPE5.jpg)

DEEP LEARNING JP
[DL Papers]
Toward Fast and Stabilized GAN Training for Highfidelity Few-shot Image Synthesis
1
Takumi Ohkuma, Nakayama Lab M2
http://deeplearning.jp/


# Page. 2

![Page Image](https://bcdn.docswell.com/page/87DD4QY37G.jpg)

2
(
•
)
•
Few-shot Learning,
•
DL
•
•
GAN
, etc


# Page. 3

![Page Image](https://bcdn.docswell.com/page/VJPXML6P78.jpg)

3
•
Toward Fast and Stabilized GAN Training for High-fidelity Few-shot Image
Synthesis
•
ICLR2021 Under Review
•
Anonymous
• URL
https://openreview.net/forum?id=1Fqg133qRaI


# Page. 4

![Page Image](https://bcdn.docswell.com/page/2EVL9Q3VJQ.jpg)

4
GAN
•
•
•
•
Light-weight GAN
GPU 1
excitation module
1024 1024
GAN Generator
Self-supervised Discriminator
100
skip-layer channel-wise


# Page. 5

![Page Image](https://bcdn.docswell.com/page/57GXZW31EL.jpg)

5
FFHQ [1]
Size:1024 1024
:10
(RTX 2080-Ti 1 )
Nature Photograph (unsplash.com)
Size:1024 1024
:20
(RTX 2080-Ti 1 )


# Page. 6

![Page Image](https://bcdn.docswell.com/page/4EQ8L35NJP.jpg)

6
•
GAN
e.g., BigGAN[2], StyleGAN2[3]
•
•
•
[4]
[5]


# Page. 7

![Page Image](https://bcdn.docswell.com/page/KJ42D193E1.jpg)

7
GAN
GAN
1.
2.
3.
←
←
←
Discriminator
D


# Page. 8

![Page Image](https://bcdn.docswell.com/page/LE1MZGKZ7G.jpg)

8
Light-weight GAN
•
Light-Weight GAN
• Light-Weight GAN
• G (Generator)
Generator
Skip-layer channel-wise excitation module
•
•
Self-supervised Discriminator
•
Discriminator


# Page. 9

![Page Image](https://bcdn.docswell.com/page/GEW39KD6E2.jpg)

9
Skip-layer Channel-wise excitation module
Up-Sampling
•
•
•
ResBlock[6]
connection
module (SLE Module)
G
Skip-
Skip-layer channel-wise excitation


# Page. 10

![Page Image](https://bcdn.docswell.com/page/47ZV9ZGRE3.jpg)

10
SLE Module
SLE Module (
) ResBlock
1.
channel-wise
• ResBlock
element-wise
• Channel-wise
Conv
(
)
1
• Element-wise
Conv
2. Connection
Skip-layer
•
•
Skip-connection
Channel-wise


# Page. 11

![Page Image](https://bcdn.docswell.com/page/YJ65KZY17V.jpg)

11
Generator
G
•
Upsample, Conv, BatchNorm,
)
GLU(
• 1
Conv
SLE Module
•
• 2
•
4
128 8
channel-
wise
• 512, 1024
3


# Page. 12

![Page Image](https://bcdn.docswell.com/page/GJ5YPWGLJ4.jpg)

12
GAN
• GAN
Mode-collapse
•
• Discriminator
• WGAN
GPU
•
•
GAN
• Self-supervised Discriminator
[7]


# Page. 13

![Page Image](https://bcdn.docswell.com/page/9E2G6QY3ER.jpg)

13
Self-supervised Discriminator
• Discriminator Encoder
Decoder
•
Decoder
Decoder


# Page. 14

![Page Image](https://bcdn.docswell.com/page/D7YV9WG87M.jpg)

14
Discriminator
Real image
•
5%
•
16%
•
•
•
• Decoder
•
1024%
CNN
logit
1
8%
1/8
Real/Fake
Decoder
128%
128%
Down Sampling
Crop
)*+,-
) 8%
16%
Up sampling + Conv + BatchNorm + GLU
Real/Fake
Loss
Decoder
4
8%
Crop


# Page. 15

![Page Image](https://bcdn.docswell.com/page/VENML9G9J8.jpg)

15
Adversarial Loss,
•
•
Loss
Adversarial Loss
G
D
D
Reconstruction Loss
D


# Page. 16

![Page Image](https://bcdn.docswell.com/page/Y79N42W373.jpg)

16
(1000
•
)
Baseline, StyleGAN2
• Baseline
•
•
•
•
•
DCGAN [8]
Spectral Normalization [9]
Exponential-moving-average on Generator [10]
Differentiable-augmentation [11]
GLU (
) [12]
Baseline SLE Module,
256$
•
•
1024$
8
StyleGAN2
•
@0.5
Training time, vram (GPU
)


# Page. 17

![Page Image](https://bcdn.docswell.com/page/G78KQ5NLJD.jpg)

17
256$
•
(FID score,
• StyleGAN2 20 GPU hours
• StyleGAN2
Obama
FFHQ
Ablation Study
•
•
)
5 GPU hours
Finetuning
SLR Module (Skip),
mode-collapse
(decoder)


# Page. 18

![Page Image](https://bcdn.docswell.com/page/L7LZXYPQER.jpg)

18
2
1024%
•
• vram
RTX TITAN
•
•
StyleGAN2
• Decoder
StyleGAN2


# Page. 19

![Page Image](https://bcdn.docswell.com/page/4EMVLN2KEW.jpg)

19
Real Data
StyleGAN2


# Page. 20

![Page Image](https://bcdn.docswell.com/page/PER2KDM679.jpg)

20
Real Data
StyleGAN2


# Page. 21

![Page Image](https://bcdn.docswell.com/page/P7XDL1VDJX.jpg)

21
•
• Style GAN
(5 day 2 )
Baseline
•
1 GPU Days
8
StyleGAN2
•
•
•
•
16 10GPU Days
StyleGAN2 FFHQ
FFHQ
…
Baseline


# Page. 22

![Page Image](https://bcdn.docswell.com/page/37KXL2QDJD.jpg)

22
Back-Tracking
• Back-Tracking
•
•


# Page. 23

![Page Image](https://bcdn.docswell.com/page/LJ3534PP75.jpg)

23
Style-mixing
• SLE Module
StyleGAN2
Style-mixing


# Page. 24

![Page Image](https://bcdn.docswell.com/page/8JDD4QL3JG.jpg)

24
•
GAN
• Generator SLE Module, Discriminator
•
1 GPU Day
100


# Page. 25

![Page Image](https://bcdn.docswell.com/page/VEPXML2PJ8.jpg)

25
1024
•
•
•
•
•
StyleGAN2
•
StyleGAN2
•
•
•
StyleGAN2
V100 8
Lightweight GAN
80 GPU Day


# Page. 26

![Page Image](https://bcdn.docswell.com/page/27VL9QDVEQ.jpg)

26
1. Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial
networks. In Proc. of CVPR, 2019.
2. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image
synthesis. arXiv preprint arXiv:1809.11096, 2018.
3. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and
improving the image quality of stylegan. In Proc of CVPR, 2020.
4. Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning gans. arXiv
preprint arXiv:2002.10964, 2020.
5. Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative
adversarial networks with limited data. arXiv preprint arXiv:2006.06676, 2020.
6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc.
of CVPR 2016.
7. Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, and Ngai-Man Man Cheung. Self-supervised
gan: Analysis and improvement with multi-class minimax game. Advances in NeurIPS, 2019.
8. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional
generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.


# Page. 27

![Page Image](https://bcdn.docswell.com/page/5JGXZWY17L.jpg)

27
9. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative
adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
10. Yasin Yazıcı, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, and Vijay Chandrasekhar. The
unusual effectiveness of averaging in gan training. arXiv preprint arXiv:1806.04498, 2018.
11. Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan
training. arXiv preprint arXiv:2006.10738, 2020.
12. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional
networks. In Proc of. ICML, 2017.