---
title: [DLHacks 実装]Perceptual Adversarial Networks for Image-to-Image Transformation
tags: 
author: [Deep Learning JP](https://www.docswell.com/user/DeepLearning2023)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/5EGX5942JL.jpg?width=480
description: Deep Learning JP:  http://deeplearning.jp/hacks/
published: October 19, 17
canonical: https://www.docswell.com/s/DeepLearning2023/ZP9YW4-dlhacks-perceptual-adversarial-networks-for-imagetoimage-transformation-80975022
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/5EGX5942JL.jpg)

Image-to-Image Translation with
Conditional Adversarial Nets (Pix2Pix)
&amp;
Perceptual Adversarial Networks for
Image-to-Image Transformation (PAN)
2017/10/2 DLHacks
Otsubo


# Page. 2

![Page Image](https://bcdn.docswell.com/page/4JQ8ZWG97P.jpg)

Topic : image-to-image “translation”
1


# Page. 3

![Page Image](https://bcdn.docswell.com/page/K7423K8RJ1.jpg)

Info
Pix2Pix [CVPR2017]
• Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
-
iGAN [ECCV 2016]
interactive-deep-colorization [SIGGRAPH 2017]
Context-Encoder [CVPR 2016]
Image Quilting [SIGGRAPH 2001]
Texture Synthesis by Non-parametric Sampling [ICCV 1999]
• University of California
• 178 citations
PAN [arXiv2017]
• Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao
• University of Technology Sydney, The University of Sydney,
Universite Paris-Est
2


# Page. 4

![Page Image](https://bcdn.docswell.com/page/LJ1M1X2VEG.jpg)

Background
• Many tasks are regarded as “translation”
from input image to output image
- Diverse methods exist for them
Is there single framework to achieve them?
3


# Page. 5

![Page Image](https://bcdn.docswell.com/page/GJW38NR3J2.jpg)

Overview
Pix2Pix
• General-purpose solution to image-to-image
translation using single framework
- Single framework: conditional GAN (cGAN)
PAN
• Pix2Pix - (per-pixel loss)
+ (perceptual adversarial loss)
4


# Page. 6

![Page Image](https://bcdn.docswell.com/page/4EZV83RGJ3.jpg)

Naive Implementation : U-Net (①)
①per-pixel
loss (L1/L2)
5


# Page. 7

![Page Image](https://bcdn.docswell.com/page/Y765PXRQEV.jpg)

Pix2Pix (①+②)
②adversarial
loss
6


# Page. 8

![Page Image](https://bcdn.docswell.com/page/G75YKX8274.jpg)

Pix2Pix’s loss (①+②)
①
②
②
7


# Page. 9

![Page Image](https://bcdn.docswell.com/page/9J2GWXK1JR.jpg)

PAN (②+③)
③perceptual
adversarial loss
8


# Page. 10

![Page Image](https://bcdn.docswell.com/page/DEYVL8N4EM.jpg)

PAN’s loss (②+③)
②
②
③
③
m : constant
L1 norm
9


# Page. 11

![Page Image](https://bcdn.docswell.com/page/VJNM4QX678.jpg)

Example1 : Image De-Raining
• Removing rain from single images via a deep
detail network [Fu, CVPR2017]
• ID-GAN (cGAN) [Zhang, arXiv2017]
- per-pixel loss
- adversarial loss
- pre-trained VGG’s
perceptual loss
Input
Output
(Ground Truth)
10


# Page. 12

![Page Image](https://bcdn.docswell.com/page/YE9NQD8RE3.jpg)

Example1 : Image De-Raining
• Removing rain from single images via a deep
detail network [Fu, CVPR2017]
• ID-GAN (cGAN) [Zhang, arXiv2017]
- per-pixel loss
- adversarial loss
- pre-trained VGG’s
perceptual loss
(cf. PAN uses discriminator’s
perceptual loss)
Input
Output
(Ground Truth)
11


# Page. 13

![Page Image](https://bcdn.docswell.com/page/GE8KG38G7D.jpg)

Example2 : Image Inpainting
• Globally and Locally Consistent Image
Completion [Iizuka, SIGGRAPH2017]
• Context Encoders (cGAN) [Pathak, CVPR2016]
- per-pixel loss
- adversarial loss
Input
Output
(Ground Truth)
12


# Page. 14

![Page Image](https://bcdn.docswell.com/page/LELZGVRXJR.jpg)

Example3 : Semantic Segmentation
Cityscape / Pascal VOC
• DeepLabv3 [Chen, arXiv2017]
• PSPNet [Zhao, CVPR2017]
http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?
cls=mean&amp;challengeid=11&amp;compid=6
Cell Tracking / CREMI
• Learned Watershed
[Wolf, ICCV2017]
• U-Net
[Ronneberger, MICCAI2015]
Input
http://www.codesolorzano.com/Challenges/CTC/Welcome.html
Output
(Ground Truth)
13


# Page. 15

![Page Image](https://bcdn.docswell.com/page/4JMVQGR3JW.jpg)

Result1 : Image De-Raining
(≒pix2pix)
(≒pix2pix)→
14


# Page. 16

![Page Image](https://bcdn.docswell.com/page/PJR28ZRRE9.jpg)

Result2 : Image Inpainting
15


# Page. 17

![Page Image](https://bcdn.docswell.com/page/PEXD89RY7X.jpg)

Result3 : Semantic Segmentation
16


# Page. 18

![Page Image](https://bcdn.docswell.com/page/3EKXKPR47D.jpg)

Discussion
Why is perceptual adversarial loss so efficient?
vs. No perceptual loss (Pix2Pix)
- Perceptual loss enables D to detect more
discrepancy between True/False images
vs. Pre-trained VGG perceptual loss (ID-GAN)
- VGG features tend to focus on content
- PAN features tend to focus on discrepancy
- PAN’s loss leads to avoid adversarial
examples [Goodfellow, ICLR2015] (?)
17


# Page. 19

![Page Image](https://bcdn.docswell.com/page/L735ZX8DE5.jpg)

Minor Difference
• Pix2Pix uses Patch-GAN
- Small size(70×70) patch-discriminator
- Final output of D is average of
patch-discriminator’s responses
(convolutionally applied)
18


# Page. 20

![Page Image](https://bcdn.docswell.com/page/87DDR2M27G.jpg)

To Do
• Implement
1. Pix2Pix (Patch Discriminator)
2. PAN (Patch Discriminator)
3. PAN (Normal Discriminator)
Wang et al. might compare 1 with 3.
19


# Page. 21

![Page Image](https://bcdn.docswell.com/page/VJPXW1RL78.jpg)

20


# Page. 22

![Page Image](https://bcdn.docswell.com/page/2EVL8YR6JQ.jpg)

Implementation
2017/10/17 DLHacks
Otsubo


# Page. 23

![Page Image](https://bcdn.docswell.com/page/57GX59M2EL.jpg)

My Implementation
• https://github.com/DLHacks/pix2pix_PAN
• pix2pix
- https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
• PAN
- per-pixel loss à perceptual adversarial loss
- not same as paper’s original architecture
- num of parameters is same as pix2pix
22


# Page. 24

![Page Image](https://bcdn.docswell.com/page/4EQ8ZWR9JP.jpg)

My Experiments
• Facade (label à picture)
• Map (picture à Google map)
• Cityscape (picture à label)
23


# Page. 25

![Page Image](https://bcdn.docswell.com/page/KJ423KQRE1.jpg)

Result (Facade pix2pix)
24


# Page. 26

![Page Image](https://bcdn.docswell.com/page/LE1M1XPV7G.jpg)

Result (Facade PAN)
25


# Page. 27

![Page Image](https://bcdn.docswell.com/page/GEW38N23E2.jpg)

Result (Map pix2pix)
26


# Page. 28

![Page Image](https://bcdn.docswell.com/page/47ZV83DGE3.jpg)

Result (Map PAN)
27


# Page. 29

![Page Image](https://bcdn.docswell.com/page/YJ65PX1Q7V.jpg)

Result (Cityscape pix2pix)
28


# Page. 30

![Page Image](https://bcdn.docswell.com/page/GJ5YKX42J4.jpg)

Result (Cityscape PAN)
29


# Page. 31

![Page Image](https://bcdn.docswell.com/page/9E2GWXV1ER.jpg)

Result (PSNR[dB])
30


# Page. 32

![Page Image](https://bcdn.docswell.com/page/D7YVL8347M.jpg)

Discussion – Why pix2pix &gt; PAN?
• per-pixel loss is needed?
• patch discriminator is not suited for PAN?
• positive margin m
•
(bad pix2pix implementation in PAN’s paper…?)
31


