Vanilla Generative Adversarial Networks (GAN) Review

Paper Review: Generative Adversarial Networks (GAN)

Generative Adversarial Networks (GAN) is one of the most popular methods in deep learning for generating synthetic data. In this paper review, we will take a closer look at GAN and its main components.

Basic Knowledge

Before we dive into GAN, let's take a look at two concepts that are essential to understanding GAN: Kullback-Leibler Divergence and Jensen–Shannon Divergence.

Kullback-Leibler Divergence

Kullback-Leibler Divergence (KL Divergence) is a measure of how different two probability distributions are from each other. KL Divergence is used to measure the difference between the true distribution and the predicted distribution.

Jensen–Shannon Divergence

Jensen–Shannon Divergence (JSD) is another measure of similarity between two probability distributions. JSD is used to measure the similarity between the generated distribution and the true distribution.

Generative Adversarial Networks: Summary

Generative Adversarial Networks (GAN) is a two-model approach that involves a generative model G and a discriminative model D. The generative model G captures the data distribution, while the discriminative model D estimates the probability that a sample came from the training data rather than G. The generator uses the following objective function to train the image:

$G_{min}D_{max}V(D, G) = E_{x∼pdata(x)}[log D(x)] + E_{z∼pz(z)}[log(1 − D(G(z)))].$

The generative model $G(z)$ generates a new data instance, while the discriminator model $D(x)$ determines the probability that a sample came from the real distribution (1) or the generated distribution (0).

Key of Proof = How can the formula lead to $P_g = P_{data}$

The proof of global optimality aims to make the following conditions true:

(1) $P_g = P_{data}$: make generated data distribution the same as the original data distribution.

(2) $D(G(z)) = \frac{1}{2}$: generated data is not distinguishable by the Discriminator model.

Generative Adversarial Networks: Proof

The optimal discriminator D for a fixed G is calculated as follows:

$D_{∗G}(x) = \frac{p_{data(x)}}{p_{data(x)} + p_{g(x)}}$

The global optimum is $P_g = P_{data}$, which is calculated as follows:

$C(G) =max_DV(D, G) = E_{x∼pdata(x)}[log D^(x)] + E_{z∼pz(z)}[log(1 − D^(G(z)))].$

The proof shows that KL Divergence and JSD are used to measure the similarity between the generated and true distributions.

Proposition 1. For G fixed, the optimal discriminator D is

$D_{∗G}(x) = \frac{p_{data(x)}} {p_{data(x)} + p_{g(x)}}$

Proof: for G fixed,

$V (D, G)$

$= E_{x∼pdata(x)} [log D(x)] + E_{z∼pz(z)} [log(1 − D(G(z)))]$

$E[X] = \int_x xf(x)dx$

$= \int_x p_{data(x)} log(D(x))dx + \int_z p_z(z) log( 1-D(G(z)))dz$

$= \int_x p_{data(x)} log(D(x))+ p_g(x) log( 1-D(x))dx$

It has the same shape with following function.

$function\ y => alog(x) + blog( 1- x)$

achieves its maximum in $[0,1] \ at \ \frac{a}{a+b}$

Proposition 2. Global Optimum is $P_{g} = P_{data}$

$C(G)$

$=max_DV (D, G)$

$= E_{x∼pdata(x)} [log D^*(x)] + E_{z∼pz(z)} [log(1 − D^*(G(z)))]$

$D_{∗G}(x) = \frac{p_{data(x)}} {p_{data(x)} + p_{g(x)}}$

$= E_{x~p_{data}(x)}[log\frac{P_{data}(x)}{P_{data}(x)+P_{g(x)}}] + E_{x ~ p_g(x)}[log(1-\frac{P_{data}(x)}{P_{data}(x)+P_{g}(x)})]$

$= E_{x~p_{data(x)}}[log\frac{2 \ * \ P_{data}(x)}{P_{data}(x)+P_{g}(x)}] + E_{x ~ p_g(x)}[log(1-\frac{ 2 \ * \ P_{data}(x)}{P_{data}(x)+P_{g}(x)})] - log(4)$

$= KL(p_{data(x)}||\frac{p_{data}(x) + P_g(x)}{2}) + = KL(p_{g(x)}||\frac{p_{data}(x) + P_g(x)}{2}) - log(4)$

$KL(P_{data}||P_g) = \int_{-\infty}^{\infty}P_{data}(x)log(\frac{P_{data}}{P_g})dx$

$KL(P_{data}||P_g) = \int_{-\infty}^{\infty}P_{data}(x)log(\frac{P_{data}}{P_g})dx$

$= 2* JSD(p_{data}||p_g) - log(4)$

$JSD(p||q) = \frac{1}{2}KL(p||\frac{p+q}{2}) + KL(q||\frac{p+q}{2})$

$= 2* 0- log(4)$

Code (TensorFlow 2.0)

To implement GANs using TensorFlow 2.0, check out the following GitHub link:

GitHub - erickim20/Tensorflow2.0-GAN

Conclusion

Generative Adversarial Networks (GAN) is a powerful deep learning technique used for generating synthetic data. This paper review provides an overview of GAN and its main components, as well as the key concepts of KL Divergence and JSD. We hope that this review has been helpful in your understanding of GAN.