GANs: Two Networks in a Duel

A generative adversarial network, or GAN, learns to make new data by setting two neural networks against each other. One, the generator, turns random noise into fakes. The other, the discriminator, learns to tell the fakes from real examples. Each gets better because the other does. The generator never sees a single real example: everything it learns comes through the discriminator’s judgement. Here you’ll train a tiny GAN in your browser, watch the two networks reach a stalemate, and see one of the ways the duel goes wrong.

A duel between two networks#

Ian Goodfellow and colleagues introduced GANs in 2014 with an analogy: the generator is a team of counterfeiters trying to pass fake banknotes, and the discriminator is the police trying to catch them. Training alternates between the two:

1. The discriminator’s turn. Show it a batch of real examples and a batch of the generator’s fakes, and nudge its weights so it says “real” (close to 1) for the first and “fake” (close to 0) for the second. It’s an ordinary classifier.
2. The generator’s turn. Make a batch of fakes from fresh random noise, ask the discriminator about them, and nudge the generator’s weights so the discriminator would say “real” more often. The gradient flows backwards through the discriminator into the generator, telling each fake which way to move.

The generator is just a function: random numbers in, a sample out. To keep things visible, the GAN here works on single numbers rather than images. The real data is a set of numbers clustered in two bumps, and the generator has to learn to turn ordinary bell-curve noise into numbers that look like they came from those bumps.

Early on, the generator’s samples (blue) sit in one lump in the wrong place, and the discriminator’s line rises wherever real data is and falls wherever the fakes are. The generator follows that slope. Within a few hundred steps its samples split between the two bumps, and the discriminator’s line sags towards 0.5 near the data. The match isn’t perfect: the generated bumps are usually narrower than the real ones, sometimes piled into a spike, and their width keeps shifting as training goes on. Two networks chasing each other don’t settle the way an ordinary model does. Look at the lines at the bottom: to turn one smooth bell curve of noise into two separate bumps, the generator learns to stretch the middle of the noise into a fast jump from one bump to the other. The few samples it leaves in the gap are exactly where the discriminator still says “fake”.

The losses tell the same story. The discriminator’s loss doesn’t fall to zero; it settles near 1.39, which is 2 × ln 2, the loss of a classifier guessing 50/50 on every example, while the generator’s settles near ln 2, 0.69. A trained GAN isn’t a winner and a loser: it’s a stalemate.

When the duel goes wrong#

GANs are famously hard to train, because each network is chasing a target that moves. One common failure has a name: mode collapse, where the generator finds one kind of output that fools the discriminator and makes nothing else. The original paper called it “the Helvetica scenario”.

We trained each setting 20 times from different random starts, for 1,500 steps each. The balanced setting covered both bumps every time. With the discriminator ahead, the generator collapsed onto a single bump in 6 of the 20 runs; with the generator ahead (a learning rate 20 times the discriminator’s), it collapsed twice and wobbled on the way in most of the others. The original paper warned specifically about training the generator too much; in this little GAN either imbalance can do it. Small changes to the balance between the two networks decide whether a GAN learns at all. Much of the GAN research after 2014 was about making that balance less delicate.

GANs then and now#

• Photorealistic faces. NVIDIA’s StyleGAN (2019) produced faces of people who don’t exist, convincing enough to start a public conversation about fake images.
• Image-to-image translation. pix2pix and CycleGAN (both 2017) turned sketches into photos, summer into winter and horses into zebras.
• Then diffusion took over. In 2021 Prafulla Dhariwal and Alex Nichol showed diffusion models beating the best GANs on image quality, and diffusion models, which train more steadily and cover the data more fully, now power most image generators.
• The adversarial idea lives on. The autoencoder inside Stable Diffusion was trained with a discriminator to keep its images sharp, and adversarial training is used to distil slow diffusion models into ones that generate in a single step.

In code#

The figure’s GAN in PyTorch (pip install torch):

import torch
from torch import nn

torch.manual_seed(0)

def real_batch(n):                              # the real data: two bumps, at -2 and +2
    centres = torch.randint(0, 2, (n, 1)) * 4.0 - 2.0
    return centres + 0.4 * torch.randn(n, 1)

def mlp():
    return nn.Sequential(nn.Linear(1, 24), nn.LeakyReLU(0.1), nn.Linear(24, 24), nn.LeakyReLU(0.1), nn.Linear(24, 1))

G, D = mlp(), mlp()                             # generator: noise -> sample; discriminator: sample -> "real?" score
opt_g = torch.optim.Adam(G.parameters(), lr=0.002, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=0.002, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
real_label, fake_label = torch.ones(64, 1), torch.zeros(64, 1)

for step in range(1500):
    # 1. the discriminator learns to tell real from fake
    fake = G(torch.randn(64, 1)).detach()
    loss_d = bce(D(real_batch(64)), real_label) + bce(D(fake), fake_label)
    opt_d.zero_grad(); loss_d.backward(); opt_d.step()
    # 2. the generator learns to make the discriminator say "real"
    loss_g = bce(D(G(torch.randn(64, 1))), real_label)
    opt_g.zero_grad(); loss_g.backward(); opt_g.step()

with torch.no_grad():
    x = G(torch.randn(1000, 1))
    near = lambda c: ((x - c).abs() < 1).float().mean().item()
    print(f"near -2: {near(-2):.0%}   near +2: {near(2):.0%}")
    print(f"losses: discriminator {loss_d.item():.2f}, generator {loss_g.item():.2f}")
# near -2: 54%   near +2: 43%
# losses: discriminator 1.36, generator 0.73

.detach() in step 1 keeps the discriminator’s update from changing the generator. Using “real” labels for the fakes in step 2 is the non-saturating trick from the note above. For images, swap the small networks for convolutional ones; the training loop stays the same.