Today’s Lesson

1. Generative Adversarial Networks (GAN)
2. Applications to Sketch Simplification
   1. Semi-Supervised
   2. Interactive Neural Networks
3. Convex Optimization Layers
4. Neural Radiance Fields (NERF)

GAN Optimization (1)

• ${\color[RGB]{255,117,74} D(\cdot) \;\text{maximizes classification}}$
• ${\color[RGB]{71,194,255} G(\cdot) \;\text{tries to fool}\; {\color[RGB]{255,117,74}D(\cdot)} \;\text{with minimization}}$
• Alternate training scheme

   $${\color[RGB]{71,194,255} \min_G} \;
  {\color[RGB]{255,117,74} \max_D} \;
  {\color[RGB]{255,117,74} \underbrace{\E_{y^*\sim\rho_y}}_\text{real data} \log D(y^*)} +
  {\color[RGB]{71,194,255}\underbrace{\E_{z\sim\mathcal{N}(0,1)}}_\text{random variable}}{\color[RGB]{255,117,74}\log(1-D({\color[RGB]{71,194,255}G(z)}))}$$ 

• In practice, $\min_G\; \E_{z\sim\mathcal{N}(0,1)} \log(1-D(G(z)))$ can saturate, and instead $\max_G\; \E_{z\sim\mathcal{N}(0,1)} \log(D(G(z)))$ is optimized

GAN Optimization (2)

GAN Optimization (3)

Proposition 1

For $G$ fixed, the optimal discriminator $D$ is

 $$D^\star_G(x) = \frac{\rho_y(x)}{ \rho_y(x) + \rho_g(x) }$$ 

Proof.

• Optimization of a discriminator can be seen as maximizing

   $$\begin{align}
  V(G,D) &= \int_x \rho_y(x)\log(D(x))dx + \int_z \rho_z(z) \log(1-D(G(z)))dz \\
  &= \int_x \rho_y(x)\log(D(x))dx + \rho_g(x) \log(1-D(x))dx \\
  \end{align}$$ 

• For any $(a,b)\in \mathbb{R}^2 \setminus 0$, the function $y \rightarrow a \log(y) + b \log(1-y)$ achieves the maximum in $[0,1]$ at $\frac{a}{a+b}$

GAN Optimization (4)

Theorem 1

The global minimum of the virtual training criterion $C(G)=\max_D V(G,D)$ is achieved if and only if $\rho_g = \rho_y$. At that point, C(G) achieves the value $−\log 4$.

Proposition 2

If $G$ and $D$ have enough capacity, and at each step of the algorithm, the discriminator is allowed to reach its optimum given $G$, and $\rho_g$ is updated so as to improve the criterion

 $$\E_{x\sim \rho_y}[ \log D^*_G(x) ] + \E_{x\sim \rho_g}[ \log(1-D_G^*(x)) ]$$ 

then $\rho_g$ converges to $\rho_y$

GAN Optimization (5)

Algorithm. Generative Adversarial Network training.

1. repeat
   1. for $k$ steps
      1. Sample minibatch of $m$ noise samples $\{z_1, \ldots, z_m\}$ from noise prior $\mathcal{N}(0,1)$
      2. Sample minibatch of $m$ examples of real data $\{y^*_1, \ldots, y^*_m\}$
      3. Update discriminator with gradient ascent. $\nabla \frac{1}{m} \sum_{i=1}^m \left[ \log(D(x_i)) + \log(1-D(G(z_i))) \right]$
   2. Sample minibatch of $m$ noise samples $\{z_1, \ldots, z_m\}$ from noise prior $\mathcal{N}(0,1)$
   3. Update generator with gradient descent. $\nabla \frac{1}{m} \sum_{i=1}^m \log(1-D(G(z_i)))$

Common Issues

• Mode collapse
  • Instead of modelling the entire distribution, $G$ ends up realistically modelling only a subset
  • Rare cases are ignored
• Overfitting
  • Model learns to copy and paste patterns
  • Unable to extrapolate
  • Model is not learning the full manifold

Overview

• Strengths
  • Unsupervised learning
  • Applicable to virtually any problem
  • Able to boost performance with proper tuning
• Weaknesses
  • Optimization is extremely unstable
  • Very susceptible to changes in hyperparameters
  • Not clear what is being learnt

Popular Extensions

• Wasserstein GAN (Improved): better loss function? (maybe just hyperparameters?)
• pix2pix: conditional generation
• CycleGAN: unsupervised conditional generation
• Progressive GAN: higher resolution
• Spectral Normalization: spectral-based layer normalization
• Self-Attention: self-attention
• …

GAN Exercise (12-2.ipynb)

1. Finish GAN loss
2. Not a hand-in

Sketch Simplification (1)

Sketch Simplification (2)

Dataset Bias

Proposed Framework

 $$\begin{align}
\min_S \max_D \; &\overbrace{\E_{(x,y^*)\sim\rho_{x,y}}}^\text{Supervised} \left[\; {\color[RGB]{127,162,55} \overbrace{\|S(x)-y^*\|_2}^\text{Standard Loss}} + {\color[RGB]{55,127,162} \overbrace{\alpha \log D(y^*)  + \alpha \log(1-D(S(x)))}^\text{Adversarial Loss}} \;\right] \\
&+ {\color[RGB]{162,55,127} \underbrace{\beta \overbrace{\E_{y\sim\rho_y}}^{Line} \left[\; \log D(y) \;\right] + \beta \overbrace{\E_{x\sim\rho_x}}^\text{Rough}\left[\; \log(1-D(S(x))) \;\right]}_\text{Unsupervised Adversarial Loss} }
\end{align}$$ 

Towards Transductive Machine Learning (1)

• Inductive Machine Learning
  • Learn parametric model from data
  • Use deduction with the parametric model to infer properties of testing data
  • Common machine learning (in particular deep learning) paradigm
• Transductive Machine Learning
  • Infer directly about testing data from training data
  • Commonly done with Gaussian processes, etc.
  • Possible to use proposed framework to optimize for a single image using discriminator

Towards Transductive Machine Learning (2)

Motivation

Interactive Neural Networks

Training Framework

• Normalize line drawings for higher performance
• Simulate user edits heuristically

Line Normalization

Simulating User Edits

Training

 $$L(y,y^*) = \| {\color[RGB]{107,142,35} \underbrace{\left(y-y^*\right)}_{L_1\text{ loss}}} \odot {\color[RGB]{182,70,35} \underbrace{\left(1 + \gamma\left(1-y^*\right)\right)}_{\text{Weight lines with }\gamma}} \|_1 \nonumber$$ 

Interface

• Web interface with python backend

Video

User Study

• Comparison (time) with proposed approach vs Clip Studio Pro
• Total of 10 users and 10 unique images
• Each user processes random 5 images with each tool
• Total average time of 2.8 hours per user
• Overall 1.8× speed-up with ours

Discussion

• Interactive Neural Networks
  • Very useful for fast real-time interaction
  • Need good heuristics to get good results
  • Trade off between interactivity and automatic results
• GAN loss not used due to instability
  • Can get good results also by rethinking data → line normalization
  • Perceptual loss is also a good alternative

Formulation

• Consider embedding a quadratic problem as a layer

   $$\begin{split}
  z_{i+1} =\quad & \argmin_z   \quad & \frac{1}{2}z^\intercal Q(z_i) z + q(z_i)^\intercal z \\
   & \text{subject to} \quad & A(z_i)z = b(z_i) \\
   & & G(z_i) z \leq h(z_i)
  \end{split}$$ 

  • $z_i$: $i$-th layer output vector
  • $z\in\mathbb{R}^n$: optimization variable
  • $Q(z_i)$, $q(z_i)$, $A(z_i)$, $b(z_i)$, $G(z_i)$, $h(z_i)$: parameters of the optimization problem
    • In practice, fixed matrices are considered, i.e., $Q(z_i) = Q \in \mathbb{R}^{n \times n}$
  • $n$ variables, $m$ equality constraints, $p$ inequality constraints

Lagrangian and Differenciation

• Differenciate KKT equations
• Lagrangian of the layer

   $$L(z, v, \lambda) = \frac{1}{2} z^\intercal Q z + q^\intercal z + v^\intercal (Az-b) + \lambda^\intercal(Gz-h)$$ 

• KKT conditions

   $$\begin{align}
  Qz^\star + q + A^\intercal v^\star + G^\intercal \lambda^\star = 0 \\
  Az^\star - b = 0 \\
  D(\lambda^\star)(Gz^\star-h) = 0
  \end{align}$$ 

  • $D(\cdot)$: creates diagonal matrix

Deriving KKT Conditions

• Deriving KKT conditions in vector form

 $$\begin{align} \begin{bmatrix}
Q & G^T & A^T \\
D(\lambda^\star)G  & D(Gz^\star-h) & 0 \\
A & 0 & 0 \\
\end{bmatrix}
\begin{bmatrix}
\mathsf{d} z \\
\mathsf{d} \lambda \\
\mathsf{d} \nu \\
\end{bmatrix} = \\
\begin{bmatrix}
-\mathsf{d} Qz^\star - \mathsf{d} p - \mathsf{d} G^T\lambda^\star - \mathsf{d} A^T\nu^\star \\
-D(\lambda^\star)\mathsf{d} Gz^\star + D(\lambda^\star)\mathsf{d} h \\
-\mathsf{d} Az^\star + \mathsf{d} b \\
\end{bmatrix} \end{align}$$ 

• Partial derivatives can be calculating by solving system
  • e.g., setting $\mathsf{d}b=I$ and other differential terms to 0 and solving will give $\frac{\partial z^\star}{\partial b}\in\mathbb{R}^{n \times m}$

Computation for Back-Propagation

• We are not interested in full jacobians
• Want to compute $\frac{\partial l}{\partial z^\star}\in\mathbb{R}^n \frac{\partial z^\star}{\partial b}$
  • Backwards pass vector $\frac{\partial l}{\partial z^\star}\in\mathbb{R}^n$ is given
• Multiplying the backward pass vector by the transpose of differential matrix

   $$\begin{bmatrix}
  d_z \\ d_\lambda \\ d_\nu
  \end{bmatrix}
  =
  \begin{bmatrix}
  Q & G^T D(\lambda^\star) & A^T \\
  G  & D(Gz^\star-h) & 0 \\
  A & 0 & 0 \\
  \end{bmatrix}^{-1}
  \begin{bmatrix}
  \left(\frac{\partial \ell}{\partial z^\star}\right)^T \\ 0 \\ 0
  \end{bmatrix}$$ 

Solving for Partials

 $$\begin{aligned}
\frac{\partial \ell}{\partial p} &= d_z
& \frac{\partial \ell}{\partial b} &= -d_\nu \\
\frac{\partial \ell}{\partial p} &= d_z
& \frac{\partial \ell}{\partial Q} &= \frac{1}{2}(d_z z^T + zd_z^T) \\
\frac{\partial \ell}{\partial A} &= d_\nu z^T + \nu d_z^T
& \frac{\partial \ell}{\partial G} &= D(\lambda^\star)(d_\lambda z^T +
\lambda d_z^T)
\end{aligned}$$ 

Efficiently Computing

• Use primal-dual interior point method
  • Use Lower-Upper (LU) decomposition at each iteration
  • Backward pass can be extracted for “free” from factored KKT matrix at solution
• Partial block LU factorization to solve KKT system
• Solution of previous iteration is used in each iteration

Application: Solving Sudoku with a NN

• Consider $4 \times 4$ mini-sudoku
• Don’t know rules beforehand
  • If rules are known can be solved by linear programming
• Solution modeled as $4 \times 4 \times 4$ one-hot encoding
• Model
  • 10 convolutional layers (512 filters, $3 \times 3$ kernels)
  • QP layer

     $$\argmin_z \quad \frac{1}{2}z^\intercal 0.1I z + q^\intercal z \quad \text{subject to} \quad Az = b, \quad -z \leq 0$$ 

    • Positive inequality constraint ($z \geq 0$)
    • Arbitrary equality constraint matrix $Az = b$
    • $Q = 0.1 I$ to make sure problem is feasible
    • Linear term $q$ is one-hot encoding of input
• Training on 9000 examples, testing on 1000 examples

Results

Wrap-Up

• It is now possible to easily incorporate convex optimization into neural networks
  • Differentiable Convex Optimization Layers
• Very useful to exploit knowledge about the domain problem
• Possible to enforce constraints
• Care must be taken to formulate solvable problems

Ray Tracing

Lighting Basics

• When there is no participating medium, lighting can be written using the reflectance equation:

   $$L_o(\bm{v}) = \int_{\bm{l}\in\Omega} \overbrace{f(\bm{l},\bm{v})}^\text{brdf} \, \overbrace{L_i(\bm{l}) \, (\bm{n}\cdot\bm{l})}^\text{light} \, d \bm{l}$$ 

The Rendering Equation

• Generalized version of the reflection equation

   $$L_o(\bm{p},\bm{v}) = L_e(\bm{p},\bm{v}) + \int_{\bm{l}\in\Omega} f(\bm{l},\bm{v}) \, L_o(r(\bm{p},\bm{l}),-\bm{l}) \, (\bm{n}\cdot\bm{l})^+ \, d \bm{l}$$ 

• Emitted radiance from surface $L_e(\bm{p},\bm{v})$ is a new term
• Also $L_i(\bm{p},\bm{l}) = L_o(r(\bm{p},\bm{l}),-\bm{l})$
  • Incoming radiance at $\bm{p}$ from $\bm{l}$ is equal to outgoing in the opposite direction
  • Ray casting function $r(\bm{p},\bm{l})$

Participating Media

• Can no longer assume $L_i(\bm{c},-\bm{v}) = L_o(\bm{p},\bm{v})$
• Have to consider radiance along the view ray

   $$L_i(\bm{c},-\bm{v}) = T_r(\bm{c},\bm{p}) \, L_o(\bm{p},\bm{v}) + \int_{t=0}^{\|\bm{p}-\bm{c}\|} T_r(\bm{c},\bm{c}-\bm{v}t) \, L_\text{scat}(\bm{c}-\bm{v}t,\bm{v}) \, \sigma_s \, dt$$ 

  • $T_r(\bm{c},\bm{x})$ is transmittance function
  • $L_\text{scat}(\bm{x},\bm{v})$ is light scattered along view ray at point $\bm{x}$
• Beer-Lambert Law:

   $$T_r(\bm{x}_a,\bm{x}_b) = e^{-\tau} \quad\text{ with }\quad \tau = \int_{\bm{x}=\bm{x}_a}^{\bm{x}_b} \sigma_t(\bm{x}) \, \|d\bm{x}\|$$ 

Volume Rendering with Radiance Fields

• Given rays $\bm{r}(t) = \bm{o} + t \bm{d}$, we can define the expected colour of a camera ray with:

   $$\begin{align}
  C(\bm{r}) &= \int_{t_n}^{t_f} T(t) \, \sigma(\bm{r}(t)) \, \bm{c}(\bm{r}(t),\bm{d}) \, dt \\
  T(t) &= \exp\left( -\int_{t_n}^{t_f} \sigma(\bm{r}(s)) \, ds \right)
  \end{align}$$ 

• Assumptions:
  • Only transmittance
  • Radiance is equivalent to colour
  • Approximate with NN: $F_\Theta \colon (\overbrace{x,y,z}^\text{position},\overbrace{\theta,\phi}^\text{orientation}) \rightarrow (\overbrace{\bm{c}}^\text{rgb},\overbrace{\sigma}^\text{density})$

Optimization

 $$\mathfrac{L} = \sum_{\bm{r} \in R} \left[ \| \overbrace{\hat{C}_c(\bm{r})}^\text{coarse prediction} - \overbrace{C(\bm{r})}^\text{GT} \|_2^2 + \| \overbrace{\hat{C}_f(\bm{r})}^\text{fine prediction} - C(\bm{r}) \|_2^2 \right]$$ 

• Uses hierarchical volume sampling for efficiency
• Use two networks: one for coarse details, and one for fine details with a coarse-to-fine approache
  • Similar to importance sampling
• Use COLMAP structure-from-motion to estimate camera parameters from data
• Single scene takes 1-2 days on a NVIDIA V100 GPU