CMSC 740 Fall 2017 is taught by an awesome teacher Prof. Matthias Zwicker. 

 

Projects

The course project is based on Nori, an educational ray tracer.

 

Notes

Acceleration structures,

Uniform grid

Object subdivision

Bounding volume hierarchies (BVHs)

Partition space into non-overlapping cells

Each cell stores reference to all objects

that overlap it

Spatial subdivision

Binary space partitioning (BSP) trees

Recursively divide space into two parts using

dividing planes (with arbitrary position,

orientation)

k-d-trees, nearest neighbor search

 

Light Transport Effects

Fluorescence

Reflection

Thin film interference: https://www.shadertoy.com/view/XddXRj

Diffraction

Refraction

Dispersion

Rayleigh Scattering: https://www.shadertoy.com/view/MdXSzX (Subsurface Scattering)

Polarization

 

Radiometry

Quantify spatial energy distribution of light

 (in a way that is compatible with the geometrical optics model)

Physical measurement of electromagnetic energy

Unit: Watts

Assume light consists of photons with

Position x

Direction of motion w

Wavelength lambda

Each photo has an energy of hv

H Planck’s constant

V = 1 / lambda frequency

Measuring energy means “counting photons”

Spectral Radiance

Energy per time per wavelength per solid angle per area

 means differetial area perpendicular to

 

Power carried along a ray is measured in radiance

Radiance along ray is constant (in vacuum)

In practice: assume steady state, measure at discrete wavelengths R, G, B

Radiance L: power per solid angle per area, vector of 3 values for R, G, B

Irradiance: Power per unit area

Integration of radiance over hemisphere

Irradiance E: power per area perpendicular to the surface normal

Radiance L: power per area perpendicular to the incident ray \omega

 

When we do ray tracing, we use radiance for rays

 

Integration in spherical coordinates

 

Radiant intensity I in all direction, has a total power

 

 

Photometry

Perceptual measurement of perceived brightness

Lumen

 

 

 

Deep Learning

Concepts

• Layers
• Activation function
  • logistic sigmoid function: \frac{1}{1+e^{-x}}
  • unit step
  • sign
  • linear
  • piece-wise linear
  • hyperbolic tangent
• Feature map – the output of one filter applied to the previous layer
• Fully connected layer
• Convolutional layer (convolutional neural networks, CNN)
• Loss functions
  • The loss function computes the error for a single training example.
  • The cost function is the average of the loss functions of the entire training set.
• Stochastic gradient descent
  • use ONLY ONE or SUBSET of training sample from your training set to do the update for a parameter in a particular iteration 
  • Pros: converges much faster compared to GD
  • Cons: the error function is not as well minimized as in the case of GD
  • Batch size (64, 128, or 256)

    • Size of subset as an approximation of the whole databaseIt, which impacts the time efficiency of training and the noisiness of the gradient estimate. Updating the parameters using all training data is not efficient. Updating by one single sample (online updating) is noisy if the sample is not a good representation of the whole data.

  • Batch normalization
    • It is always great if the layer get data that is always normalized(mean 0 variance 1).
    • We just normalize each batch with its running mean and variance.
    • Batch normalization potentially helps in two ways: faster learning and higher overall accuracy. The improved method also allows you to use a higher learning rate, potentially providing another boost in speed.
• Automatic differentiation – numerically evaluate the derivative of a function
  • Error back-propagation
    • compute gradients of a loss function using the chain rule
    • proceed ‘backwards’ with respect to the computations performed to compute the loss itself
• Feed-forward and recurrent neural network
  • Feed-forward
    • The connections between the units do not form a cycle
  • Recurrent Neural Network
    • Sequence of words and images, unsegmented, connected handwriting recognition, or speech recognition
• Autoencoder / autoassociator / Diabolo network 
  • Used for unsupervised learning of efficient codings, earn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction, widely used for learning generative models of data.
• Encoder / Decoder
  • The encoder encodes the input sequence to an internal representation called ‘context vector’ which is used by the decoder to generate the output sequence. The lengths of input and output sequences can be different, as there is no explicit one on one relation between the input and output sequences.
  • An Encoder/Decoder is not an autoencoder
• Reinforcement learning
  • Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Instead the focus is on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge).
• Skip connections
• Residual network
• u-net

Denoising Monte Carlo Renderings

Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings

Specular has large dynamic range

• Take advantage of additional information (besides RGB) produced by renderer
• Network
  • 8 hidden layers, 100 kernels of 5×5 in each layer
  • L1 loss
• Training data
  • 600 frames from movie “Finding Dori”
    • Noise: 32 spp
    • Reference: 1024 spp
• Test data
  • 25 frames from other movies

In terms of timing, for an HD image of 1920×1080, our network takes about 12 seconds to evaluate and output a full denoised image

Denoising animations

Frame-by-frame denoising leads to temporal flickering in animations

Need to filter in 3D space-time

How to implement this with CNNs?

Chaitanya et al. Interactive Reconstruction of Monte Carlo Image Sequences using a Recurrent Denoising Autoencoder, SIGGRAPH 2017

 

Input to network 

• RGB colros
• G-buffer with view-space shading normals, depth, material roughness
• Divided by diffuse albedo (remove texture complexity from input)

Loss

• Spatial: L1, L1 on spatial gradients
• Temporal: L1 on temporal gradients
• Back propagation through time
• Replicate feed forward parts to unroll recurrent connections
• “Adam” SGD solver
• Gradients are noisy in SGD
• Compute moving average over SGD steps
• feed-forward network

Input with 1spp

• 54.9ms for 720p on Titan X
• Path tracer from 70ms to 260ms for 1spp

 

What else could we learn to improve rendering efficiency?

• Superresolution
• Foveated rendering
• Antiliasing
• Multiple Importance Sampling
• Antialiasing for light fields

 

• Learning Light Transport the Reinforced Way, Dahm and Keller
• Use Q learning to learn importance sampling incrementally during path tracing

Deep Learning: The Future of Real-Time Rendering