Textures

• Process applying an image, function, or other data source to determine the appearance of a surface
• Most simple application is to apply 2D images on polygon meshes known as texture mapping
• Other applications include
  • Bump mapping allows simulating rough textures on flat surfaces
  • Parallax mapping makes a surface seems deformed when rendering
  • Displacement mapping truly displaces a surface by modifying the triangles
• Pixels in texture images are known as texels to differentiate them from pixels on the screen

Texturing Pipeline

1. Projector function (mapping): Converts from object space location to parameter space coordinates.
2. Corresponder function(s): Converts the parameter space coordinates to texture space locations.
3. Obtain value: Obtains the texture value at the texture space location.
4. Value transform function: Transforms the value as necessary.

Projector Function

• Objective to obtain surface’s location and project it into texture coordinate space
  • Input is usually three-dimensional $(x,y,z)$, however, surface normals and other information can also be used
  • Usually outputs two-dimensional $(u,v)$ space
• Modeling software usually defines $(u,v)$-coordinates per vertex
  • Can be edited the same way as vertices
  • Usually created by mesh unwrapping algorithms
  • Commonly used projections are spherical, cylindrical, and planar projections
  • Issue is matching textures at seams where faces meet
• Parametric curved surfaces usually have a natural set of $(u,v)$ values and don’t have issues

Corresponder Function

• Converts texture coordinates to texture-space coordinates
• Can be made to use a subimage instead of a complete image
• Can apply a transformation matrix
• Can also control what happens outside the $[0,1]$ range:
  • gl_REPEAT: image repeats itself across the surface (OpenGL default)
  • gl_MIRROR: image repeats but is flipped on every other repetition providing continuity
  • gl_CLAMP_TO_EDGE: values outside are clamped to the edge
  • gl_CLAMP_TO_BORDER: values outside are rendered with a separately defined border colour
• Can usually be defined separately for each texture axis

Texture Value

• Use texture-space coordinates to produce texture values
• Actually access the texture data to retrieve texel information
• Can also be done procedural where a function is called instead of a texture look-up
• Most values are RGB triplets, however, it is possible to consider opacity (alpha in RGBA) or pretty much any 1 to 4 dimensional value
• Values can be transformed before use such as converting normals to $[-1,+1]$ range from the $[0,1]$ texture value

Image Texturing

• Effectively gluing a two-dimensional image onto the surface of one or more triangles
• Shaders:
  • Recieve the coordinates in the vertex shader and pass them to the fragment shader
  • Set up a uniform sampler in the fragment shader (sampler2D for 2D images)
  • Sample from the texture and use it to set output colour
• WebGL Javascript:
  • Create a texture with gl.createTexture();
  • Bind texture with gl.bindTexture(...) and set the data with gl.TexImage2D(...)
  • Set the texture to the uniform value in the shader
  • Pass texture coordinates to the vertex shader

Texture Fragment Shader and Loading

• Fragment Shader:

  uniform sampler2D my_texture; // Texture is defined as a uniform
  in vec2 texcoords; // Set in the vertex shader
  out vec4 colour_out; // Output colour
  void main () {
   // Sample the texture at corresponding position and output
   colour_out = texture( my_texture, texcoords );
  }

• WebGL texture loading code:

  // Create a new texture
  var texture = gl.createTexture();
  gl.bindTexture(gl.TEXTURE_2D, texture);
  // Fill the texture with a 1x1 red pixel
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE,
   new Uint8Array([255, 0, 0, 255]));
  // Asynchronously load an image
  var image = new Image();
  image.src = "path/to/image.png";
  image.addEventListener('load', function() {
  // Copy the image to the texture
   gl.bindTexture(gl.TEXTURE_2D, texture);
   gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA,gl.UNSIGNED_BYTE, image);
  });

Texture Vertex Shader and Drawing

• Vertex Shader:

  in vec2 tex_vertex; // Vertex texture coordinate
  out vec2 texcoords; // Passed to fragment shader
  ... // Vertex position stuff
  void main () {
   texcoords = tex_vertex; // You could do transformations here if needed
   ... // Vertex position stuff
  }

• WebGL vertex loading code (VBO):

  var loc_tex_vertex = gl.getAttribLocation(program, "tex_vertex");
  // Create buffer
  var texcoordBuffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, texcoordBuffer);
  // Set buffer data here with gl.bufferData(...)

• WebGL drawing code:

  gl.enableVertexAttribArray( loc_tex_vertex );
  gl.vertexAttribPointer( loc_tex_vertex, 2, gl.FLOAT, false, 0, 0 );

Other Notes

• OpenGL defines the origin $(0,0)$ at the bottom left of the image
• How to handle subpixels?
  • OpenGL defines the values $(0.5,0.5)$ to be the center of a pixel
• WebGL1 and early GPUs only support Power-Of-Two (POT) textures ($2^m \times 2^n$ texels with $m,n \in \mathbb{Z}_+$)
  • An image of size 400 would have to be embedded in one of size 512
  • WebGL2 and modern GPUs support Non-Power-Of-Two (NPOT)
• Usually size is limited to $16,384 \times 16,384$ texels
• How to deal when texture is getting scaled up/down? Magnification and minification

Sampling and Filtering Theory

• Sampling converts a continuous signal into a sampled discrete signal
• Discrete signals have reduced information with respect to continuous signals
• Discrete signal needs to be reconstructed to obtain the continuous signal again
• Filtering helps reconstruct the original signal
• Whenever sampling is done, aliasing may occur
  • Aliasing is when sampling at too low of a frequency
  • Signal becomes confused with other lower frequency signals

Nyquist Rate

• For a signal to be properly sampled (i.e. able to reconstruct original signal), the sampling frequency has to be twice the maximum frequency of the sampled signal
  • This is know as the Nyquist rate or Nyquist limit after the Swedish scientist Harry Nyquist who discovered this in 1928
  • Relies on the existence of a maximum frequency which implies a band-limited signal
• 3D scenes are never band-limited when rendering with point samples
  • Edges of triangles, shadow boundaries, and other phenomena produce a signal that changes discontinuously and thus infinite frequencies
  • In general, there will always be aliasing in computer graphics
  • Issue is how to minimize the visual effects

Reconstructing Signals

• Given a band-limited sampled signal it is possible to reconstruct it with filters
• Filters are functions that convert discrete signals to continuous
  • Area of the function has to be 1 to not amplify or decrease the signal
• Some common filters:
  • Box filter: centers boxes on each discrete point
  • Tent filter: centers triangles on each point
  • Sinc filter: ideal low-pass filter $\text{sinc}(x)=\frac{\sin(\pi x)}{\pi x}$
    • Ideal when sampling frequency is 1 (maximum frequency signal is 1/2)
    • It is a box in the frequency domain
    • Not possible to implement due to having infinite width

x *= 2.0; /* For visualization purposes. */
y = sin( M_PI * x) / (M_PI * x); /* Sinc filter. */

Resampling

• Used to magnify or minify a sampled signal
• Assume that original sample points are at integer coordinates (0, 1, 2, …)
• Resampling converts the sample points to be located uniformly with an interval $\alpha$
  • For $\alpha>1$ minification (downsampling) takes place
  • For $\alpha<1$ magnification (upsampling) takes place
• Magnification can be solved with the sinc filter
• Minification gets aliased with only the sinc filter and should first have $\text{sinc}(x/\alpha)$ applied before resampling

Screen-Based Antialiasing

• Rapid changing phenomena like shadow boundaries, specular highlights, etc. can produce aliasing artifacts
• Developing anti-aliasing for individual components can be complicated
• Screen-based antialiasing is done on a screen level, and improves results overall
• General strategy is to use more samples per pixel and use a weighted sum

   $$\bm{p}(x,y) = \sum_{i=1}^n w_i \bm{c}(i,x,y)$$ 

  • $n$ is the number of samples for a pixel
  • $\bm{c}(i,x,y)$ is sample function
  • $w_i \in [0,1]$ is a weight

Supersampling

• Using multiple samples to determine pixel values is known as supersampling (or oversampling)
• Is applied to the entire screen leading to higher computational cost
• However, is able to anti-aliasing everything to a certain extent without additional necessary computation

Supersampling Algorithms

• Full-Scene Anti-Aliasing (FSAA) renders the game at a higher resolution (usually either 2x or 4x) and then filters
  • Very computationally efficient
  • Very simple
• Accumulation Buffer uses a single buffer that is rendered to over and over with small displacements and averaged
• Multi-Sampling Anti-Aliasing (MSAA) computes surface’s shade once per pixel and shares among samples
  • Each pixel has multiple samples with independent colour and $z$-depth
  • Pixel shader is evaluated once per fragment at a good position
  • Fewer shader calls and avoids off-triangle problems, but can cause incorrect derivatives

Sampling Patterns

• Humans are most disturbed by aliasing on horizontal or vertical edges (45 degrees is next)
• Stochastic patterns seem to perform really well
• In practice, most GPU manufacturers hardcode $N$-rook based patterns

Morphological Methods

• Since aliasing often results from edges (geometry, shadows, highlights, etc.) it is possible to exploit the structure for antialiasing
• Morphological Antialiasing (MLAA) is done as post-processing
  • Looks at neighbourhoods of pixels and tries to determine orientation of object
  • Buffers such as depths and normals are used to improve results
• Most popular variants are Fast Approximate Antialiasing (FXAA) and Subpixel Morphological Antialiasing (SMAA)

Magnification

• Projecting a texture onto a surface that uses a larger size that the original texture is called magnification
  • Example: a $64 \times 64$ texel image on $256 \times 256$ pixels
• Common algorithms:
  • Nearest neighbour: each pixel uses closest texel value
  • Bilinear: each pixel is interpolated from nearby texels
  • Bicubic: each pixel is computed from a weighted sum of a $4 \times 4$ or $5 \times 5$ array of texels

Nearest and Linear Magnification

• Given $x \in [0,1]$, we want to estimate $y \in [f(0), f(1)]$ with $f(\cdot)$ only defined at $0$ and $1$
• Nearest Neighbour

   $$y = \begin{cases}
  f(0), & \text{if}\;x<0.5 \\
  f(1), & \text{otherwise}
  \end{cases}$$ 

• Linear Interpolation

   $$y = f(0) + \left(f(1) - f(0)\right) \frac{x-0}{1-0} = x\,f(1) + (1-x)\,f(0)$$ 

  • Values match texture when $x=x_{-1}$ or $x=x_{+1}$
  • Smooth but not differentiable at vertices

Hermite and Bicubic Magnification

• Bicubic
  • We also are given points $f(-1), f(2)$ adjacent to the points for interpolation
  • Derived from a cubic polynomial (4 coefficients)
  • Impose constraint on values of $f(0)$ and $f(1)$ and derivative at points $x=0$ and $x=1$
• Hermite Interpolation

   $$y = f(0) + x^2 (3-2x) f(1)$$ 

  • Imposes $f'(0) = f'(1) = 0$
  • Is differentiable everywhere
  • Used in GLSL smoothstep()

Deriving Hermite Interpolation

• Define model: $m(x) = ax^3 + bx^2 + cx + d$
• Define constraints: $m(0)=0, m(1)=1, m'(0)=m'(1)=0$
• Solve:
  • $m(0)=0 \implies d=0$
  • $m'(x) = 3ax^2 + 2bx + c$
  • $m'(0)=0 \implies c=0$
  • $m(1)=1 \implies a+b=1 \implies a=1-b$
  • $m'(1)=0 \implies 3a+2b=0 \implies (3-3b)+2b=0 \implies b=3$
  • $a=1-b \implies a=-2$
• Solution: $m(x) = -2x^3 + 3x^2 = x^2(3-2x)$
• Can do the same thing with a quintic polynomial by imposing $m''(0)=m''(1)=0$

Minification

• When a texture is minimized, several texels may cover a single pixel cell
  • The correct colour should be obtained by integrating over the texels in the pixel
  • Not possible to accurately compute integral in real-time
• Like magnification, different approaches to solve the issue
  • Nearest neighbour: causes aliasing
  • Bilinear: aliasing when more than 4 texels are in the same pixel
  • Mipmapping: precompute smaller textures
  • Summed-Area Table: precompute sum of areas and use that
  • Unconstrained Anisotropic Filtering: back-project pixels onto texture

Minification Aliasing

Mipmapping

• Mip stands for multum in parvo (latin for “many things in a small place”)
• Compute a pyramid of downsampled images with more expensive algorithms
  • At each level, the image is shrunk to 1/4 of the area
  • Performed recursively until image becomes very small or 1 texel
• Objective is then to try to find at least 1:1 pixel-to-texel ratio to achieve the Nyquist rate
  • Usually computed by largest absolute value of $\partial u / \partial x$, $\partial v / \partial x$, $\partial u / \partial y$, and $\partial v / \partial y$
  • Level Of Detail (LOD) bias can also modify the sharpness of a texture
• Main strength: more efficient and high quality minification
• Main issue: overblurring when looking at a textured surface nearly edge-on

Unconstrained Anisotropic Filtering

• Most used approach on current graphics hardware
• Idea is to back-project the pixel cell to the texture
• Multiple mipmap squares are used to cover the quad
• Makes the average area smaller for each mipmap sample
• Longer size of the quad is used to create a line of anisotropy parallel to the longer side

Anisotropic Filtering Concept

Mipmap vs Anisotropic Filtering

Screen-door Transparency

• Render transparent triangle with pixel-aligned checkerboard fill-pattern
• Every other pixel is visible, showing items behind
• Usually pixels are close enough on-screen that the checkerboard is not visible
• Advantages:
  • Very simple
  • Doesn’t require special hardware
• Major drawbacks:
  • Only one transparent object can be rendered at a time
  • Hard to handle non 50% transparency without being noticeable

Stochastic Transparency

• Sub-pixel screen-door masks with stochastic sampling
• Noisy image is created by using random stipple patterns to represent alpha coverage of fragment
• Advantages:
  • No blending is needed
  • Handles antialiasing, transparency, and other phenomena in single framework
• Issues:
  • Large number of samples are needed per pixel to reduce noise
  • Lots of memory needed for sub-pixel samples

Alpha Blending

• Add another channel (alpha) to the textures
• When rendered, the RGB values, $z$ buffer depth, and alpha are associated with each pixel
  • Alpha of 1.0 indicates opaque and 0.0 means completely transparent
• Textures can be loaded with an alpha map to create an RGBA texture
• Especially important in 2D animations and games
• Have to draw opaque objects first, then transparent ones
  • This is due to the limitation of the $z$-buffer which only has one value per pixel

Blending Over

• Transparent object are rendered over existing scene
• Blending is done usually with the over operator:

   $$\bm{c}_o = \alpha_s \bm{c}_s + (1-\alpha_s) \bm{c}_d$$ 

  • $\bm{c}_s$ is the colour of the transparent source object
  • $\alpha_s$ is the alpha of the source object
  • $\bm{d}_s$ is the colour of the destination
  • Issue: not very convincing for 3D scenes
• Additive blending:

   $$\bm{c}_o = \alpha_s \bm{c}_s + \bm{c}_d$$ 

  • Good for lighting effects

Blend Under

• The over operator requires drawing from back to front due to the source transparency limitation
  • Usually $z$-buffer replacement is disabled when drawing transparency
• under operator:

   $$\begin{align}
  \bm{c}_o &= \alpha_d \bm{c}_d + (1-\alpha_d) \bm{c}_s \\
  \alpha_o &= \alpha_s(1-\alpha_d) + \alpha = \alpha_s - \alpha_s\alpha_d + \alpha_d \\
  \end{align}$$ 

  • Is rendered from front to back (opposite of over)
  • Have to maintain alpha value for each pixel
  • All transparent objects are rendered into a buffer with under operator than that is rendered with over on top of the opaque objects

Order-Independent Transparency (OIT)

• Most alpha blending requires sorting objects based on viewport
• Depth peeling allows for order-independent transparency
  • Core idea: use two $z$-buffers and multiple passes
  • First all surface depths are rendered in the first $z$-buffer
  • In second pass all transparent objects are rendered
  • If the $z$-depth of an object matches value in the first $z-buffer$, we know this is the closest transparent object and save RGBA to a colour buffer
  • If the depth value is further away we save the $z$-depth in another buffer (peeling)
  • Swap depth buffers and repeat, this time using the under operator to merge into the colour buffer
  • Stop after a number of passes and render on the opaque image with the over operator

Depth-Peeling Improvements

• Use a pixel counter to count how many pixels where drawn to know when to stop
• Threshold pixels added to pixel counter based on resulting alpha (alphas near 1.0 won’t change anything)
• Duel depth peeling strips off furthest and closest layers each pass reducing rendering passes by half

A-Buffer

• Another OIT approach is the A-buffer (form of multisampling)
• Each triangle rendered creates a coverage mask for each screen grid cell fully or partially covered
• Each pixel stores a list of all relevant fragments
  • Opaque fragments cull fragments behind them (like the $z$-buffer)
  • All fragments are stored for transparent surfaces
  • Usually implemented with linked lists of fragments
• Pixels are rendered by walking through and resolving fragments in a final pass

Weighted Sum

• Single-pass and runs on almost any GPU
• Does not take into account ordering of object
  • Blue gauze on top of red gauze gives purple and not blue with some red showing
• Formula:

   $$\bm{c}_o = \sum_{i=1}^n (\alpha_i \bm{c}_i) + \bm{c}_d (1-\sum_{i=1}^n \alpha_i)$$ 

  • $n$ is number of transparent surfaces, $\bm{c}_i$ and $\alpha_i$ are colour and transparency values, and $\bm{c}_d$ is the opaque portion of the scene
• Issues:
  • First sum saturates
  • Background colour can have a negative effect

Weighted Average

• Solution is to weight the averages:

   $$\begin{align}
  \bm{c}_\text{sum} &= \sum_{i=1}^n \alpha_i \bm{c}_i, \quad
   \alpha_\text{sum} = \sum_{i=1}^n \alpha_i \\
  u &= (1-\alpha_\text{sum}/n)^n \\
  \bm{c}_o &= (1 - u) \frac{\bm{c}_\text{sum}}{\alpha_\text{sum}} + u \bm{c}_d
  \end{align}$$ 

• $\bm{c}_\text{sum}$ and $\alpha_\text{sum}$ are created by transparency rendering
• $u$ is estimated visibility of the destination
• Final computation is the over operator

Premultiplied Alpha and Compositing

• over operator is used for blending together photographs or synthetic renderings knows as compositing
• RGBA values are stored for each pixel
• Common to premultiply alpha to make blending more efficient: $\bm{c}'_s = \alpha_s \bm{c}_s$
• Have to be careful when rendering to framebuffers
  1. Say you have an unmultiplied alpha $\bm{c}, \alpha$
  2. You render it to a transparent framebuffer and obtain the value $\bm{c}' = \bm{c}\alpha, \alpha' = \alpha$
  3. If you then render the framebuffer to screen, it gets multiplied again and the final result is $\bm{c}_o = \bm{c}'\alpha' = \bm{c}\alpha^2$
• OpenGL allows specifying the blend function to modify blending as needed with gl.blendFunc()

Example

void main () {
   vec2 p = (2.0 * gl_FragCoord.xy - u_resolution.xy) / u_resolution.y;
   /* Try playing with values of m. */
   float m = 1.0 / u_resolution.x; /* Compute one pixel in normalized coordinates. */
   float d = sdStar( p, 0.8, 10, 7.0 ); /* Try making this abs(). */
   float b = smoothstep( -m, 0.0, -d ); /* Blend factor. */
   colour_out = vec4( mix( vec3(0.0), vec3(1.0), b ), 1.0 );
}

Transparency Example

void main () {
   vec2 p = (2.0 * gl_FragCoord.xy - u_resolution.xy) / u_resolution.y;
   float m = 1.0 / u_resolution.x;
   float d1 = sdCircle( p+vec2(sin(u_time),0.0), 0.5 );
   float d2 = sdStar( p, 0.8, 10, 7.0 );
   float b1 = smoothstep( -m, 0.0, -d1 ) * 0.5;
   float b2 = smoothstep( -m, 0.0, -d2 );
   colour_out.rgb = mix( vec3(0.0), vec3(0.0,0.0,1.0), b2 );
   colour_out.rgb = mix( colour_out.rgb, vec3(1.0,0.0,0.0), b1 );
   colour_out.a = 1.0; }

Bump Mapping

Normal Mapping

Displacement Mapping