Wenzel Jakob

Photorealistic images created using physical simulations of light have become a ubiquitous element of our everyday lives. The most successful techniques for producing such images replicate the key physical phenomena in a detailed software simulation, including the emission of light by sources, transport through space, and scattering in the atmosphere and at the surfaces of objects. Mathematically, this computation involves the approximation of many high-dimensional integrals, one for each pixel of the image, usually using Monte Carlo methods. Although a great deal of progress has been made on rendering algorithms, so that physically based rendering is now routinely used in many applications, commonly occurring situations can still cause these algorithms to become impractically slow, forcing users to make unrealistic scene modifications to obtain satisfactory results.

Light transport is complex because light can flow along a great variety of different paths through a scene, though only a subset of these makes relevant contributes to the final image. The simulation becomes ineffective when it is difficult to find the important paths. Commonly occurring materials like smooth metal or glass surfaces can easily lead to such situations, where only very few lighting paths participate, leading to spiky integrands and poor convergence. How to efficiently handle such cases in general has been a long-standing problem.

We present a novel approach to isotropically triangulate or quadrangulate surfaces using a unified local smoothing operator that optimizes both the edge orientations and vertex positions in the output mesh. Our algorithm produces meshes with high isotropy while naturally aligning and snapping edges to sharp features. The method is simple to implement and parallelize, and it can process a variety of input surface representations, such as point clouds, range scans and triangle meshes. Our full pipeline executes instantly (less than a second) on meshes with hundreds of thousands of faces, enabling new types of interactive workflows. Since our algorithm avoids any global optimization, and its key steps scale linearly with input size, we are able to process extremely large meshes and point clouds, with sizes exceeding several hundred million elements. To demonstrate the robustness and effectiveness of our method, we triangulate and quadrangulate hundreds of models of varying complexity and provide our cross-platform reference implementation in the supplemental material.

The simulation of light transport in the presence of multi-bounce glossy effects and motion is challenging because the integrand is high dimensional and areas of high-contribution tend to be narrow and hard to sample. We present a Markov Chain Monte Carlo (MCMC) rendering algorithm that extends Metropolis Light Transport by automatically and explicitly adapting to the local shape of the integrand, thereby increasing the acceptance rate. Our algorithm characterizes the local behavior of throughput in path space using its gradient as well as its Hessian. In particular, the Hessian is able to capture the strong anisotropy of the integrand. We obtain the derivatives using automatic differentiation, which makes our solution general and easy to extend to additional sampling dimensions such as time.

However, the resulting second order Taylor expansion is not a proper distribution and cannot be used directly for importance sampling. Instead, we use ideas from Hamiltonian Monte-Carlo and simulate the Hamiltonian dynamics in a flipped version of the Taylor expansion where gravity pulls particles towards the high-contribution region. Whereas such methods usually require numerical integration, we show that our quadratic landscape leads to a closed-form anisotropic Gaussian distribution for the final particle positions, and it results in a standard Metropolis-Hastings algorithm. Our method excels at rendering glossy-to-glossy reflections on small and highly curved surfaces. Furthermore, unlike previous work that derives sampling anisotropy with pen and paper and only considers specific effects such as specular BSDFs, we characterize the local shape of throughput through automatic differentiation. This makes our approach very general. In particular, our method is the first MCMC rendering algorithm that is able to resolve the anisotropy in the time dimension and render difficult moving caustics.

A layered BSDF model describes the directional reflectance properties of a material whose internal structure consists of a stack of scattering and/or absorbing layers separated by smooth or rough interfaces. The bottom of the stack could be an opaque interface (such as a metal) or a transparent one. Such structural decompositions into layers and interfaces dramatically enlarge the size of the "language" that is available to describe materials, and for this reason they have been the focus of considerable interest in computer graphics in the last years.

In this document, we present layerlab, a Python-based toolbox for computations involving layered materials that implements a model recently proposed by Jakob et al. The purpose of this document is to serve both as a gentle introduction into the underlying theory and as a hands-on tutorial on solving practical layering problems with this tool.

Renderings of animation sequences with physics-based Monte Carlo light transport simulations are exceedingly costly to generate frame-by-frame, yet much of this computation is highly redundant due to the strong coherence in space, time and among samples. A promising approach pursued in prior work entails subsampling the sequence in space, time, and number of samples, followed by image-based spatio-temporal upsampling and denoising.

These methods can provide significant performance gains, though major issues remain: first, in a multiple scattering simulation, the final pixel color is the composite of many different light transport phenomena, and this conflicting information causes artifacts in image-based methods. Secondly, motion vectors are needed to establish correspondence between the pixels in different frames, but it is unclear how to obtain them for most kinds of light paths (e.g. an object seen through a curved glass panel).

To reduce these ambiguities, we propose a general decomposition framework, where the final pixel color is separated into components corresponding to disjoint subsets of the space of light paths. Each component is accompanied by motion vectors and other auxiliary features such as reflectance and surface normals. The motion vectors of specular paths are computed using a temporal extension of manifold exploration and the remaining components use a specialized variant of optical flow. Our experiments show that this decomposition leads to significant improvements in three image-based applications: denoising, spatial upsampling, and temporal interpolation.

Cloth is essential to our everyday lives; consequently, visualizing and rendering cloth has been an important area of research in graphics for decades. One important aspect contributing to the rich appearance of cloth is its complex 3D structure. Volumetric algorithms that model this 3D structure can correctly simulate the interaction of light with cloth to produce highly realistic images of cloth. But creating volumetric models of cloth is difficult: writing specialized procedures for each type of material is onerous, and requires significant programmer effort and intuition. Further, the resulting models look unrealistically "perfect" because they lack visually important features like naturally occurring irregularities.

This paper proposes a new approach to acquiring volume models, based on density data from X-ray computed tomography (CT) scans and appearance data from photographs under uncontrolled illumination. To model a material, a CT scan is made, yielding a scalar density volume. This 3D data has micron resolution details about the structure of cloth but lacks all optical information. So we combine this density data with a reference photograph of the cloth sample to infer its optical properties. We show that this approach can easily produce volume appearance models with extreme detail, and at larger scales the distinctive textures and highlights of a range of very different fabrics such as satin and velvet emerge automatically—all based simply on having accurate mesoscale geometry.

We present a general and practical method for computing BSDFs of layered materials. Its ingredients are transport-theoretical models of isotropic or anisotropic scattering layers and smooth or rough boundaries of conductors and dielectrics. Following expansion into a directional basis that supports arbitrary composition, we are able to efficiently and accurately synthesize BSDFs for a great variety of layered structures.

Reflectance models created by our system correctly account for multiple scattering within and between layers, and in the context of a rendering system they are efficient to evaluate and support texturing and exact importance sampling. Although our approach essentially involves tabulating reflectance functions in a Fourier basis, the generated models are compact to store due to the inherent sparsity of our representation, and are accurate even for narrowly peaked functions. While methods for rendering general layered surfaces have been investigated in the past, ours is the first system that supports arbitrary layer structures while remaining both efficient and accurate.

We validate our model by comparing to measurements of real-world examples of layered materials, and we demonstrate an interactive visual design tool that enables easy exploration of the space of layered materials.

This paper investigates rendering glittery surfaces, ones which exhibit shifting random patterns of glints as the surface or viewer moves. It applies both to dramatically glittery surfaces that contain mirror-like flakes and also to rough surfaces that exhibit more subtle small scale glitter, without which most glossy surfaces appear too smooth in close-up. These phenomena can in principle be simulated by high-resolution normal maps, but maps with tiny features create severe aliasing problems under narrow-angle illumination. In this paper we present a stochastic model for the effects of random subpixel structures that generates glitter and spatial noise that behave correctly under different illumination conditions and viewing distances, while also being temporally coherent so that they look right in motion. The model is based on microfacet theory, but it replaces the usual continuous microfacet distribution with a discrete distribution of scattering particles on the surface. A novel stochastic hierarchy allows efficient evaluation in the presence of large numbers of random particles, without ever having to consider the particles individually. This leads to a multiscale procedural BRDF that is readily implemented in standard rendering systems, and which converges back to the smooth case in the limit.

Rendering a complex specular surface under sharp point lighting is far from easy. Using Monte Carlo point sampling for this purpose is impractical: the energy is concentrated in tiny highlights that take up a minuscule fraction of the pixel. We instead compute the accurate solution that Monte Carlo would eventually converge to, using a completely different deterministic approach with minimal approximations. Our method considers the true distribution of normals on a surface patch seen through a single pixel, which can be highly complicated. This requires computing the probability density of the given normal coming from anywhere on the patch. We show how to evaluate this efficiently, assuming a Gaussian surface patch and Gaussian intrinsic roughness. We also take advantage of hierarchical pruning of position-normal space to quickly find texels that might contribute to a given normal distribution evaluation. Our results show complicated, temporally varying glints from materials such as bumpy plastics, brushed and scratched metals, metallic paint and ocean waves.

It is a long-standing problem in unbiased Monte Carlo methods for rendering that certain difficult types of light transport paths, particularly those involving viewing and illumination along paths containing specular or glossy surfaces, cause unusably slow convergence. In this paper we introduce Manifold Exploration, a new way of handling specular paths in rendering. It is based on the idea that sets of paths contributing to the image naturally form manifolds in path space, which can be explored locally by a simple equation-solving iteration. This paper shows how to formulate and solve the required equations using only geometric information that is already generally available in ray tracing systems, and how to use this method in in two different Markov Chain Monte Carlo frameworks to accurately compute illumination from general families of paths. The resulting rendering algorithms handle specular, near-specular, glossy, and diffuse surface interactions as well as isotropic or highly anisotropic volume scattering interactions, all using the same fundamental algorithm. An implementation is demonstrated on a range of challenging scenes and evaluated against previous methods.

Woven fabrics have a wide range of appearance determined by their small-scale 3D structure. Accurately modeling this structural detail can produce highly realistic renderings of fabrics and is critical for predictive rendering of fabric appearance. But building these yarn-level volumetric models is challenging. Procedural techniques are manually intensive, and fail to capture the naturally arising irregularities which contribute significantly to the overall appearance of cloth. Techniques that acquire the detailed 3D structure of real fabric samples are constrained only to model the scanned samples and cannot represent different fabric designs.

This paper presents a new approach to creating volumetric models of woven cloth, which starts with user-specified fabric designs and produces models that correctly capture the yarn-level structural details of cloth. We create a small database of volumetric exemplars by scanning fabric samples with simple weave structures. To build an output model, our method synthesizes a new volume by copying data from the exemplars at each yarn crossing to match a weave pattern that specifies the desired output structure. Our results demonstrate that our approach generalizes well to complex designs and can produce highly realistic results at both large and small scales.

State-of-the-art density estimation methods for rendering participating media rely on a dense photon representation of the radiance distribution within a scene. A critical bottleneck of such kernel-based approaches is the excessive number of photons that are required in practice to resolve fine illumination details, while controlling the amount of noise. In this paper, we propose a parametric density estimation technique that represents radiance using a hierarchical Gaussian mixture. We efficiently obtain the coefficients of this mixture using a progressive and accelerated form of the Expectation-Maximization algorithm. After this step, we are able to create noise-free renderings of high-frequency illumination using only a few thousand Gaussian terms, where millions of photons are traditionally required. Temporal coherence is trivially supported within this framework, and the compact footprint is also useful in the context of real-time visualization. We demonstrate a hierarchical ray tracing-based implementation, as well as a fast splatting approach that can interactively render animated volume caustics.

The appearance of complex, thick materials like textiles is determined by their 3D structure, and they are incompletely described by surface reflection models alone. While volume scattering can produce highly realistic images of such materials, creating the required volume density models is difficult. Procedural approaches require significant programmer effort and intuition to design specialpurpose algorithms for each material. Further, the resulting models lack the visual complexity of real materials with their naturallyarising irregularities.

This paper proposes a new approach to acquiring volume models, based on density data from X-ray computed tomography (CT) scans and appearance data from photographs under uncontrolled illumination. To model a material, a CT scan is made, resulting in a scalar density volume. This 3D data is processed to extract orientation information and remove noise. The resulting density and orientation fields are used in an appearance matching procedure to define scattering properties in the volume that, when rendered, produce images with texture statistics that match the photographs. As our results show, this approach can easily produce volume appearance models with extreme detail, and at larger scales the distinctive textures and highlights of a range of very different fabrics like satin and velvet emerge automatically -- all based simply on having accurate mesoscale geometry.

We propose a novel system for designing and manufacturing surfaces that produce desired caustic images when illuminated by a light source. Our system is based on a nonnegative image decomposition using a set of possibly overlapping anisotropic Gaussian kernels. We utilize this decomposition to construct an array of continuous surface patches, each of which focuses light onto one of the Gaussian kernels, either through refraction or reflection. We show how to derive the shape of each continuous patch and arrange them by performing a discrete assignment of patches to kernels in the desired caustic. Our decomposition provides for high fidelity reconstruction of natural images using a small collection of patches. We demonstrate our approach on a wide variety of caustic images by manufacturing physical surfaces with a small number of patches.

The radiative transfer framework that underlies all current rendering of volumes is limited to scattering media whose properties are invariant to rotation. Many systems allow for "anisotropic scattering," in the sense that scattered intensity depends on the scattering angle, but the standard equation assumes that the structure of the medium is isotropic. This limitation impedes physics-based rendering of volume models of cloth, hair, skin, and other important volumetric or translucent materials that do have anisotropic structure. This paper presents an end-to-end formulation of physics-based volume rendering of anisotropic scattering structures, allowing these materials to become full participants in global illumination simulations.

We begin with a generalized radiative transfer equation, derived from scattering by oriented non-spherical particles. Within this framework, we propose a new volume scattering model analogous to the well-known family of microfacet surface reflection models; we derive an anisotropic diffusion approximation, including the weak form required for finite element solution and a way to compute the diffusion matrix from the parameters of the scattering model; and we also derive a new anisotropic dipole BSSRDF for anisotropic translucent materials. We demonstrate results from Monte Carlo, finite element, and dipole simulations. All these contributions are readily implemented in existing rendering systems for volumes and translucent materials, and they all reduce to the standard practice in the isotropic case.

Hair models for computer graphics consist of many curves representing individual hair fibers. In current practice these curves are generated by ad hoc random processes, and in close-up views their arrangement appears plainly different from real hair. To begin improving this situation, this paper presents a new method for measuring the detailed arrangement of fibers in a hair assembly. Many macrophotographs with shallow depth of field are taken of a sample of hair, sweeping the plane of focus through the hair's volume. The shallow depth of field helps isolate the fibers and reduces occlusion. Several sweeps are performed with the hair at different orientations, resulting in multiple observations of most of the clearly visible fibers. The images are filtered to detect the fibers, and the resulting feature data from all images is used jointly in a hair growing process to construct smooth curves along the observed fibers. Finally, additional hairs are generated to fill in the unseen volume inside the hair. The method is demonstrated on both straight and wavy hair, with results suitable for realistic close-up renderings. These models provide the first views we know of into the 3D arrangement of hair fibers in real hair assemblies.

Recently, Heitz and D'Eon proposed a method for importance sampling the distribution of visible normals in the context of microfacet BSDF models. One of their sampling routines internally relies on a discontinuous mapping, which can cause problems in conjunction with Quasi Monte Carlo sampling and Markov Chain Monte Carlo integration. In this report, we develop an alternative method that does not have this drawback.

We present a general and practical method for computing BSDFs of layered materials. Its ingredients are transport-theoretical models of isotropic or anisotropic scattering layers and smooth or rough boundaries of conductors and dielectrics. Following expansion into a directional basis that supports arbitrary composition, we are able to efficiently and accurately synthesize BSDFs for a great variety of layered structures.

Reflectance models created by our system correctly account for multiple scattering within and between layers, and in the context of a rendering system they are efficient to evaluate and support texturing and exact importance sampling. Although our approach essentially involves tabulating reflectance functions in a Fourier basis, the generated models are compact to store due to the inherent sparsity of our representation, and are accurate even for narrowly peaked functions. While methods for rendering general layered surfaces have been investigated in the past, ours is the first system that supports arbitrary layer structures while remaining both efficient and accurate.

We validate our model by comparing to measurements of real-world examples of layered materials, and we demonstrate an interactive visual design tool that enables easy exploration of the space of layered materials.

The von Mises-Fisher (vMF) distribution is a popular all-purpose distribution for statistical inference involving directional data. Many basic operations involving this distribution are prone to severe numerical issues when implemented in finite precision computer arithmetic. There is a surprising lack on information on how these can be circumvented, and hence the purpose of this document is to serve as a collection of numerically-well behaved recipes for common operations.

It is a long-standing problem in unbiased Monte Carlo methods for rendering that certain difficult types of light transport paths, particularly those involving viewing and illumination along paths containing specular or glossy surfaces, cause unusably slow convergence. In this paper we introduce Manifold Exploration, a new way of handling specular paths in rendering. It is based on the idea that sets of paths contributing to the image naturally form manifolds in path space, which can be explored locally by a simple equation-solving iteration. This paper shows how to formulate and solve the required equations using only geometric information that is already generally available in ray tracing systems, and how to use this method in in two different Markov Chain Monte Carlo frameworks to accurately compute illumination from general families of paths. The resulting rendering algorithms handle specular, near-specular, glossy, and diffuse surface interactions as well as isotropic or highly anisotropic volume scattering interactions, all using the same fundamental algorithm. An implementation is demonstrated on a range of challenging scenes and evaluated against previous methods.

The radiative transfer framework that underlies all current rendering of volumes is limited to scattering media whose properties are invariant to rotation. Many systems allow for "anisotropic scattering," in the sense that scattered intensity depends on the scattering angle, but the standard equation assumes that the structure of the medium is isotropic. This limitation impedes physics-based rendering of volume models of cloth, hair, skin, and other important volumetric or translucent materials that do have anisotropic structure. This paper presents an end-to-end formulation of physics-based volume rendering of anisotropic scattering structures, allowing these materials to become full participants in global illumination simulations.

We begin with a generalized radiative transfer equation, derived from scattering by oriented non-spherical particles. Within this framework, we propose a new volume scattering model analogous to the well-known family of microfacet surface reflection models; we derive an anisotropic diffusion approximation, including the weak form required for finite element solution and a way to compute the diffusion matrix from the parameters of the scattering model; and we also derive a new anisotropic dipole BSSRDF for anisotropic translucent materials. We demonstrate results from Monte Carlo, finite element, and dipole simulations. All these contributions are readily implemented in existing rendering systems for volumes and translucent materials, and they all reduce to the standard practice in the isotropic case.

The pervasive use of computer-generated graphics in our society has led to strict demands on their visual realism. Generally, users of rendering software want their images to look, in various ways, “real”, which has been a key driving force towards methods that are based on the physics of light transport.

Until recently, industrial practice has relied on a different set of methods that had comparatively little rigorous grounding in physics—but within the last decade, advances in rendering methods and computing power have come together to create a sudden and dramatic shift, in which physics-based methods that were formerly thought impractical have become the standard tool. As a consequence, considerable attention is now devoted towards making these methods as robust as possible.

In this context, robustness refers to an algorithm's ability to process arbitrary input without large increases of the rendering time or degradation of the output image. One particularly challenging aspect of robustness entails simulating the precise interaction of light with all the materials that comprise the input scene. This dissertation focuses on one specific group of materials that has fundamentally been the most important source of difficulties in this process.

Specular materials, such as glass windows, mirrors or smooth coatings (e.g. on finished wood), account for a significant percentage of the objects that surround us every day. It is perhaps surprising, then, that it is not well-understood how they can be accommodated within the theoretical framework that underlies some of the most sophisticated rendering methods available today.

Many of these methods operate using a theoretical framework known as path space integration. But this framework makes no provisions for specular materials: to date, it is not clear how to write down a path space integral involving something as simple as a piece of glass.

Although implementations can in practice still render these materials by side-stepping limitations of the theory, they often suffer from unusably slow convergence; improvements to this situation have been hampered by the lack of a thorough theoretical understanding.

We address these problems by developing a new theory of path-space light transport which, for the first time, cleanly incorporates specular scattering into the standard framework. Most of the results obtained in the analysis of the ideally smooth case can also be generalized to rendering of glossy materials and volumetric scattering so that this dissertation also provides a powerful new set of tools for dealing with them.

The basis of our approach is that each specular material interaction locally collapses the dimension of the space of light paths so that all relevant paths lie on a submanifold of path space. We analyze the high-dimensional differential geometry of this submanifold and use the resulting information to construct an algorithm that is able to “walk” around on it using a simple and efficient equation-solving iteration.

This manifold walking algorithm then constitutes the key operation of a new type of Markov Chain Monte Carlo (MCMC) rendering method that computes lighting through very general families of paths that can involve arbitrary combinations of specular, near-specular, glossy, and diffuse surface interactions as well as isotropic or highly anisotropic volume scattering. We demonstrate our implementation on a range of challenging scenes and evaluate it against previous methods.

Even with the processing power of today’s computers, the calculation of global illumination solutions for general scenes remains a lengthy undertaking. This project investigates the possibility of accelerating such calculations using a dedicated hardware co-processor. The algorithm of choice, bidirectional path tracing, presents an additional challenge, since its three-dimensional random walks make the use of coherent ray tracing techniques impracticable. A parallel and deeply pipelined ray-triangle intersection processor including a kd-tree traversal unit was developed during the course of the project, and numerous benchmarks were performed.