Ok, this is really cool! (This post is a bit technical, but I tried my best to make it accessible to everyone. If something is not clear, please post a question in the comments.)

The Problem

Unfortunately, the bald cap is wrinkling in most shots and has to be completely replaced with a cleaned up CG version. Here’s an example of the wrinkling:

We are going to track the head geometry (based on a 3D scan of Bill Oberst, Jr.) per frame relative to the camera. (The actual head tracking is done by a custom geometry tracker written especially for PRIMITIVE. I will describe it in a future post.) Here’s a screenshot of what the head tracking looks like in Maya:

So all we have do now is render out the clean CG bald cap and comp it over the original, right? Well, almost. Even though we have the camera and the head geometry, we still don’t know what the lighting is! In order to match the lighting in the original shot, I could take one of the following approaches:

1. Match the lighting manually by placing lights around the CG bald cap and adjusting their colors. This would be a very labor intensive process.
2. I actually took still Low Dynamic Range images of a chrome ball on the day of the shoot. Even though these are not true High Dynamic Range images, they could still be used for Image Based Lighting, especially if the clipped highlights are expanded. The only problem is that these images are now so far back in my pipeline that I would have to go digging through old hard drives to find them all. Boring.
3. Write a tool which will estimate the lighting for me.

Guess which one I did?

Image Based Lighting

The tool I wrote is based around the idea of Image Based Lighting (IBL). For the uninitiated, here’s a very short explanation. IBL was introduced by my boss, Dr. Paul Debevec, in a Siggraph 98 paper called Rendering Synthetic Objects into Real Scenes. (I graduated high school that year!) The general idea is that if you’ve captured a High Dynamic Range panoramic environment of your scene, then you can use this environment as a spherical light source in an IBL capable renderer. Here’s an example environment (image from HDRLabs.com):

In IBL, we take this environment image and map it onto the inside of a giant sphere. We can then render objects placed inside this sphere using the environment map as a light source. Here’s an example of Bill’s head geometry lit by this environment:

You can actually see the blue light on his cheek coming from the TV screen.

Reflectance Fields

Reflectance fields were also introduced by Paul Debevec in a Siggraph 2000 paper called Acquiring the Reflectance Field of a Human Face. There’s an excellent explanation of reflectance fields in this short video:

http://people.ict.usc.edu/~debevec/Research/LS/debevec-imagebasedlighting-s2000.mov

The idea is that if you’ve captured or rendered your subject from all possible lighting directions, then it is possible to arbitrarily relight the subject by adding the images together in different proportions. In my case, the reflectance field (also known as a lighting basis) is one of the inputs to my program. I used a simple grid lighting basis, 16×8 pixels, for a total of 128 lighting conditions. Here’s the lighting basis I used:

I literally mapped this sequence of images onto a large sphere and rendered my object with IBL settings turned on in Mental Ray. Here’s the result:

As you can see, this produces 128 images, one image for each pixel in the 16×8 grid. Now let’s take our original environment of the room with the TV and scale it down to 16×8 pixels. Let’s call this the target lightmap.

Now let’s multiply each basis render with it’s corresponding pixel:

Now if we add up all these images together, we will get Bill’s face lit by the room’s environment. Simple, right? Let’s call this the target image:

Inverse Lighting

In the above case, the lighting (the image we called target lightmap) is known. But in the case of the bald cap, the lighting is unknown. Luckily, it is possible to reverse engineer the lighting through a process known as Inverse Lighting. Here’s a good paper about Inverse Lighting:

http://graphics.stanford.edu/~srm/publications/CIC97-invlig.pdf

My tool is simply an implementation of this paper in Python/Numpy. The tool takes as input the target image and the 128 basis renders. Assuming that the lighting in the target image can be reproduced by some combination of the basis images added together in different proportions, all we need to know is what those proportions are. The proportions (coefficients) are, in fact, the colors of the pixels in the estimated lightmap! My tool solves the coefficients (per channel) using a least squares solver and outputs an estimated lightmap, like this one:

As you can see, this lightmap is very close to our target lightmap, especially in areas where we have data, like the front of the face. Areas where we don’t have any data, like the back of the head, are not close, but that doesn’t matter because we will only be rendering from one point of view!

Now let’s compare the original target image rendered by the known ground truth target lightmap:

With the result image rendered by the estimated lightmap produced by my tool:

As you can see, they are almost identical!

Now of course, this test data set is synthetic and therefore these results are ideal. In the real world case of the bald cap there will be error. For example, the target image will be a still of the bald cap (masked out of course) and the 128 basis renders will be of the CG bald cap, which may not be the exact same shape or color as Bill’s head. This will introduce some error, but we can be confident that the tool will produce the best possible lightmap approximation while trying to minimize the error.

So in summary, I just saved myself countless hours of work, if I were to try matching the lighting manually. I’m considering selling this tool. If you’re interested, please contact me.

I’m not sure if this will impress Paul Debevec. But considering that I failed Algebra twice, this will impress my mom.