I am only a layman but have read this and related posts carefully several times to try and understand the difficulties of the problem and the ingenuity of the solution. I am currently reading Jonah Berg’s “Imagine: How Creativity Works” and I wonder if either Oleg Alexander or William Lambeth approached this problem by stepping away from it and daydreaming, or did a more ‘constant-focus-on-the-problem’ process lead to the solution?

I’ll try to answer this excellent question in this post.

I devote a lot of thought to the subject of creativity. I’m constantly trying to stay conscious of and to streamline my own creative process. So far I’ve identified two kinds of creative problems: Puzzles and Wicked Problems.

Puzzles

Think of your favorite puzzle games. Like jigsaw puzzles, tangrams, untangle puzzles, unblock puzzles, matchstick puzzles, Cut the Rope, etc. Solving puzzles is a creative process. Puzzles are “easy” problems because they are well defined. For example, in tangrams, you must match a given silhouette using all 7 tangram pieces. Puzzles usually have a structure like: do X, given the constraints Y. (In computer science, puzzles are called constraint satisfaction problems.) The constraints are the key to a well defined puzzle problem. Just think of other problems which are not usually referred to as puzzles, but which have the same structure. For example, creative writing assignments. Like, describe one of your family members, but the number of words you can use must match this family member’s age. Or Pictionary/Charades, describe a phrase by drawing it or acting it out. Improv/Who’s Line Is It Anyway fall into the same category.

Puzzles are solved by trial and error (aka optimization in computer science). It is rare that one solves a puzzle problem through one grand “inspiration”. It’s more like hundreds of tiny inspirations, each of which moves you closer to the optimal solution.

The bald cap wrinkling problem is an example of a puzzle problem. It is very well defined: Get rid of the goddamn wrinkles!

Now I will describe the creative process of solving the bald cap wrinkling issue as best as I remember it.

The first thing I tried was simple 2D tricks like using blur or median filters on the wrinkles. That looked like shit, so then I knew that only way to get rid of the wrinkles was to replace the bald cap completely with a CG one. This meant two things: the head would have to be matchmoved and the lighting would have to match. Luckily, the matchmoving was going to happen anyway for a different reason: to warp the head to Neanderthal proportions. So the matchmoving pipeline was already developed and work on matchmoving was already in progress.

This left only the lighting issue. I made a list of possible ways to match the lighting:

1. Manually (Too labor intensive.)
2. Dig through old hard drives to find the chrome ball lighting reference from the shoot. (Too boring.)
3. Optimize the lighting to match a target image. (Sounds like fun!)

I didn’t know exactly how I was gonna optimize the lighting, but I knew it was possible. After consulting with my PhD math wiz friends and coworkers and doing a few Google searches for things like “reverse engineer lighting”, I came upon “inverse lighting”, the technical term for what I was trying to do. The papers on inverse lighting basically confirmed the approach I was about to take: to solve the lighting using linear least squares. This kind of thing happens to me all the time, I “invent” a solution to a problem because I know it’s possible, and then do a Google search to find out what this problem is officially called and what potential solutions there are in the literature.

So I prototyped the oa Match Lighting tool and it worked! (Needless to say, the tool went through several iterations of its own, which I won’t go into here.) Now I had a viable solution for matching the lighting. All the pieces were in place. Time to do an actual test frame.

Next came the familiar iterative process of “tweaking”. When you’re tweaking, you know you’re almost at the end! There are usually no new surprise variables, you’re just tweaking existing variables. But as you’ll see, tiny inspirations can come even at the tweaking stage. I’ll post images here of the different iterations.

Here’s the raw problem image again, to keep the context.

Here’s the first iteration of the CG bald cap.

Looking at the first shitty iteration, you might think that William Lambeth and I were worried. But we were not, because we knew that in a few more iterations we’ll get to a good place. William touched up the textures a bit more and this was the second iteration.

A bit better, but William still felt that the bald spot was not shiny enough. That it was too matte. It certainly didn’t match the greasy highlight on the nose. I explained to him that the problem was not with the shininess of the material on the CG bald cap, but a bigger problem of the lighting itself being too diffuse. The reason for this was that the material of the bald cap in the original raw image was already matte! Therefore any lighting recovered from a matte surface will also be very diffuse. “There’s nothing we can do about it”, I said. Luckily, William refused to accept my “logic” and sent me the following image in which he added a fake highlight in Photoshop.

After seeing this image, I realized that he was absolutely right: the bald spot must be made shinier. But how? It was at this moment that I had a flash of inspiration. If William can paint a fake highlight on the final image, then why can’t I paint a fake highlight on the target image for the lighting tool to match? I knew in that instance that adding a fake highlight to the target image would translate into my tool creating a bright “sun” light in the lightmap! So I tried it and of course it worked! So much for “nothing we can do about it”. Here’s the target image I used with an added fake highlight.

And here’s the final iteration we ended up with!

It is a fact that anything I post on this blog has gone through a similar creative process, and you only get to see the final result. But of course the fun is in the creative process itself!

Wicked Problems

Wicked problems are creative problems that are much harder that puzzles. What makes them hard is that in addition to not knowing what the solution is, you also don’t even know what the problem is! An example would be trying to come up with an idea for a screenplay. Where do you start? Anything goes so the solution space is infinite.

Well, I like to think that the best place to start is with some constraints. Like nailing down the genre or theme. Maybe trying random configurations of genre and theme. Like “hazing…in space”. In other words, before a wicked problem can be solved, it must first be converted into a puzzle, a well defined problem. And it is VERY difficult to design a good puzzle!

No one has a magic formula for creativity. But currently I try to use the following formula:

CREATIVITY = CONSTRAINTS + RANDOMNESS

Let me end with a quote from a Cinefex article about Avatar. When designing the world of Pandora, James Cameron said:

When you’ve got all these possibilities, when you can do any kind of action imaginable, you have to be very disciplined. Applying rigor and discipline to the process has been the biggest challenge, in fact. Early on, we came up with the principle of denying ourselves infinite possibility–which sounds wrong. You’d think you’d want to embrace the infinite possibility; but you don’t because you’ll never get there. Ever. We stood by the principle of making a creative decision in the moment, and never second-guessing it. And just by making that decision, we had eliminated possibility. Every single day was about eliminating possibility.

Before:

After:

My inverse lighting tool (which is now called oa Match Lighting) worked like a charm! I used a blurred version of the original frame as my target image. I also added a fake highlight to make the head shinier. Pretty cool that I can just “paint” the kind of lighting I want and my lighting tool will match it! Here’s the target image I used:

Here’s the closest match preview from my tool:

And here’s the lightmap that my tool generated. I used this lightmap to render the bald cap model at high res. Then composited the render over the original frame.

Special thanks to William Lambeth for creating the model and textures for the bald cap. His input on this issue has been invaluable.

Comments welcome!

]]> http://olegalexander.com/?feed=rss2&p=1275 1 http://olegalexander.com/?p=1239 http://olegalexander.com/?p=1239#comments Mon, 30 Jan 2012 02:50:48 +0000 olegalexander http://olegalexander.com/?p=1239 Ok, this is really cool! (This post is a bit technical, but I tried my best to make it accessible to everyone. If something is not clear, please post a question in the comments.)

The Problem

Unfortunately, the bald cap is wrinkling in most shots and has to be completely replaced with a cleaned up CG version. Here’s an example of the wrinkling:

We are going to track the head geometry (based on a 3D scan of Bill Oberst, Jr.) per frame relative to the camera. (The actual head tracking is done by a custom geometry tracker written especially for PRIMITIVE. I will describe it in a future post.) Here’s a screenshot of what the head tracking looks like in Maya:

So all we have do now is render out the clean CG bald cap and comp it over the original, right? Well, almost. Even though we have the camera and the head geometry, we still don’t know what the lighting is! In order to match the lighting in the original shot, I could take one of the following approaches:

1. Match the lighting manually by placing lights around the CG bald cap and adjusting their colors. This would be a very labor intensive process.
2. I actually took still Low Dynamic Range images of a chrome ball on the day of the shoot. Even though these are not true High Dynamic Range images, they could still be used for Image Based Lighting, especially if the clipped highlights are expanded. The only problem is that these images are now so far back in my pipeline that I would have to go digging through old hard drives to find them all. Boring.
3. Write a tool which will estimate the lighting for me.

Guess which one I did?

Image Based Lighting

The tool I wrote is based around the idea of Image Based Lighting (IBL). For the uninitiated, here’s a very short explanation. IBL was introduced by my boss, Dr. Paul Debevec, in a Siggraph 98 paper called Rendering Synthetic Objects into Real Scenes. (I graduated high school that year!) The general idea is that if you’ve captured a High Dynamic Range panoramic environment of your scene, then you can use this environment as a spherical light source in an IBL capable renderer. Here’s an example environment (image from HDRLabs.com):

In IBL, we take this environment image and map it onto the inside of a giant sphere. We can then render objects placed inside this sphere using the environment map as a light source. Here’s an example of Bill’s head geometry lit by this environment:

You can actually see the blue light on his cheek coming from the TV screen.

Reflectance Fields

Reflectance fields were also introduced by Paul Debevec in a Siggraph 2000 paper called Acquiring the Reflectance Field of a Human Face. There’s an excellent explanation of reflectance fields in this short video:

http://people.ict.usc.edu/~debevec/Research/LS/debevec-imagebasedlighting-s2000.mov

The idea is that if you’ve captured or rendered your subject from all possible lighting directions, then it is possible to arbitrarily relight the subject by adding the images together in different proportions. In my case, the reflectance field (also known as a lighting basis) is one of the inputs to my program. I used a simple grid lighting basis, 16×8 pixels, for a total of 128 lighting conditions. Here’s the lighting basis I used:

I literally mapped this sequence of images onto a large sphere and rendered my object with IBL settings turned on in Mental Ray. Here’s the result:

As you can see, this produces 128 images, one image for each pixel in the 16×8 grid. Now let’s take our original environment of the room with the TV and scale it down to 16×8 pixels. Let’s call this the target lightmap.

Now let’s multiply each basis render with it’s corresponding pixel:

Now if we add up all these images together, we will get Bill’s face lit by the room’s environment. Simple, right? Let’s call this the target image:

Inverse Lighting

In the above case, the lighting (the image we called target lightmap) is known. But in the case of the bald cap, the lighting is unknown. Luckily, it is possible to reverse engineer the lighting through a process known as Inverse Lighting. Here’s a good paper about Inverse Lighting:

http://graphics.stanford.edu/~srm/publications/CIC97-invlig.pdf

My tool is simply an implementation of this paper in Python/Numpy. The tool takes as input the target image and the 128 basis renders. Assuming that the lighting in the target image can be reproduced by some combination of the basis images added together in different proportions, all we need to know is what those proportions are. The proportions (coefficients) are, in fact, the colors of the pixels in the estimated lightmap! My tool solves the coefficients (per channel) using a least squares solver and outputs an estimated lightmap, like this one:

As you can see, this lightmap is very close to our target lightmap, especially in areas where we have data, like the front of the face. Areas where we don’t have any data, like the back of the head, are not close, but that doesn’t matter because we will only be rendering from one point of view!

Now let’s compare the original target image rendered by the known ground truth target lightmap:

With the result image rendered by the estimated lightmap produced by my tool:

As you can see, they are almost identical!

Now of course, this test data set is synthetic and therefore these results are ideal. In the real world case of the bald cap there will be error. For example, the target image will be a still of the bald cap (masked out of course) and the 128 basis renders will be of the CG bald cap, which may not be the exact same shape or color as Bill’s head. This will introduce some error, but we can be confident that the tool will produce the best possible lightmap approximation while trying to minimize the error.

So in summary, I just saved myself countless hours of work, if I were to try matching the lighting manually. I’m considering selling this tool. If you’re interested, please contact me.

I’m not sure if this will impress Paul Debevec. But considering that I failed Algebra twice, this will impress my mom.

• I fixed most of the vertical parallax issues in my stereo footage.

• I finished the alpha cleanup on all the shots.

Vertical Parallax Fix

You may remember that PRIMITIVE was shot with the wrong beam splitter rig. This was my fault. I originally bought the cheaper version of the beam splitter rig from 3dfilmfactory, because I never imagined that I could later afford shooting with 2 RED cameras. The beam splitter rig that I got was never designed to hold 2 REDs, but somehow my DP managed to shove 2 REDs in there anyway. The result was a vertically misaligned 3D camera. So, I knew going in that I was gonna have vertical parallax problems. What I didn’t know was how big a deal it would be to fix this in post.

Here’s an example shot with really severe vertical parallax problems. Not all shots were this bad. The red lines connect corresponding points in the left and right images. Ideally, these lines are supposed to be completely horizontal. Notice that the problem can’t be fixed with a simple 2D shift because the vertical parallax is different for points close to the camera and points far from the camera. Looking at this image in 3D would give you a major headache because in order to achieve stereopsis your eyes would have to diverge vertically!

Here is the same shot with my vertical parallax fix applied. Notice how the same lines are almost horizontal now, which means that viewing this shot in 3D will be a pleasant experience.

So how does it work? Well, I tried several different “obvious” approaches, all of which failed. I became discouraged by the possibility that I would have to fall back to making the film in 2D. Then I said: “Let me try one last thing. If this doesn’t work, nothing will.” The idea I came up with was to convert each frame of the film into a low res 3D scan, like an animated “rubber sheet”. Then rerender the left view from a corrected position. Here’s a screenshot revealing the magic trick.

I won’t go into the highly technical implementation details. Needless to say, it took me months to write the custom computer vision software required for this to work. Luckily, it did work and the film is still going forward in 3D! And by the way, lesson learned. Next time I’ll get it right in camera!

Alpha Cleanup

Whereas fixing the vertical parallax was a creative challenge, cleaning up the alpha channels was a monotonous, never-ending hell. The task was commonplace: isolate the subject from the background using a combination of chroma key, paint, and rotoscoping. Here’s an example shot.

And here is the cleaned up alpha channel. Having this alpha channel allows us to composite the kids over a new background, like a sky.

Most of the shots were easy because all I had to do was chroma key the blue screen. Sometimes the chroma key had a few holes in it which I filled with a paint brush. But sometimes there would be severe blue spill from the cliff onto the Neanderthal’s body. Or sometimes the Neanderthal’s fingers would go outside the blue screen. In such cases I had to resort to my new mortal enemy: rotoscoping. Here’s an example of a particularly evil roto shot. This shot alone took me weeks to complete.

The thing is, I fully anticipated having to do this kind of work even before shooting. But I highly underestimated how long roto actually takes. And I overestimated how much free help I would get from others. I ended up doing most of this work myself, with some help from two other people. So, another hard lesson learned: next time allocate money in the budget for roto! In fact, I would recommend this to any independent filmmaker: Allocate money in the budget for all non-creative tasks. I’m just glad it’s over.

Next Steps

The next steps are as follows:

• 3D track the Neanderthal’s head geometry. This will allow us to warp Bill Oberst’s head to Neanderthal proportions. This head warping effect will be what makes or breaks this film, and it could still go either way! We’ve developed a badass custom pipeline for the head tracking and I can’t wait to use it.

• Fix the bald cap wrinkling issues. The head tracking will help with that, too.

• Model, animate, and render props, like the nest and a more menacing branch.

• Finish 3D environments. This is already progressing nicely.

• Finish sound design. Can be completed in parallel with the VFX work.

• Composite all the elements!

]]> http://olegalexander.com/?feed=rss2&p=1200 2 http://olegalexander.com/?p=1179 http://olegalexander.com/?p=1179#comments Sun, 05 Jun 2011 02:44:45 +0000 olegalexander http://olegalexander.com/?p=1179 Don’t miss this excellent lecture about human computation by Luis Von Ahn! You can also play the games here. ESP game is my favorite

This was my first ADR session as a director and I learned a lot! Big thanks to everyone who made this session possible!

Please pass it on…

http://www.hartford-hwp.com/archives/24/042.html

And here is a related TED talk on why societies collapse by Jared Diamond:

Related: Earth 2100. A possible near future featuring interviews with some very credible people. This show has high production value and is not what you might expect from a typical doomsday scenario show. It even offers a glimmer of hope at the end. Watch all 9 parts:

http://www.youtube.com/watch?v=bISCbOUozMk

1. NVidia 3D Vision Kit. One pair of active shutter glasses and infrared emitter. $200.
2. Samsung SyncMaster 2233RZ 120Hz LCD monitor. $200 on Ebay.
3. Stereoscopic Player software. Free trial allows up to 5 minutes of playback at a time, which is long enough to check shots.

To check a shot in 3D, all you have to do is render out a jpg sequence in the over/under format, with the left view on top and the right view on the bottom. (The shot has already been converged, usually on the subject’s eyes.) Like this:

Then convert the jpg sequence into an avi using ffmpeg. Like this:

ffmpeg -i F:\SUNSET_0100_N\topBottom2048.%04d.jpg -y -qscale 1 -r 24 F:\SUNSET_0100_N\topBottom2048.avi

Finally, open the avi in the Stereoscopic Player, set the viewing method to NVIDIA 3D Vision, and go to full screen mode. Put on your 3D glasses and enjoy!

There are a couple of other nice things about this setup:

• I can play any PC game in 3D! Nothing beats climbing the rooftops of Jerusalem in Assassin’s Creed in 3D.
• The Stereoscopic Player supports the Fujifilm FinePix W1 MPO format, which means I can view all the pictures I’ve taken with the W1 in 3D.

So, after having seen my shots in 3D, do I still think it was worth the trouble to make PRIMITIVE in 3D? Fuck yeah!