June 4, 2007
Madame Tutli-Putli is a remarkable work for many reasons, but the one that many (including me) have seized on is the one that continues the tradition of combining live-action and animation: the "gimmick" of compositing live actors' eyes onto the movies' stop-motion puppets. (The image to the left is Laurie Maher—Madame Tutli-Putli's co-creator, as it were—providing a reaction shot.) It is, perhaps, a form of motion capture that Robert Zemeckis and company never dreamed of.
That bit of trickery was achieved by Montreal-based painter/animator/compositor Jason Walker, who dropped me a line earlier today and pointed me to his website, which provides a glimpse into the process of capturing actors' ocular performances and then matching them up seamlessly. Still, after poking around I found I needed more, so I went straight to the source.
Jason Walker: When I'm painting a portrait, one of the priorities, to me, is creating an exact likeness of the subject. Painting a human face requires a great deal of accuaracy. If any feature is even slightly off, you have a different person. The placement of eyes, especially, are key to making a person look human.
For Madame Tutli-Putli I placed, rotated, and scaled each eye individually, by the pixel, to make sure that Tutli's character was consistent through the four years of compositing. I used the same approach as painting a likeness. Tutli's eyes were also slightly stretched vertically to give her more of a sympathetic look.
ET: You're a little vague in your description of your process for matching human eyes to stop-motion actors. You used "every trick in the book and more," as you put it. Can you give us a detailed breakdown of one of those tricks?
JW: Vague, you say! Okay, you asked for it!
First of all, this technique of adding human eyes to stop-motion puppets is extremely complicated and starts long before the compositing stage. I will be adding more information on the technique to my website once Tutli has had time be in a festival or two before showing her "un-masked."
We decided that seeing Tutli with no eyes, and just the silicone puppet, is quite shocking and should be held back. The technique itself is a system I came up with back in 2003 when Tutli was still in talks with the NFB. I had worked on post-production effects for Clyde Henry Productions for several years before Tutli-Putli, and this was simply our latest collaboration and a chance to try a new challenge.
I would start by applying makeup to an actor. For Tutli's character this makeup would evolve throughout the film. With the "Pervert" character, and the small boy, I would add texture to the actor's skin to resemble the puppet they were to inhabit. I would then choreograph the actor's moves based on a chart that I would create. A "Wunderbar," as they became known. This was my way of breaking down the moves that the stop-motion puppet was making.
This timeline for each eye shot in the film would also indicate every time there would be a light flash or shadow pass on the puppet. With this puppet's actions indicated in colours, I could teach the actor to replicate the head moves. Once the moves were rehearsed, Chris and Maciek could then direct the actors for the context of the scene while I would call out the moves, and light flashes, shadows, etc.
The actors were incredible at learning their choreographed moves, and giving great acting performances. We had Laurie Maher (Tutli-Putli) cry for two long days of eye takes for the dining car scenes.
Depending on the complexity of the shot, we would film between 15-20 takes of eyes for each scene supplying a range of acting, and a varying degree of head angles. Chris and Maciek would then review the takes and make a final decision based only on acting, and then I would import the take, and try a quick test on the puppet footage to see if I could make the timing and moves work. Very few takes had to be discarded, so I got the angles I needed, and Chris and Maciek got the acting take they wanted.
Matching the eyes to the puppet footage presented many unique problems. When you film a person going through their moves, it never matches up... ever! The timing is way off, no matter how hard you try. So I decided not to even try matching the timing, just the head angles and lights. One trick was to re-time the footage in an extreme way. This is what gives Tutli her stop-motion style of realism. The eyes are in fact "re-animated" frames. If I had a puppet take that lasted 200 frames, and the chosen eye take lasted 3,000 frames, I would selectively take only the frames that I could use to re-build the acting performances whilst staying within the restrictions of the moves and light changes. Tutli might need to blink over 10 frames, so I could re-create a blink which retains the acting from the video blink that lasts 20+ frames. For example, you can make a blink sleepy or sudden with the same take if it's re-timed differently.
I tried at the beginning to rely on the computer for tracking but it wasn't nearly subtle enough. I decided that placing the eyes by hand for each frame was the only way to do it, and was actually faster. I would use alpha masks to remove all of the actor apart from their eyes, eyebrows, and partial under-eye, using varying feathered edges to match the facial structure of the puppet. This required a lot of painting experience. Many pieces were painted still patches, touched up in Photoshop, and positioned over problem joins or missing skin, fading in and out over time.
This was one hurdle, another was matching colour. Almost every frame of Tutli-Putli flickers because she is on a moving train. The only way to match eyes into this was to film as many of the big flashes at the time of the eye take, the rest have to be created with brightness and contrast tools, and colour balancing the darks, mediums and highlights to match every frame. Film grain was matched. Motion blurs [were matched], and making the eyes look like they were behind dirty glass in some shots.
Making Tutli's puppet hair fall back over her composited eyes was a handy little trick I used.
Filming the eyes would take about 3 hours per shot, compositing them seamlessly into a shot would take about 2 to 10 days.
ET: What software applications were in your compositing toolbox?
JW: I used Adobe After Effects for the compositing, and Photoshop to paint the facial patches. After Effects had its quirks, but it was quite solid over four years. From the start of the film in 2003 I went through 3 versions of After Effects.
ET: In the four years that Madame Tutli-Putli was in production, computer and video hardware got faster and more flexible while software became more powerful. As time went on, did this allow you to do more in less time, or did it open up more options?
JW: No, I got faster with practice, but the computer was always a bit slow. This project has been a 2D effect from the start so compositing shouldn't be too hard on the computer. However, as your readers will know, being able to flip through your last few frames in stop-motion is crucial to developing the flow of the move. Sometimes I would have to manipulate the eyes so much that the computer was never fast enough at frame advancing all the separate layers, masks, and colour effects attached to each eye. This was always a problem. This effect has to be in full resolution mode all the time to see if it matches. Very slow.