"Avengers: Infinity War":
THANOS Artist Interview
KEY CONCEPT: The Age of Pixelated Humans is Here and Now.
Figure 1. Thanos, a CGI superhero character, stars in the "Avengers : Infinity War" film.
How did the team behind the movie pull this off, and what might it mean for the future? I recently caught up with the very busy Jan Philip Cramer, Head of Animation at Digital Domain (DD) in Vancouver, for a highly informative discussion about the process that brought the very evil, but very empathetic, Thanos, to life. A keen observer of the face in action, Phil was kind enough to mention that reading my book, “The Artist’s Complete Guide to Facial Expression,” several times was an important part of his early training.
In press coverage on the making of the movie, much attention has been paid to new software that allowed for an unprecedented level of efficiency and accuracy in transforming the details of Brolin’s facial performance to Thanos. For years there has been a push in the industry to make this pipeline – actor to character - more and more automated, and it sounded like the version employed in Infinity War was more efficient than ever—so much for the artists.
In press coverage on the making of the movie, much attention has been paid to new software that allowed for an unprecedented level of efficiency and accuracy in transforming the details of Brolin’s facial performance to Thanos. For years there has been a push in the industry to make this pipeline – actor to character - more and more automated, and it sounded like the version employed in Infinity War was more efficient than ever—so much for the artists.
 |
Figure 2. The blog author interviewing Digital Domain’s Jan Philip Cramer (at left) with impressive nimation and VFX credits on "Avengers: Infinity War" (2018); "Beauty and the Beast" (2017); "Spider-Man : Homecoming" (2017); "Deadpool" (2016), "X-Men: Days of the Future Past" (2014); "Jack the Giant Slayer" (2013) and more.
Phil assured me that I needn’t worry. As far as he is concerned, the current workflow allows for the best of both worlds. DD’s new MoCap to CG pipeline drives as much as two-thirds of the facial animation, freeing up the artists to fine-tune the other one-third of the performance by hand, including the most critical and subtle details.
And one critical and subtle detail stands out above all as central to the success of Thanos which proved to be the greatest failing of previous attempts to create a convincing CG main character: the eyes.
And one critical and subtle detail stands out above all as central to the success of Thanos which proved to be the greatest failing of previous attempts to create a convincing CG main character: the eyes.
Figure 3. Josh Brolin making facial expressions for high resolution captures of his face in action in order to build a more detailed and complex Thanos on screen.
IT’S ALL IN THE EYES
It’s always been obvious to me as a facial expression specialist that motion capture technology was simply not up to the challenge of successfully recording the subtle movements of the eyes.
It’s always been obvious to me as a facial expression specialist that motion capture technology was simply not up to the challenge of successfully recording the subtle movements of the eyes.
Figure 4. Digital Domain used new machine-learning algorithms to advance the facial performance capture process.
It’s one thing to put 250 dots on a face and track the precise movements of large, less mission-critical areas like the forehead, jaw, or cheeks. It’s quite another to expect a few data points to catch the amount of exposure of the iris, the exact course of the eyelids, or the direction of the gaze. And within the jewel-like precision of that geography rests the essence of what makes sadness look sad, or anger look angry.
In the case of Thanos, the Digital Domain team employed several first-time software tools to automate the bulk of the human-to-model transfer. One piece of software – Masquerade – used machine learning to match super high resolution Brolin poses stored in a digital library with much lower resolution on-set shots filmed by a MoCap helmet camera while he is in performance. Another piece of software – Direct Drive – then mapped the resulting hi-res Brolin mask onto DD’s Thanos rig, giving Phil’s animation team a huge head start on their work; Phil estimated that a shot that might have previously taken three to four weeks to animate could now be done in three to four days (a shot being several seconds of screen time) given the sheer amount of machine-modified imagery that now flowed into their animation systems based on the transfer techniques.
In the case of Thanos, the Digital Domain team employed several first-time software tools to automate the bulk of the human-to-model transfer. One piece of software – Masquerade – used machine learning to match super high resolution Brolin poses stored in a digital library with much lower resolution on-set shots filmed by a MoCap helmet camera while he is in performance. Another piece of software – Direct Drive – then mapped the resulting hi-res Brolin mask onto DD’s Thanos rig, giving Phil’s animation team a huge head start on their work; Phil estimated that a shot that might have previously taken three to four weeks to animate could now be done in three to four days (a shot being several seconds of screen time) given the sheer amount of machine-modified imagery that now flowed into their animation systems based on the transfer techniques.
HERE'S THE PUNCH LINE : After this multi-stage digital translation (Phil explains that the new-style transfers still required continual supervision and intervention) the DD team would now have a scowling or shouting Thanos on their computer screens that was a reasonable match for Brolin, EXCEPT the eyes. To my surprise, the eyes were not included at all in the main pipeline – too crucial, too outside the rendering ability of the MoCap process.
Figure 5. Thanos conveys his character with intense, low-key expressions.
OLD-FASHIONED HAND CRAFTING
Instead, the eyes of Thanos in Infinity War were all hand-created based, of course, on close-ups of Brolin in performance, but adjusted according to a huge number of other considerations, with the DD artists driving the process. Phil explained the various details that his animators focused on in their eye work – the refraction and reflection of the light on the eye depending on the conditions of the shot and the direction of the gaze, the local color of the eye whites, the blink rate, the way the eye moves as it tracks its surroundings (eyes in general are more jittery than steady), and the asymmetrical way the left and right eyes are modified by their lids and brow (a constant feature of Thanos’ poses.)
From a perceptual point of view, this matches exactly the expectations that audiences unconsciously bring to their experience of a human face, where they unconsciously use the eyes to signal the degree of human presence; if any of these elements – and there are lots! – aren’t right, the Uncanny Valley rears its ugly and disconcerting head, and the audience perception of Thanos goes from human to robotic.
Ultimately, Thanos works as well as he does because of the hyper-realism of his eyes more than any other single detail. Phil tells the story of the test that originally helped reassure the filmmakers of the process. The animators were particularly aware of two things – the directors saw Thanos as a dialed-back villain (think anti-Joker) visualized as scarier for his restraint and, they knew that getting his eyes “right” would be critical to their success. “I got the most positive feedback I’ve ever received in my career," Phil told me, “in response to that short.” Ironically, the content of the reel is simply Brolin/Thanos sitting in the dark on a throne, talking quietly, using random dialogue having nothing to do with the movie. No fist-pumps, no explosions, but great eyes.
Instead, the eyes of Thanos in Infinity War were all hand-created based, of course, on close-ups of Brolin in performance, but adjusted according to a huge number of other considerations, with the DD artists driving the process. Phil explained the various details that his animators focused on in their eye work – the refraction and reflection of the light on the eye depending on the conditions of the shot and the direction of the gaze, the local color of the eye whites, the blink rate, the way the eye moves as it tracks its surroundings (eyes in general are more jittery than steady), and the asymmetrical way the left and right eyes are modified by their lids and brow (a constant feature of Thanos’ poses.)
From a perceptual point of view, this matches exactly the expectations that audiences unconsciously bring to their experience of a human face, where they unconsciously use the eyes to signal the degree of human presence; if any of these elements – and there are lots! – aren’t right, the Uncanny Valley rears its ugly and disconcerting head, and the audience perception of Thanos goes from human to robotic.
Ultimately, Thanos works as well as he does because of the hyper-realism of his eyes more than any other single detail. Phil tells the story of the test that originally helped reassure the filmmakers of the process. The animators were particularly aware of two things – the directors saw Thanos as a dialed-back villain (think anti-Joker) visualized as scarier for his restraint and, they knew that getting his eyes “right” would be critical to their success. “I got the most positive feedback I’ve ever received in my career," Phil told me, “in response to that short.” Ironically, the content of the reel is simply Brolin/Thanos sitting in the dark on a throne, talking quietly, using random dialogue having nothing to do with the movie. No fist-pumps, no explosions, but great eyes.
Figures 7, 8 & 9. With convincing facial expressions, Thanos manages to vault across the Uncanny Valley.
WHAT'S NEXT?
The success of this superhero movie – one of the most lucrative of all time – is almost certain to be a game-changer in terms of what we can expect to see coming down the road in vfx. For clients willing to spend the money – and in terms of photorealistic CG characters, that realism comes at a cost – the possibilities seem endless. One can imagine historical figures, like Lincoln and Napoleon, coming to life; younger and older versions of living actors populating the same movie; Pinocchio turning into a “real” little boy; actors interacting with every manner of environments without the need of stunt doubles.
Phil and I ended our conversation by discussing the current state of the art of the robotics industry, where their efforts to create empathetic faces for their mechanical creations remind Phil of the animation industry in the early days of the CG revolution. Even though there are promising signs, today’s robots have a long way to go to reach the level of realism possible in CG. But the first steps are being taken!
Unlike robot designers, filmmakers now have the electronic tools to literally make convincing humans out of thin air. Fortunately, actors and artists are still 100% required. Tools are one thing, the vision and heart to exploit these resources in ground-breaking ways will be revealed in years to come.
The success of this superhero movie – one of the most lucrative of all time – is almost certain to be a game-changer in terms of what we can expect to see coming down the road in vfx. For clients willing to spend the money – and in terms of photorealistic CG characters, that realism comes at a cost – the possibilities seem endless. One can imagine historical figures, like Lincoln and Napoleon, coming to life; younger and older versions of living actors populating the same movie; Pinocchio turning into a “real” little boy; actors interacting with every manner of environments without the need of stunt doubles.
Phil and I ended our conversation by discussing the current state of the art of the robotics industry, where their efforts to create empathetic faces for their mechanical creations remind Phil of the animation industry in the early days of the CG revolution. Even though there are promising signs, today’s robots have a long way to go to reach the level of realism possible in CG. But the first steps are being taken!
Unlike robot designers, filmmakers now have the electronic tools to literally make convincing humans out of thin air. Fortunately, actors and artists are still 100% required. Tools are one thing, the vision and heart to exploit these resources in ground-breaking ways will be revealed in years to come.
Figure 10. Thanos, for now, reigns supreme as the most convincing CGI character
in a Hollywood movie.
in a Hollywood movie.
CREDITS: Figures 1, 5-10. Photos of the CGI character, Thanos, are from Avengers: Infinity War, a 2018 American superhero film based on the Marvel Comics superhero team the Avengers, produced by Marvel Studios and distributed by Walt Disney Studios Motion Pictures. The 2680 VFX shots in the movie were created by Perception, Cinesite, Digital Domain, Double Negative, Elstree Effects, Framestore, Industrial Light & Magic (ILM), Method Studios, RISE Visual Effects Studios, ScanlineVFX, Lola Visual Effects, Territory Studio & Weta Digital. The Thanos character was primarily produced by Digital Domain, which animated more than 400 shots; Figure 2. Photo of Phil Cramer by Pamela Belyea; Figure 3. Photo of Josh Brolin using the Masquerade program provided by Digital Domain; Figure 4. Images of .Josh Brolin and his Thanos character from "Making of Thanos in Avengers: Infinity War" produced by Marvel Studios with Digital Domain.
RSS Feed