This article is also available in:
French
Deep learning is the hot new trend in VFX. In the last couple of weeks, we featured a Russian TV ad featuring a fake Bruce Willis, a tool that can generate realistic dance moves, and some interesting tech from NVIDIA that showed us a glimpse of the future. Deep Learning in VFX is still in its infancy, wich makes the future all the more exciting.
In France, VFX & animation studio Mac Guff (not to be confused with Illumination Mac Guff: what was a single company split in two in 2011, and Mac Guff/Illumination Mac Guff operate independently from one another) saw that such technologies had a huge potential, and decided to invest heavily in this area. As a result, the studio managed to improve both the quality and speed of certain tasks, such as face replacement or de-aging. The upcoming French TV show Hôtel du Temps (which translates to “Time Hotel”) is a good example of what Mac Guff is now capable of. In this TV show, French anchor Thierry Ardisson will interview deceased celebrities. In just 3 months, Mac Guff was able to produce 50 minutes of content with celebrities such as Lady Di, former French President Mitterrand and French actor Jean Gabin.
Here the trailer for this upcoming French TV show:
“Hôtel du Temps”
Such a TV show is quite challenging, since the viewers already know what the celebrities look like. Moreover, the show will feature lots of close-up shots.
Mac Guff President Rodolphe Chabrier explained us that the goal was not to make the celebrities look more attractive than they were in real life, even if it would be doable. Instead, they were tasked to re-create the faces as true to life as possible.
Another issue was to decide how old each celebrity would be in the show. Should Jean Gabin be interviewed as a young actor, or later in his career? Such questions must of course be answered before the TV show is filmed, since only the face is replaced. The actor playing Jean Gabin, the actress playing Lady Di must interpret their character at the right stage in their life, and wear the right hairstyle (Face Engine, the tool created and used by Mac Guff, only handles the face, not the hair).
Once all these settings and details have been decided, the hardest part is yet to come: finding the data that will bring the celebrities back to life. This is due to the fact that the choice of input data used to train a deep learning tool has a major impact on the results you get at the end.
Unfortunately, archival footage from the pre-HD era, pre-digital era is quite often only available in low quality: low resolution, grainy picture. Which is where “Face Retriever” comes into play. The idea behind this concept, put forward by French presenter Thierry Ardisson when he’s interviewed about the show, is to highlight the fact that Mac Guff had to “retrieve” the faces by carefully evaluating and selecting available sources, then to improve this data before feeding it to the deep learning tool. Unfortunately, Mac Guff won’t tell too much about this, as the team doesn’t want to help other VFX studios to replicate this process.
That being said, Rodolphe Chabrier gave us some insight. He highlighted that people sometimes don’t look like themselves on video. An unusual facial expression or weird lighting can affect the likeness of the face. It is better not to keep this kind of footage.
Furthermore, a common issue with deep learning is the fact that this is kind of a “black box” technology, which prevents the user to control the end result. This drawback has been eliminated by Mac Guff thanks to a mixed approach: the team does not rely solely on deep learning (once again, Rodolphe Chabrier could not go into details about the process). In other words, the artists at Mac Guff can control what comes out of the pipeline. If needed, gaze direction, facial expressions can be tweaked and corrected.
From hacking to industry-ready
Mac Guff managed to create some experimental tools, then to make them production-ready. As Rodolphe Chabrier puts it: “We went from hacking to industry-ready”.
The studio has been working on this transition for a while (as shown by the impressive work done on the TV series Le Bureau des légendes). The Covid-19 pandemic paradoxically boosted this process: at the beginning of the pandemic, with many projects being cancelled or delayed, Mac Guff didn’t have as much work to do as usual. Rodolphe Chabrier and Mac Guff VFX Supervisor Martial Vallanchon were therefore able to focus on deep learning for 3 to 4 months, learning, testing, improving. They were able to bring the whole management on board with the idea that deep learning was indeed viable for VFX work. Mac Guff then relied on a pragmatic approach, moving forward step by step to explore this technique.
The result of this tedious work: 50 minutes of content created for “Hôtel du Temps” in 3 months. This is both an impressive speed and a game-changing approach when working on a show: with this new pipeline, a lot of time is needed to carefully pick the right data and train the tool. The final render stage, on the other hand, is way faster than it would be with traditional CG techniques (but faster does not mean real-time, at least not yet).
In other words, switching from CG only to a mixed approach that includes deep learning technology can’t be summed up as switching from one tool to another. This is revolutionary technique has a huge impact on the whole production and supervision process. “Workflows change, costs change, visual fidelity changes”, as Rodolphe Chabrier puts it.
However, he explains, just like improvements made in the past (including the birth of visual effects), the goal with AI is not to kill jobs but to create new ones, to widen the scope of what can be told in a story, to make what was impossible possible.
A competitive advantage
Such a technology can of course provide a competitive advantage to studios that will be the first to make the jump. Even if Mac Guff is above all a generalist VFX house (instead of focusing mostly on creatures or physics simulations like some studios do), this work on faces might attract customers that are specifically interested in this area, in France and beyond.
Tomorrow and beyond
Mac Guff won’t stand stand idly by, and the team aims at expanding the way AI is used in the studio.
Hair, to begin with, is an area of interest. As we wrote, for Hôtel du Temps Mac Guff only worked on the faces: the hair you see in the trailer is either real hair or a wig worn by an actor. Another idea would be to work on the body and not the face, with a dedicated “Body Engine”. The pragmatic/step by step approach chosen by Mac Guff might also lead to faster and cheaper rotoscoping. Which would also allow the studio to relocate in France a task usually sent to Asian studios.
To make this dream come true, Mac Guff launched what will become a dedicated AI & deep learning department, and hired some engineers.
This is of course a big investment, since deep learning/AI engineers are very much sought after. Fortunately, Mac Guff received a grant from the CNC (Centre national du cinéma et de l’image animée, an agency of the French Ministry of Culture focused on promoting audiovisual arts in France). Even though the amount received is not as high as what Mac Guff was hoping for, Rodolphe Chabrier highlights that without this grant, Mac Guff would not have been able to launch the new AI department.
The secret behind a long-lasting VFX studio ?
De-aged faces, digital make-up, face replacement, and tomorrow hairstyles, bodies, animation, sets… Mac Guff just openned a new chapter.
Three decades after the studio pioneered morphing technology (it should be reminded that ILM outpaced Mac Guff by just a couple of months), Mac Guff keeps focusing on what made the success of the company: innovation.
Hôtel du Temps will be broadcasted on French TV channel France 3.
Here are all the pictures Mac Guff was allowed to send us. For each pair of pictures, the one on the left is the footage straight from camera, the one on the right is straight from Face Engine, before the finishing stage (no color grading has been applied yet). Click on any pair of pictures to view them at full size.
