Green Screens and Lip Sync
A deepfake of Ronald Reagan for a Cold War installation
Marinemuseum Den Helder and IJsFontein approached us to help develop a deepfake of Ronald Reagan for their interactive Cold War installation. Being a first deep learning project, it presented several problems and an unusually steep learning curve. The work also opened the door to a planned second exposition — on the neo-Cold War of fake news, deepfakes, and active measures aimed at destabilising democracy. Covid put a hold on all of it; the museum was only briefly open during that period.
Using a deep neural network, software is taught to identify the structures and expressions of a face so it can build a mathematical model of one. Those models can then be used to swap faces with similar facial expressions between different videos.
The first attempts trained on around 1,000 images. The output glitched badly. Scaling the training set to roughly 25,000 images fixed some of it and exposed the rest — alongside the separate problem of replicating the Oval Office convincingly enough that the face had somewhere believable to sit.
To get the mouth right, a separately recorded mouth was superimposed over roughly 500 frames. It solved the shape and created a new problem: synchronisation. Getting the superimposed mouth to sit in time with the rest of the face turned into its own body of work.
What finally worked was a stack of them: a zoomed-in source pass combined with AI upscaling, the mouth re-edited by hand, several deep learning architectures compared against each other, and manual editing of facial landmarks where the model would not agree with the face.
