Clean plates are a VFX artist’s best friend, and now they’re easier to achieve than ever before with After Effects’ latest (and possibly greatest) feature, Content-Aware Fill for video.

Get ready to have your mind blown!

At Adobe MAX 2017, Adobe teased an interesting concept called “Project Cloak,” which had the ability to take virtually anything out of a scene, even when the camera is moving. It was a great showcase of what’s possible for the VFX industry with Adobe Sensei’s artificial intelligence.

As demonstrated in the video above, this algorithm didn’t simply paint a content-aware fill from Photoshop over every frame. The example showing that process led to warped distortions across the video, because it only took into consideration one frame at a time.

To work for video, the process had to contextualize the video as a whole, drawing from shots not only adjacent to each frame, but going all the way from the first to last frames of the scene.

Then, it had to interpolate that data across every frame to make this patchwork as seamless as possible. Not only did the demonstration accomplish this task with astounding results, but it did it extremely quick. This was an exciting proof-of-concept, but it really didn’t go beyond that for Adobe’s audience. 

There was no clear indication of whether or not After Effects users would ever have the opportunity to take advantage of this technology. Thankfully, a year and a half later, Creative Cloud users can now get in on the action.

While Project Cloak was once shrouded in secrecy, Adobe surprised the entire industry by including it as a new feature in the spring 2019 update of the Creative Cloud.

Now simply dubbed “Content-Aware Fill” in After Effects, VFX artists everywhere can take advantage of this powerful new tool that simplifies the workflow of removing objects from a scene, yielding clean plates ready for further visual effects compositing.

One of the best features of Content-Aware Fill in After Effects is how closely the workflow resembles that of Photoshop’s Content-Aware Fill feature.

Here’s a breakdown of how to use Content-Aware Fill in After Effects:

Step 4. Open the Content-Aware Fill panel by going to Window>Content Aware Fill.

Step 5. Click “Generate Fill Layer” in the Content-Aware Fill panel, and you’re done!

Very important: After your footage is analyzed, the Content Aware Fill tool actually creates a PNG sequence of all the patchwork, and it saves it automatically in your project location in a folder called “Fills.”  If you create additional Content-Aware Fills, they are just dumped into ordered folders within the Fills folder. 

This means if you are backing up or transporting your project, you’ll need to include this Fills PNG sequence folder!

If you processed your scene and the results weren’t exactly what you wanted, there’s also the useful “Create Reference Frame” button in the panel. It automatically opens the selected frame in Photoshop, and allows you to make adjustments there, so that the frame can serve as a guide for the rest of the Content-Aware Fill.

Check out this video from Adobe below to see it in action:

This should let you achieve better results if your scene needs a little extra finessing, and you can even do this with multiple frames if there are significant changes to the footage, such as dynamic lighting.

For optimal results, you’ll want to pay attention to the settings available in the Content-Aware Fill window, and make the appropriate selections for your footage.

Adobe offers a helpful explanation on the panel’s options below:

Object: Removes an object from the footage. It fills the transparent area by taking pixels from the current and surrounding frames. It removes an object from the footage. It estimates the motion of the scene behind the object and uses this to find appropriate color values. For best results, use this to replace moving objects, like a car on a road.
Surface: Replaces the surface of an object. It works similar to Object as it takes pixels from surrounding frames but uses the motion estimated in the comp under the transparent area. For best results, use this for static and flat surfaces, like a stain on a shirt, or a sign on a building.
Edge Blend: Blends surrounding edge pixels. It fills the transparent area by sampling pixels at the edges of the transparent area and blending them together, and renders fast. For best results, use this to replace static objects on surfaces that lack texture, like text on paper.

You may find that the “surface” option tends to work better if your subject is in front of a wall or tiled pattern, so don’t automatically assume “object” is the best choice.

Try out the various fill methods and see what works best. Also, your selection doesn’t have to be exact, so don’t stress over that. Also, feathered masks will blend your footage even better once you generate the fill. 

The inclusion of Content-Aware Fill in After Effects is perhaps one of the most innovative features to ever be included in the Creative Cloud. Not only can you easily navigate the process within After Effects itself, but there’s a clear emphasis on speed and precision as well. 

While it may not be a perfect substitute for something like Mocha, it’s evident that Adobe is making great strides in evolving their AI technology to aid industry professionals in a way that was simply impossible only a few short years ago.

First time here? ActionVFX creates action stock footage for VFX and filmmaking. (We also have some great free stuff!)

Remember to connect with us on our social networks to stay updated on our latest news, giveaways, announcements and more!

YouTube - Facebook - Instagram - Twitter - Linkedin