In The Foundry’s Nuke, several top of the line keying packages are right there from your first launch, but their application is far more complicated than a simple drag, drop, and adjustment.

To key well in Nuke you need an expert understanding of several concepts. From spill suppression to alpha math and formulas, these concepts are solidified through experience and practice. 

For beginners, there are some less obvious points to consider when approaching an advanced key. More than the buttons you press, the way you approach a shot can mean the difference between a nice clean alpha, and a noisy swiss cheese of black and white. Here are three tips to remember as you key in Nuke. 

Before you start your key you want to look at your footage… Really look at your footage. One of the most common mistakes with keying (and related compositing workflows) is jumping right in without a plan. 

The reason this is especially important for keying is to identify problem areas. These problem areas are portions of your scene, which you can tell by scrubbing through, that are going to give you trouble. This is usually due to either poorly lit areas of your background causing an uneven color or materials on your actor/foreground which either reflect the color of the screen or have transparency like fur/hair. 

The best way to look at your footage is through Nuke’s built-in channel viewers for Red, Blue, Green, and Luminance; these are enabled using the “r”, “b”, “g”, and “y” keys respectively. Flip through the channels and pay close attention to the differences you see between your foreground and background, especially in the color of your screen.  

Once you’ve identified these areas, you can start to consider the best method for tackling them and sketch out an outline of your comp. 

Preparation is key. Whether you’re actually chopping down a tree or pulling an alpha off a green screen (sometimes closer to grey or brown), the first step of the process can make or break your final success. 

The most frustrating part of keying is working on bad footage. For independent and student productions especially, there isn’t always a VFX Supervisor on set and the cameras aren’t kind when it comes to noise and edge mingling. 

Your first step to combatting this is denoise. The industry standard here is a plug-in called Neat Video, but in many cases, Nuke’s built-in denoise node will do the trick. Tweak your denoise sliders until you find a good amount of denoise while still retaining sharpness in your details. Once you’ve got a good look, you want to write out your file and use it as your new plate. This is crucial as the denoise node is extremely resource intensive and will bog down your script.

The next step to preparing your footage is performing basic color corrections. Like most things, coloring in Nuke is extremely complex and takes time to master, but the fundamental practices are universal. This step is about correcting the screen if there were issues while shooting it.

Make sure your white balance is correct and your shot has correct contrast and exposure. While you make adjustments, continuously sample your footage and keep track of what you’re doing to the channel values. The goal is to create an even difference between your foreground and background elements.

This final tip is the most important for beginners in terms of setting the expectation for future shots. Keying takes a lot of time and is far more involved than what it may appear when transitioning to more difficult shots in Nuke. 

Even with plates that are captured professionally, chances are you’ll need multiple keys. Once you start introducing elements such as hair, windows, glasses, and other high detail materials, you will find that a part of your key which works perfectly for your actors left arm doesn’t work at all for their right. 

This tip goes hand in hand with identifying your problem areas. As you gain experience keying, you’ll start to know right away where you’ll need to use individual or specific keys for certain parts of your footage. With hair, for example, you’ll want to be much more careful with your edges and take more time preserving detail than you’d need to on the shoulder of a leather jacket. 

If you have unwanted elements in your scene such as stands or tracking markers that are more complicated than a garbage matte can address, you can key these separately as well—anything to save yourself from roto work. 

Once you have your keys finished and ready, you can use a Keymix node to combine all of your alphas into one master key. The keymix node is very powerful and offers several options for merging multiple keys, so keep track of the math it’s performing on your alphas and read through The Foundry’s site!

One of the important things to understand when working Nuke is that you’re changing your footage with every operation you perform. Think of each node as a machine which takes data (the information from all the channels stored in the pipe), performs an action to it (this is where the math and code comes in), and then spits it out the other side. 

This may seem overly fundamental, but it’s crucial to understanding the why behind certain workflows and processes inside of Nuke.

For keying, it informs why you should always keep your alpha and your color separated. This means that you should always retain a pipeline with RGB channels from your original footage which haven’t been touched by the process of pulling your matte.

This structure shows a (very) simplified version of this concept. As you see, the alpha which is created in Keylight is then applied to the color data taken directly from the footage. This ensures that your RGB channels are untouched by the keying process and that your alpha isn’t affected by any color operations either.

Keying is a multistep process. The primary portion of it and the one that people most commonly conflate with keying as a whole is pulling your alpha. This is of course when you remove the foreground from the background for use later (usually with a new background). However, several detailed and intricate measures must be taken to create a convincing composition from keyable footage. 

Despill is one of these steps which is often hacked off the end of a hobbyist’s keying workflow. This step is necessary because of the nature of light itself. The only way you can see a green screen is because the light within the green wavelengths is traveling to your eye. If it’s traveling to your eye it must also be traveling through your characters hair, onto their skin, and across their clothing, right? There are many ways to despill your footage and get rid of this problem, but the most simple come in two forms.

First, there are despill tools which are built into the common keyers. In primatte, for example, the spill sponge is a tool which tries to remove spill from a sampled area according to your background color. Keylight offers a more simple trick for a quick and dirty despill. 

Set the channel value to "1" for the color of your background. In the case of a green screen, you will see that the node lowers the green values throughout your scene—usually leaving you with a reddish brown background. Be careful! Use this sparingly as it will affect all the colors in your shot.

Second, there are basic despill expressions which can be used to dampen the values in the channel respective to your background. The two common despill expressions are g > r ? r : g, and g > (r+b)/2 ? (r+b)/2 : g. In the first expression, if the green value is greater than the red, then the value of green is set to the value of red. In the second expression, if the green value is greater than the average of red and blue, then it is set to the average of red and blue. 

These expressions work similarly to any other despill method but allow you a look under the hood to really understand the goal of a lot of despill tools and techniques. Below you can see the resulting color of a feux green screen with an RGB value of 0.01,1,0 run through Keylight with green channel set to 1 along with the two despill expressions. Consider how this would translate to a real world situation with green values spilling onto a brown shirt.
 

As with everything you do in Nuke, always try to unveil both the why and the how. The math may not be as exciting as comping an epic scene or explosions, but you should treat your color values like children. Learn where they’re going and what changes are happening to them as they run through your node graph—connect your viewer to any point in your comp and control click for a look inside! 

First time here to ActionVFX? From Fire and massive Explosions to Blood and Gore VFX, we have the largest VFX library in the world ready to be composited in your project. Check out over 7,000 VFX stock footage elements right here. You can also download our free products!

As a subscriber with full access to the library on the ActionVFX Subscription, you'll not only be saving 80% off compared to purchasing assets without a subscription, but you can also get 1,000+ free bonus assets. Click here for more information and to subscribe today!

Stay informed about our release schedule to ensure that you’re always getting fresh VFX and sign up for our email newsletter to see when we release new VFX collections!

Remember to connect with us on our social networks to stay updated on our latest news, giveaways, announcements, and more!

Forum - YouTube - Facebook - Instagram - Twitter - TikTok - Linkedin