Creating a segmentation boundary
You use the Roto Brush tool to specify which parts of the clip are in the foreground and which are in the background. You add strokes to distinguish the two, and then After Effects creates a segmentation boundary between the foreground and background.
Creating a base frame
To use the Roto Brush tool to isolate a foreground object, you start by adding strokes to a base frame to identify foreground and background areas. You can start on any frame in the clip, but in this exercise, you’ll use the first frame as the base frame. First, you’ll add the strokes that identify the boy as the foreground object.
Move the current-time indicator across the time ruler to preview the footage.
Press the Home key to move the current-time indicator to the beginning of the time ruler.
Select the Roto Brush tool () in the Tools panel.
You use the Roto Brush tool in the Layer panel, which you’ll open now.
Double-click the boy.mov layer in the Timeline panel to open the clip in the Layer panel.
Choose Fit from the Magnification Ratio pop-up menu at the bottom of the Layer panel if you don’t see the entire image.
By default, the Roto Brush tool creates green foreground strokes. You’ll start by adding strokes to the foreground—the boy. Generally, it’s most efficient to start with broad strokes and then use smaller brushes to refine the border.
Choose Window > Brushes to open the Brushes panel. Set up a hard round 100-pixel brush. (You may need to resize the Brushes panel to see the options.)
When you’re drawing strokes to define the foreground object, follow the skeletal structure of the subject. Unlike traditional rotoscoping, you don’t need to define a precise boundary around the object. Start with broad strokes, and work down to small regions as After Effects determines where the boundary is supposed to be.
Draw a green stroke from the boy’s head down toward the bottom of the clip.
A magenta outline identifies the boundaries After Effects created for the foreground object. After Effects recognized only about half of the boy, because you initially sampled only a small area of the subject. You’ll help After Effects find the boundary by adding some more foreground strokes.
Still using a large brush, draw a green stroke from left to right across the boy’s coat, including the black strip on the right.
Use a smaller brush to add any neglected areas to the foreground.
It can be tricky to get some areas without accidentally adding background as well. It’s okay if you haven’t captured every detail in the foreground. You’ll use background strokes to remove any extraneous areas of the matte.
Press Alt (Windows) or Option (macOS) to switch to the red background stroke brush.
Add red strokes to background areas you want to exclude from the matte. Switch back and forth between the foreground and background brushes to fine-tune the matte. Don’t forget to deselect the area under the boy’s hat where the background shows through. In fact, one click may be all it takes to exclude that area from the matte.
Don’t worry about being exact about your brush strokes. Just make sure the matte is within 1 to 2 pixels of the edge of the foreground object. You’ll have an opportunity to refine the matte later. However, After Effects uses the information on the base frame to adjust the matte for the rest of the span, so you want the matte to be accurate.
Click the Toggle Alpha button () at the bottom of the Layer panel. The selected area is white against a black background, so you can see the matte clearly.
Click the Toggle Alpha Overlay button () at the bottom of the Layer panel. The foreground area appears in color, and the background has a red overlay.
Click the Toggle Alpha Boundary button () at the bottom of the Layer panel to see the outline around the boy again.
As you use the Roto Brush tool, the Alpha Boundary is the best way to see how accurate your boundary is, because you can see everything in the frame. However, the Alpha and Alpha Overlay options let you see your matte without the distraction of the background.
Refining the boundary across the span
You used the Roto Brush tool to create a base frame, which includes a segmentation boundary that divides the foreground from the background. After Effects applied the segmentation boundary across the entire length of the layer. The Roto Brush span appears below the time ruler at the bottom of the Layer panel. When After Effects has propagated the current boundary for a frame, the bar beneath that frame on the time ruler is green.
As you move forward and backward through the footage, the segmentation boundary moves with the foreground object (in this case, the boy). You’ll step through the frames in the span and make adjustments to the segmentation boundary as necessary.
In the Effect Controls panel, make sure Quality is set to Standard.
When Standard is selected, frames are processed at a lower resolution and the segmentation boundary propagates more quickly, but the matte may include fewer edge details. When Quality is set to Best, the matte will have more detailed edges, but the process will be slower. For most projects, Standard is sufficient.
Press the 2 key on your main keyboard (not the numeric keypad) to move forward one frame.
Working from the base frame, After Effects tracks the edge of the object and attempts to follow its movement. Depending on how complex your foreground and background elements are, the boundary may or may not conform exactly to the area you hoped it would. As you progress through this clip, you may notice changes to the segmentation boundary along the boy’s right sleeve (the left edge of the clip) as more of the coat is revealed in the frame. Likewise, you may need to refine the segmentation boundary around the flaps of the cap and the edges of the hood.
Using the Roto Brush tool, paint foreground and background strokes to refine the matte for this frame. If the matte is accurate, you don’t need to paint any strokes.
If you make a stroke you don’t like, you can always undo the stroke and try again. As you move through the span, each change you make affects the frames after it. The more you refine your selection, the better the overall results will be. You may find it useful to move forward a few frames to see how changes affect the boundary.
Press the 2 key again to move forward to the next frame.
Use the Roto Brush tool to add to the foreground or subtract from the background as necessary to refine the boundary.
Repeat steps 4 and 5 until you reach the end of the layer.
Pay special attention to the left earflap of the cap when it crosses in front of the tree. The dark areas overlap, making it more difficult to get a consistent edge. Remember that you are trying to get the segmentation boundary as close to the edge of the foreground object as possible.
When you have completed refining the segmentation boundary for the entire layer, choose File > Save to save your work so far.