Home / Articles / Adobe After Effects / Using the Roto Brush tool in Adobe After Effects CS5

Using the Roto Brush tool in Adobe After Effects CS5

Article Description

This excerpt from Adobe After Effects CS5 Classroom in a Book shows you how to use the new Roto Brush tool in After Effects CS5, which is much faster than conventional rotoscoping, and for movies with complex backgrounds, much easier than keying.

Creating a segmentation boundary

You use the Roto Brush tool to add foreground and background strokes to a base frame, so that After Effects can identify the foreground and background areas in the image. It creates a segmentation boundary between the foreground and background.

Creating a base frame

To use the Roto Brush tool to isolate a foreground object, you start by adding strokes to a base frame to identify foreground and background areas. You’ll use the first frame as the base frame, and then add the strokes that identify the flamingo as the foreground object.

  1. Move the current-time indicator across the time ruler to preview the footage. At about 1:20, the flamingo’s head rises up to a striking pose. You’ll use the final pose in the clip to create a logo for the Casa Luna Wildlife Sanctuary.
  2. Press the Home key to move the current-time indicator to the beginning of the time ruler.
  3. Select the Roto Brush tool (roto-brush.jpg) in the Tools Panel.
  4. Double-click the Flamingo layer in the Timeline panel to open the clip in the Layer panel. This is where you will do all your Roto Brush work.

    By default the Roto Brush tool creates green foreground strokes, so you’ll start by adding strokes to the foreground—the flamingo.

  5. Choose Window > Brushes to open the Brushes panel. Then select a Hard Round 9-pixel brush, which is slightly smaller than the flamingo’s neck.

    When you’re drawing strokes to define the foreground object, start in the center of the object. Unlike traditional rotoscoping, you don’t need to define a precise boundary around the object. After Effects extrapolates from the foreground stroke to determine where the boundary is supposed to be.

  6. Draw a green stroke from the flamingo’s head, down its neck, and through the midsection of the body to the tail.

    A pink outline identifies the boundary After Effects created for the foreground object. It’s pretty good, though it may be missing the tip of the beak, the legs, and some tail feathers. You’ll help After Effects find the boundary by adding some more foreground strokes.

  7. Select a hard 3-pixel brush in the Brushes panel, and then draw additional strokes over the beak, tail, and legs to include them in the foreground.

    It can be tricky to get the legs and beak included without accidentally adding background as well. Adding background strokes will let you remove any extraneous areas in the matte.

  8. Press Alt (Windows) or Option (Mac OS) to switch to the red background-stroke brush.
  9. Add red strokes to background areas you want to exclude from the matte. Switch back and forth between the foreground and background brushes to fine-tune the boundary.

    You don’t have to follow the edge of the foreground element exactly to remove any errors in the selection. Getting close to the area to be removed and painting a background stroke may be all that is needed. Don’t worry about being exact with your brush strokes. Just make sure the mask is within 2 to 3 pixels of the edge of the foreground object. You will refine the matte later.

  10. Click the Toggle Alpha button (toggle-alpha.jpg) at the bottom of the Layer panel. The masked area is white against a black background, so you can see the matte clearly.
  11. Click the Toggle Alpha Overlay button (toggle-overlay.jpg) at the bottom of the Layer panel. The foreground area appears in color, and the background has a red overlay.
  12. Click the Toggle Alpha Boundary button (toggle-boundry.jpg) at the bottom of the Layer panel to see the outline around the flamingo again.

As you use the Roto Brush tool, the Alpha Boundary is the best way to see how accurate your boundary is, because you can see everything in the frame. However, the Alpha and Alpha Overlay options can be useful when you want to see the foreground without the distraction of the background.

Refining the boundary across the initial span

You used the Roto Brush tool to create a base frame, which includes a segmentation boundary that divides the foreground from the background. After Effects applied the segmentation boundary across a span of frames. The Roto Brush span appears below the time ruler at the bottom of the Layer panel. As you move forward and backward through the footage, the segmentation boundary moves with the foreground object (in this case, the flamingo).

You’ll step through the frames in the span and adjust the segmentation boundary as necessary.

  1. Press the 2 key to move forward one frame.

    Based on the base frame, After Effects tracks the edge of the object and attempts to follow its movement. Depending on how complex your foreground and background elements are, the boundary may or may not conform exactly to the area you hoped it would. In this case, you may notice changes to the segmentation boundary along the breast of the bird, near the feet, the tail feathers, or where the neck and body meet. This is normal, but it means you need to refine the segmentation.

  2. Using the Roto Brush tool, paint foreground and background strokes to refine the matte for this frame. If the matte is accurate, you needn’t paint any strokes.

    As you move through the span, each change you make affects the frames after it. The more you refine your selection, the better the overall results will be.

  3. Press the 2 key again to move forward to the next frame.
  4. Use the Roto Brush tool to add foreground and background strokes as necessary to refine the segmentation boundary.
  5. Repeat steps 3 and 4 until you reach 1:20. If the span doesn’t increase automatically, drag it forward to extend it beyond the initial 20 frames.

Adding new base frames

After Effects creates an initial Roto Brush span of 20 frames. As you move through the frames, the span range may increase automatically, or you can drag it to extend the span. However, the further you move from the base frame, the longer After Effects requires to propagate, or calculate, the segmentation for each frame. If the scene changes significantly, it may be better to create multiple base frames for your footage than to have one really long span. The scene in this project remains fairly consistent, so you could extend the span and make additional adjustments as needed. However, you’ll create additional base frames to gain experience using the tools, learn to connect spans, and see how the segmentation shifts as you move further from a base frame.

After you have reached 1:20 (1 second, 20 frames) in the refining process, add a new base frame to the project.

  1. Go to 2:10 in the Layer panel. This frame isn’t included in the initial span, so the segmentation boundary disappears.
  2. Use the Roto Brush tool to add foreground and background strokes, defining the segmentation boundary.

    A new base frame (represented by a gold rectangle) is added to the time ruler, and the Roto Brush Span extends twenty frames behind and ahead of the new base frame.

  3. If there is a gap between the initial span and the new span, click the left edge of the new span, and drag it back to the edge of the previous one.
  4. Press 1 to move backward one frame (from the base frame) and refine the segmentation boundary.
  5. Continue to move backward through the span, refining the segmentation boundary until you reach the beginning of the new span.
  6. Move back to the base frame, and then press 2 to move forward, correcting the segmentation boundary in each frame as necessary.
  7. When you reach 3:00, create a new base frame at 3:20, and repeat the process. Create an additional base frame at 5:00, refining the segmentation boundary in each frame in the footage.
  8. Move the time marker to the beginning of the time ruler in the Layer panel, and then press 2 repeatedly to progress through the clip. If you notice any errors, correct them with the Roto Brush tool.
  9. Choose File > Save to save your work so far.

Freezing your Roto Brush tool results

You’ve put a fair amount of time into creating the segmentation boundary across all the frames in this clip. After Effects has cached the segmentation boundary, so it can recall it without having to make the calculations again. To keep that data easily accessible, you’ll freeze it. This reduces the processing demands on your system so you can work faster in After Effects.

Once the segmentation is frozen, you cannot edit it unless you unfreeze. Refreezing the segmentation is time-consuming, so it’s best to refine the segmentation boundary as much as possible before freezing.

  1. Click the Freeze button in the lower-right area of the Layer panel. The Freezing Roto Brush dialog box displays a progress bar as it freezes the Roto Brush tool results. Freezing may take a few minutes. As it freezes the information for each frame, the cache line turns from green to blue. When it has finished freezing, a blue warning bar appears above the time ruler in the Layer panel, reminding you that the segmentation is frozen.
  2. Click the Toggle Alpha Boundary button (toggle-alpha.jpg) in the Layer panel to see the matte. Then click the Toggle Transparency Grid button (toggle-transparency.jpg). Move the time marker across the time ruler to see the flamingo move without the distraction of the background.
  3. Choose File > Save. After Effects saves the frozen segmentation information with the project.
4. Fine-tuning the matte | Next Section Previous Section