When you draw or paint on the frames of a movie, you’re rotoscoping. For example, a common use of rotoscoping is to trace an object, using the path as a mask to separate it from the background so you can work with it separately. Traditionally in After Effects, you drew masks, animated the mask paths, and then used the masks to define a matte. (A matte is a mask used to hide part of an image so that another image can be superimposed.) While effective, this is a time-intensive, tedious process, especially if the object moves a great deal or the background is complex.
If a background or foreground object is a consistent, distinct color, you could use color keying to separate the object from the background. If the subject was shot against a green or blue background (green screen or blue screen), keying is usually much easier than rotoscoping. However, keying is less efficient when you’re working with complex backgrounds.
The Roto Brush tool in After Effects is much faster than conventional rotoscoping. You use the Roto Brush tool to define the foreground and background elements. Then After Effects creates a matte, and tracks the movement of the matte over time. The Roto Brush tool does much of the work for you, leaving only a little cleanup work to be done.