Home / Articles / Adobe After Effects / Color Keying in After Effects CS3

Color Keying in After Effects CS3

  • Date: Nov 19, 2008.

Article Description

Are your matte edges crunchy, chewy, contrasty, solid, sizzling? Do these terms even mean anything to you? They will by the time Mark Christiansen finishes showing you how to work with After Effects CS3 color keying.

Like this article? We recommend

Adobe After Effects CS3 Professional Studio Techniques

Adobe After Effects CS3 Professional Studio Techniques

$59.99

Bluescreen and Greenscreen Keys

Back in the day, the alternatives to the rather pedestrian Color Difference Key in After Effects were third-party plug-ins, and for a few years there were two choices: Ultimatte for After Effects (now AdvantEdge), by Ultimatte Corporation; and Primatte Keyer (now by Red Giant Software).

Beginning with version 6.0 Professional, an alternative was included free with After Effects: Keylight. Its reputation was already well established prior to its inclusion in After Effects, having even earned it a Technical Achievement Award from the Academy of Motion Picture Arts and Sciences. The competition has by no means disappeared (I personally remain a big fan of Primatte), but Keylight is part of After Effects, so it's the main focus of this article.

Keylight is useful in many keying situations, not just studio-created bluescreen or greenscreen shots. For example, you can use Keylight for removal of a murky blue sky (see Figure 11). You wouldn't use Keylight to pull a luminance key, however, or when simply trying to isolate a certain color range within the shot; its effectiveness decreases the further you get from the three primary colors.

Keylight is typically employed to a back-plate shot against a uniform, saturated, primary color background. It's used when preservation of edge detail is of utmost importance—in other words, typically (but not exclusively) when working with footage that was specifically shot to be keyed against a screen of blue or green.

Figure 11

Figure 11 In one pass, with no adjustments other than selecting the sky color, Keylight effectively knocks out the background, even preserving motion blur of the bird in flight. The sky is nowhere near a perfect blue: Blue saturation is around 50% and only about double the amount of red.

Steps to a Perfect Key

Figure 12 shows a process tree that outlines the steps detailed in this section. No two complex shots are the same, but something like this approach should help you to pull a good matte, regardless of the tools used. The next section reprises these steps, specifically using Keylight.

Figure 12

Figure 12 This chart summarizes steps that can yield a successful color key.

These are the basic steps:

  1. Garbage matte any areas of the background that can easily be masked out. "Easily" means no articulated matte (don't animate individual mask points). As a rule, limit this to what you can accomplish in about 20 minutes or less (see Figure 13).
  2. Attempt a first pass quickly, keeping this matte on the loose side, and preserving as much edge detail as possible to be refined later.
  3. Preview the result at full resolution, in full motion, against a bright primary color. In rare cases, you're done, but before you fling up your arms in victory, switch to a view that clearly displays the alpha channel. Note any obvious holes in the foreground or areas of the background that have failed to disappear, as well as any noise in the solid areas of the matte (see Figure 14).
    Figure 13

    Figure 13 The quick-and-dirty garbage matte is your friend for eliminating unwanted parts of the stage.

    Figure 14

    Figure 14 A quick diagnosis of Keylight's first pass at this shot: The shadows must be keyed out because they extend outside the garbage matte area (and right off the set). This in turn has an effect on fine detail such as motion blur and hair, so these areas (hands, hair) may require their own holdout mattes.

    A holdout matte isolates an area of an image for separate treatment. You can think of a color key as an edge matte surrounded by two holdout mattes: one for the core, one for the background. The question then arises as to how these mattes are combined, which is explored in depth in the next section.

    If things look noisy and chaotic in the alpha channel or the edges are clamped and chewy, you can try the following:

    • Start over and try a new pass.
    • Apply noise reduction to the plate, and then start over (see the section "Noise Suppression" later in this article).
    • Articulate or track garbage/holdout mattes to isolate problem portions of the footage (see "Typical Keying Challenges").
  4. If necessary, separate the plate for multiple passes. At the very least, it's often useful to create a solid core and a completely transparent background so that you can focus only on the edge (detailed in the next section), but you may also need to separate individual parts of a layer for a separate keying pass.
  5. Refine the edge. Zoom in on a challenging area of the foreground edge (200% to 400%), and refine the key to try to accommodate it, using strategies outlined in the following sections. Possible challenging areas:

    • Fine detail such as hair
    • Motion-blurred foreground elements
    • Cast shadows

    Watch out for (and consider rotoscoping) foreground features that can threaten an effective key:

    • Areas of the foreground that reflect the background color
    • Edge areas whose color nearly matches the background
    • Areas of poor contrast (typically under-lit regions of the shot)
    • ***end indented list
  6. Preview the shot in full motion. Again, note holes and noise that crop up on individual frames, and use the strategies outlined in "Typical Keying Challenges" to overcome these problems. Following are some approaches you may employ at this stage:
    • Adding holdout mattes (typically masks), either for the purpose of keying elements individually or rotoscoping them out of the shot (see Figure 15)
    • Holding out the matte edge only, for the purpose of refining or blurring it (see "Matte Problems")

Ironically, the lazy single-pass approach either leads to a poor result or much more work, overall, as you struggle to make all areas of your shot play nice with "one size fits all" settings.

Figure 15

Figure 15 It may seem tedious and cumbersome to create individual holdout mattes for things like hair or heavy motion blur, but you could waste hours trying to eliminate the shadows while retaining the detail in one pass; meanwhile, an acceptable roto-matte might take 10 or 20 minutes.

Keylight

After Effects CS3 features version 1.2 of Keylight for After Effects; its usage is more or less identical to that of the previous version. (The big changes are under the hood, such as support for the Intel Mac.)

The first decision in Keylight is most important: sampling a color for the Screen Colour setting. (Keylight reveals its UK origins at Framestore CFC with that u in Colour.) More information to help you make an effective choice lies ahead in section "The Inner Workings of Keylight."

In the best-case scenario, you create any necessary garbage mattes and then follow these steps:

  1. Use the Screen Colour eyedropper to sample a typical background pixel. View defaults to Final Result, so you get a matte instantly. Set the background to a bright color and solo the plate layer or examine the alpha channel by pressing Alt-4 (Mac: Option-4).
  2. If in doubt about the Screen Colour setting, turn off the effect and set another instance, repeating as necessary until you have one that eliminates the maximum amount of unwanted background (see Figure 16). Then delete the rest.

    Now, as needed, look for areas to refine.

  3. Switch View to Status. Opaque pixels display white, transparent pixels black, and those containing transparency are gray (see Figure 17). It's an exaggerated version of the alpha channel that shows where your matte is not solid.
    Figure 16

    Figure 16 Even with well-shot, high-definition source (top), it's imperative to get the Screen Colour setting right to preserve all of the transparent detail. Choosing a darker background color (bottom left) creates a more solid initial background, but a lighter color selection (bottom right) preserves far more detail. (Source footage courtesy of Pixel Corps.)

    Figure 17

    Figure 17 Sample a background pixel near the center of the image for a good initial key, as seen in the Status matte. The black areas are already completely transparent, the white areas opaque. The gray pixels constitute the areas of focus; they're the semitransparent regions.

  4. Still in Status view, try Screen Balance at settings of 5.0, the default 50.0, and 95.0. Choose whichever one yields the best-looking matte. You might evaluate this result based on edge softness or firmness of the foreground matte (see Figure 18).
    Figure 18

    Figure 18 This initial result, with a Screen Balance of 95 (top), seems in Status view to be inferior to a setting of 5 (bottom), but a look at the color channel shows the default 95 setting to yield a subtler initial matte edge.

  5. If the background is not solid black, you have the option to boost Screen Gain until the gray mostly disappears in Status view. Use this trick as sparingly as possible, preserving gray at the edges of the foreground. If gray remains in the background, but separate from the foreground such that you can eliminate it with a garbage matte, do so (see Figure 19).
    Figure 19

    Figure 19 A light touch is best with Screen Gain; some artists (such as me, much of the time) never raise it above the default value of 100, although settings up to 105 or 110 are often acceptable. Don't use it to remove anything like the noise in the lower-left corner, which is easily solved instead with a garbage matte.

  6. Now, another optional step (that I often skip): For spill correction, you can set the Despill Bias using its eyedropper. Sample an area of the foreground that has no spill and should remain looking as is (typically a bright and saturated skin tone area).

    In the ideal scenario, your matte is now complete. If not (see Figure 20), this is a decision point to divide the plate. Following are the most basic useful steps for doing so. Before taking these steps, you can revert Screen Gain to 100 to loosen the basic matte.

    Figure 20

    Figure 20 Nice edges—but the core of this matte needs to be a little firmer.

As you may know, the Keylight toolset exists in other compositing applications as well. The Shake implementation of Keylight has the same underlying technology, but it has one key addition missing from the After Effects version: inputs for a garbage matte and holdout matte. If you can create these mattes procedurally such that they completely avoid the edge area, this approach allows you to consider them done and then focus only on the edge, which is all that matters.

I was inspired by what I call the "Three Pass Method" of Shake to devise the following workflow to achieve the same results in an After Effects precomp, with as few extra steps as possible. I encourage you to try it out, and let me know if your mattes are improved as a result of focusing exclusively on the edge.

  1. Create a precomp containing the Keylight layer with the Move All Attributes option checked. Rename the layer Edge. Duplicate this layer (with Keylight applied), rename it Background, and solo it. For clarity's sake, turn off effects for Edge for the moment.
  2. Create a background matte that's purely transparent and well separated from the edge, as follows: Mask in any needed overall garbage matte. Raise Clip Black just enough to remove gray pixels from the background in Status view, and then lower the Clip White setting just above the same number (see Figure 21).
    Figure 21

    Figure 21 Screen Gain can always be left at 100 for the edge matte (ideal) if the background is matted separately. A garbage matte eliminates the corner noise.

  3. Expand the matte either by raising the Screen Shrink/Grow value or by applying Simple Choker and lowering the Choke Matte setting. Make sure that the selection is completely outside the edge, separated by at least a few pixels; to check this result, switch to Final Result view and set the layer's blending mode to Stencil Alpha (see Figure 22).
    Figure 22

    Figure 22 Look closely at the edges. The matte from Figure 21 has been expanded so that it doesn't touch them, and the layer is set to Stencil Alpha so that the matte is passed through.

  4. Duplicate the original Edge layer again, solo the duplicate, rename it Core, and make sure that Keylight is enabled. In Status view, lower Clip White until the foreground is completely opaque. Raise Clip Black to a value just below the same number (see Figure 23).
    Figure 23

    Figure 23 The core matte, like the background matte, is chewy but solid, and is choked so as not to touch the edges.

  5. Shrink the matte either by lowering the Screen Shrink/Grow value or by applying another effect such as Simple Choker and raising the Choke Matte setting. Make sure that the selection is completely within the edge, separated by at least a few pixels, and then switch to Final Result view and set the layer's blending mode to Stencil Alpha. The result is a matte of the edge only (see Figure 24).
    Figure 24

    Figure 24 Thanks to the combination of Stencil Alpha for the background matte and Silhouette Alpha for the foreground, only the crucial part of the matte—the edge—remains. The core can be switched back to Normal blending for the final version.

  6. Refine the key of the edge layer, preview each layer in motion to ascertain that it doesn't contain errors, and switch the blending mode of the core layer to Normal when you're done.

This procedure may still leave particular areas needing to be separated with mattes for individual attention—the edge of the hair, for example, might be keyed separately from the body, with an articulated matte to define each.

4. Get the Best Out of Keylight | Next Section Previous Section

Adobe Press Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Adobe Press and its family of brands. I can unsubscribe at any time.

Overview

Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Adobe Press products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email ask@peachpit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security

Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children

This site is not directed to children under the age of 13.

Marketing

Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out

Users can always make an informed choice as to whether they should proceed with certain services offered by Adobe Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.adobepress.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links

This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020