How to Rotoscope in After Effects

A workflow taught by Adobe Certified Instructors at Adobe Video World.

Rotoscoping is the art of cutting a subject out of footage frame by frame — separating a person from their background when there’s no green screen to help. It used to be painstaking manual work, but After Effects’ Roto Brush makes it dramatically faster. Here’s how it works.

What Rotoscoping Is

Rotoscoping means isolating a moving subject from its background so you can replace what’s behind it, color it separately, or composite it into a new scene. Think of it as drawing a mask around your subject that updates on every single frame as they move.

It’s a core visual-effects skill used whenever you can’t shoot against green. Whenever a hero needs a new sky behind them or an object needs removing around a person, rotoscoping is the answer.

When to Roto vs. Key

If your subject was filmed against a clean green or blue screen, keying is faster — our green-screen keying guide covers that workflow. Rotoscoping is what you reach for when there’s no screen at all.

Real-world footage, archival clips, and shots where greenscreen wasn’t possible all call for roto. It’s more labor-intensive than keying, but it works on literally any footage, which makes it indispensable.

Meet the Roto Brush

After Effects’ Roto Brush is a semi-automatic rotoscoping tool. You paint a rough stroke over your subject on one frame, and the tool intelligently finds the edges and separates it from the background.

Then, as you move forward, it propagates that selection across the footage, predicting how the subject’s outline changes from frame to frame. It turns hours of manual masking into a guided, far quicker process.

Make Your First Selection

Double-click your footage layer to open it in the Layer panel, select the Roto Brush tool, and draw a stroke down the middle of your subject. After Effects instantly generates a selection boundary — shown as a magenta outline.

Add green strokes to include areas it missed and hold Alt/Option to paint red strokes that exclude background it grabbed by mistake. A minute of refinement on this first frame sets up everything that follows.

Propagate Across Frames

Once your first frame looks clean, simply move forward one frame at a time. The Roto Brush analyzes the motion and carries your selection along, adjusting the outline as your subject moves.

Watch as you go and correct any frames where the edge drifts with a quick green or red stroke. Working in short spans and fixing problems as they appear is far faster than powering through and cleaning up at the end.

Refine the Edge

Hard cutouts look fake, especially around hair and motion blur. Switch on the Refine Edge tool and paint over fine, wispy areas like hair so the tool captures soft detail instead of a hard line.

Adjust the Feather, Shift Edge, and Reduce Chatter settings in the Roto Brush properties to soften the boundary and calm any frame-to-frame flickering. A believable edge is what sells the entire composite.

Freeze the Roto

Roto Brush calculations are processor-heavy, so once your selection is clean across the whole shot, click Freeze. This caches the result so playback is smooth and the tool doesn’t recalculate every time you scrub.

Freezing also locks your work, protecting it from accidental changes. Remember to unfreeze if you need to go back and fix a frame, then freeze again when you’re done.

Composite Your Subject

With your subject separated, place a new background layer beneath it and your cutout drops straight into the new scene. Now’s the time to integrate the two believably.

Match the lighting and color of your subject to the new background with a color grade, and add a subtle edge blur if needed. Good compositing is invisible — the goal is for no one to suspect the two layers were ever apart.

Manual Masking for Tricky Shots

The Roto Brush handles most subjects, but some shots — fast motion, low contrast, or complex overlaps — still need manual help. The pen tool lets you draw precise masks and keyframe their points frame by frame for total control.

Combining the Roto Brush for the bulk of the work with manual masks for the hard sections is exactly how professional roto artists tackle difficult shots. Patience and a steady eye matter more than any single tool.

Common Rotoscoping Mistakes

• Powering ahead without checking each frame, so errors pile up.
• Ignoring Refine Edge, leaving hard, fake-looking cutouts.
• Forgetting to match color and light to the new background.
• Not freezing the roto, leading to constant recalculation and slow playback.

A Real Rotoscoping Workflow

In practice, rotoscoping a five-second shot of a person walking goes like this. You open the clip, paint a Roto Brush stroke down the body on the first frame, and refine the edges with green and red strokes until the outline is clean. Then you step forward, correcting the boundary wherever it drifts.

After propagating across the whole clip, you switch on Refine Edge for the hair, dial in feather and shift-edge settings, and freeze the result. Finally you drop a new background beneath the cutout and grade the subject to match. What once took an afternoon of manual masking now takes a focused half hour.

How Long Rotoscoping Takes

Rotoscoping is one of the more time-intensive VFX tasks, and setting realistic expectations helps. A clean, well-lit subject with simple motion might take a few minutes per second of footage with the Roto Brush. A complex shot — fast motion, fine hair, overlapping limbs — can take far longer and may need manual masking.

The biggest time-saver is good source footage: high contrast between subject and background, minimal motion blur, and steady camera movement. When you can influence the shoot, planning for the roto in advance pays for itself many times over in the edit.

Rotoscoping Tips for Cleaner Cutouts

The secret to fast, clean roto is working in short spans and fixing problems the moment they appear, rather than powering through and cleaning up at the end. Check every few frames, correct any drifting edge with a quick stroke, and keep the Refine Edge tool reserved for genuinely soft areas like hair.

It also helps enormously to start with good footage. High contrast between subject and background, a faster shutter speed to reduce motion blur, and steady camera movement all give the Roto Brush more to work with. When you can influence the shoot, planning ahead for the cutout saves hours in post.

Key Takeaways

• Use the Roto Brush to paint a selection that propagates across frames.
• Add green and red strokes to include and exclude areas.
• Use Refine Edge for hair and soft detail, then Freeze the result.
• Composite onto a new background and match color and lighting.

Frequently Asked Questions

What is rotoscoping in After Effects?

It’s the process of isolating a moving subject from its background frame by frame, usually with the Roto Brush, so you can replace the background or composite the subject into a new scene.

How does the Roto Brush work?

You paint a stroke over your subject on one frame; the tool finds the edges and separates it, then propagates that selection across the footage as you step forward, adjusting for movement.

Is rotoscoping better than green screen?

Green-screen keying is faster when available, but rotoscoping works on any footage, even with no screen at all — making it essential for real-world and archival shots.

Sources: Wikipedia’s article on rotoscoping.