Technical Art for Animation and VFX

The Mocaptionator’s Handbook V1.2

By Matthew Kapfhammer

INTRODUCTION

This article is for any animator that is using motion capture – a mocaptionator, if you will. There just doesn’t seem to be enough animator-friendly information about motion capture out on the net and it kills me to see people learning it the hard way (the way I and many others have). All of the content in this handbook is taken from the notes and thoughts I have collected over the last few years and is directly related to my own experiences, as well as those of some of the great people I have had the pleasure of working with.

Personally, I have grown to enjoy working with captured performances and wish to see the process evolve. Like any art form, the basics have to be mastered before pushing the boundaries. That is what I feel I’m sharing. Just the basics.

Before you read on, I need to point out that there is always more than one approach to working with motion capture. Hopefully, my notes will help you to refine your own and arrive at a pipeline/work flow that helps you to focus on your character performances, not the technology.

BEFORE MOCAPTIONATING

This article is meant to focus on the creation of final animation forged from a mocap’d performance base. However, a good mocaptionator has to be a good animator first. You should also know what happens before that juicy mess of curves arrives on your hard drive.

So before you read on, give the following books/articles a good read:

MOCAP – CLEARLY THE DEVIL’S DOIN’S

So you have begun using mocap to build your animations from? The following list of questions will quickly pop into your performance building brain tissues (generally in this order).

  • Where to start?
  • When to edit and when to leave the motion be?
  • WHAT to edit?
  • SHOULD I just use the mocap as reference??
  • Why is the performance so mushy/jerky now??!
  • WTF??!!!
  • Would eating an apple with razor blades hidden in it be less painful?!!
  • Seriously… WTF??!!!

A lot of those questions can be avoided by taking a little time to get to know your new enemy friend. We’ll start by quickly looking at what motion capture is, isn’t and what it can be.

Motion capture IS

  • a process – just like key framing. There are different ways to approach the process. Everyone has their own best practices resulting from experiences with their software, equipment and people. Ultimately, your processes (and tools) are only as good as the people using them.
  • a disruptive technology. Even though mocap has been in use since the late 1980′s, it is still relatively new to the vast majority of our industry and not very well understood. The off the shelf tool sets are still somewhat in their infancy, making working with the resulting motion data quite painful at the outset. Larger studios have been developing their motion capture pipelines (tools, process and all) for long enough that their new ideas have had a chance to be tested and refined. Though, that information has been slow to trickle down, since the equipment and qualified staff are still somewhat cost prohibitive/hard to locate for medium to smaller shops.
  • controversial. All disruptive technologies are. If you’re an animator crying about how motion capture will “take yer jeerrb”, you should consider learning more about the ‘enemy‘ or at least switch to crying about the rampant, overall poor use of mocap in games and broadcast. :’( However, I feel that you will find that your time is better spent learning when motion capture should be considered the right tool for the job and how to implement it.

Mocap is NOT

  • always easier, faster and/or cheaper than key framing. Motion data (due to many possible technical AND artistic reasons) can all too often be unusable. Often, you won’t get to re-shoot those motions and will be key framing the lost performances at the same level of detail at the mocap, which can take a pretty large chunk of time to accomplish. Additionally, production times and frustration levels will skyrocket when animators aren’t equipped with good motion re-targeting and editing tools/work flows.
  • exactly a good idea for cartoony, stylized work. It can be done, but the joy of cartoons is to exaggerate and break the rules of reality.
  • perfect right out of the box. Even super clean motion data still needs adjusting to fit the proportions of the final character.

Mocap CAN be…

  • an awesome tool for rapid pre-visualization. Remember what WETA did it on Lord of the Rings and King Kong? If not, go check the special features!
  • excellent for collecting animation reference. Imagine acting out your shots and getting to study the motion in 3D.
  • a great technique for getting specific, realistic subtleties. When your character needs to move exactly like Keanu Reeves, Margaret Thatcher or whoever the kids are into these days.
  • a joy to work with? Yes! If you spend the time and money on solid equipment (rental OR purchase) and software tools, know your pipeline, plan well, and hire experienced staff, building performances with motion capture CAN be a joy to work with.

TIME TO MOCAPTIONATE!

Now that we have the intro material covered, it’s time for the good stuff! Before that, I want to add a bit of a disclaimer. As I noted earlier, larger studios have their own pipelines with their own best practices. If you’re a newer animator, you can expect to be shown how your new company would like you to work. It is important to follow their instructions. My suggestions may not be the best option for your situation, so keep an open mind and think about their applications.

A Film Angle

At my first capture session, I quickly noticed that the process was closer to a film shoot than I had expected. This observation led me to study how and why film shoots work the way they do. Those lessons, in addition to my own experience on stage, led me to come to few conclusions.

  1. Its all about the performance: Hire good talent, because what you see on set is exactly what you get. I like the saying, “Garbage in, Garbage out.” Experienced performers will also not be as likely to freeze up as the crew watches them; something I’ve seen all too often. You can fix bad arcs, adjust the height of a punch, but you just can’t (don’t have time to) make that terrible double take work. Also, if the performer isn’t having a good day, YOU aren’t having a good… couple of weeks or maybe even months. See the following…
  2. Do it right the first time: Ever hear of a director getting cursed at behind their back, because they throw the phrase, “We’ll fix it in post,” around like a magic, make-everything-OK spell? Well, it isn’t. It means some poor person is going to be hacking away to correct something that probably wouldn’t have taken even a fraction of the time and money that it would have, if it had been done right the first time. The lesson here is that, while more work is needed to get things right on stage for the first shoot, its usually worth the time. I like to schedule a rehearsal session before the shoot to make sure all the difficult things are figured out and that the shoot will go smoothly. Those rehearsals weren’t always an option, but they made a very noticeable difference.
  3. Capture more than you need: When you’re on set for a film shoot, grabbing a few extra takes and varying angles gives you options in the edit. Similarly, if you can grab extra motions that you might need later, do it! I have found that shooting extra idle motions, walks, runs, jumps and running jumps sometimes avoided the need for re-shoots. Note: the walks, etc. could be blended into clips to extend performances.
  4. Take notes:Technical notes (What markers fell off? Was there a T-pose in the take?) and artistic notes (How did the director/producer/supervisor rate this take?) are the main information you’ll need. Depending on the intended output, type of system and set-up you’re using, you may have other details to keep track of. Be ready to add and take notes on random motions that get added to the motions list on the fly. A simple web form with drop-down menus and text fields will really help keep this tidy.
  5. Slate the video reference footage: For the exact same reasons you’d slate video or film shots. Its faster for everyone down the pipe that needs the footage. If you can devise an auto-slating system that is engaged when the mocap cams are rolling, you’ll be able to run with less staff and hassle.
  6. Props/environments are always a challenge: You might need to build custom props and you need to have these ready before the shoot. Having some basic sports equipment around is always good. Here are a few that you might not think of at first: a wireframe door, a couple of raised platforms, stairs, a ramp, lots of padded mats and a steering wheel rig. Spare boards, cardboard, foam pieces, and tape (duct, masking and grip) are extremely useful to help modify existing props.
  7. Lip-syncing: Even Especially if not being captured, the lip-syncing needs to be planned for. Have the scratch tracks/final audio ready to play on stage, so the performer matches the timing and mood.

Best Laid Plans

Thorough planning is ALWAYS going to be a biggie. By thorough planning, I do not mean rigidly planning every detail. I do mean preparing your team to be able to do their jobs well. This includes leaving room for flexibility, so you can adapt when necessary. The frustrating news is that learning where you’ll need to be flexible will only come from experience – usually painful experience. The good news is that you will probably need most of that flexibility during the capture session, so as an animator, you likely won’t get to worry about it.

Video Reference

While video reference of the capture session is not really necessary to complete the work, it can really help improve the quality and reduce the time it takes. I personally feel that video reference is essential for any motion used in a linear narrative like a commercial, tv series, film or game cinematic. It not only gives us the actual motion of the performer, but is a record of the acting direction they received from the director on-stage. That’s something that will cut out a HUGE chunk of the communication timesink that’s usually experienced in post. The team will be able to remain faithful to the original direction with fewer questions.

That original video reference can be used by two different types of people: clean-up artists and editing artists.

  1. The clean-up artists (AKA: trackers/mocap technicians) can use that footage when they are rebuilding marker motion paths and smoothing jitter to make sure that their final result is faithful to the real-life performance. In my opinion, this is the most tedious and frustrating part of the process for animators, since you’re primarily fixing issues caused by technical limitations.
  2. The editing artists (AKA: animators) can use the video as reference for key framing facial performances.
  3. THREE??!! Hey, man! You said TWO!” Yes. Well, there is another task where the artist can benefit from the reference footage. At some point, someone has to re-target the animation onto the final character. There are always proportional differences between the mocap performer and the final character. Quickly and correctly transferring the motion takes some skill all by itself. Depending where you work, this task may fall on either of the artists mentioned above.

Yes, it is true that all some directors don’t know what the heck they want, so remaining faithful to the original direction might not really help as change after change is requested. My suggestion is that YOU make the effort to ensure that your director (or producer, etc.) is properly educated as to how they can best make use of your team’s talent and time.

Setting Up

  • Prepare your cg assets: If you get the option, the post work can actually start before the shoot. By that I mean preparing the character, prop, and environment assets for use in your specific pipeline. If you check the special features for The Lion, the Witch and the Wardrobe, you’ll come across the mocap and rigging artists working with the actor portraying Mr. Tumnus to figure out how the elements of the actor’s movements and the cg puppet’s set-up would affect one another. By doing these tests early, they were able to shoot mocap+live action shots that they knew would work for the cg artists when they got the motion data later on. Having a realtime representation of the cg puppet is often very helpful for the performer, as it allows them to more easily visualize the character, adjust their performance accurately and get into the acting a little more. This requires a decent investment of resources and needs to be considered as early as possible. At the vary least, your technical artists can be getting the cg puppets set-up to work with mocap while everyone else is planning and/or rehearsing.
  • Use handles: It’s good to keep handles (extra frames of animation at the beginning and end of the shot) for the data you’re working on, so you can run smoothing filters and avoid the issues they can sometimes cause on the series of the first and last frames. I have also had directors ask to, “…just add more animation… at the head or tail of a shot. Making sure you have a couple seconds before and after the action to work with (even if those extra seconds are just enough to allow for blending into other motions) can help avoid some unpleasant conversations.
  • Get the scale for the cg assets: MAKE SURE to get the proper scale (the size relationships) for the character, props and environments BEFORE beginning work. A good team lead will make sure you have this basic, important information. If you aren’t sure, then you should ask! [Note: This goes for non-mocap related work, too. I’ve been burned by it before and was too much of newb at the time to force the issue.]

Analyzing

  • Think before editing: You might be able to solve several issues with a single offset adjustment if you take the time to analyze properly. Quickly learn to pick out how specific tools and techniques can actually cause more issues and avoid those types of edits.
  • Work in passes: Progress linearly and in passes to work from large changes down to smaller refinements. Try to separate the body into sections and analyze them in a way that allows you to tackle one area at a time instead of correcting an entire pose at once. I sometimes start towards the root of the rig’s hierarchy (only as far as the hips) and work my way out.
  • Only fix what is actually broken: Remember the phrase optical illusion, b/c you’re going to be constantly asking yourself if that’s what’s going on. Be sure to understand that what you’re looking at is a series of moving images and what you think is popping in the image is really the result of seeing another part of the body popping or ‘doing its own thing’. What might look incorrect on a wrist/hand might actually be the result of the position curves on the hips or the additive rotations of the spine bones. This is why it’s a good idea to approach editing in passes. If you take away the bigger problems, you can refine the smaller issues one at a time. Look at the body from several angles, if you’re stuck on what to edit. Many times the hips are fine and the arms/hands are ‘wonky’. You can always delete their animation and watch the whole thing without them in the equation. Then go back to the last saved version of the file and begin to edit.
  • What the audience sees is ALL that matters: If you’re lucky enough to be animating to a single camera view, remember that it is the only angle that matters to the viewer. Forget the other angles/views and concentrate on that silhouette.
  • Attention to detail: Periods of idle motion with little action and a relatively still or a locked camera need more attention to detail. Be sure to watch many high-res playblasts of the motion to ensure its clean! Why? Because with “bigger” motion (more extreme changes in the position and rotation) you can’t see the jitter as easily – doubly so if the camera is moving in the scene. The stillness of a locked-off camera allows the viewer to see more minute changes by using the non-moving pixels as points of reference for positions of the pixels on the character. So while the motion will be easier to shoot on stage, it might take just as long in post. All the more reason to obtain a good system and solid software solution.

Molding the Performance

  • Going all the way: At some studios, a single artist takes a motion all the way from mapping and clean-up to the finished performance. I don’t feel one way is better than another 100% of the time – its just nice when an artist that doesn’t take the time to clean properly has to fix his own damn mess. :D On a serious note, all artists on the team would benefit from gaining competency in every section of the pipeline, so they know where corners can be cut and where they can’t.
  • With mocap, everything is always moving: If you’re overriding the motion on a foot or a hand to get it to lock into place or hold a prop, keep 5-35% of the original motion on the character – as much as you can get away with and have it still stick. Use as much of the data as possible!! Get rid of jitter, but KEEP the subtlety!!
  • Don’t be afraid to mold that data into nice looking curves: Key reducing, peak removal and smoothing filters are a huge time-saver and work well as a first pass. Delete keys! Do whatever it takes, but when working destructively (deleting) be sure to save often and iterate the version number of the file!
  • Be conscious of whether you are working with FK or IK: If it is on the FK, remember that the issues you’re working on can be additive starting from the hips on out to the last controller on the finger.
  • Natural changes: When blending in and out of a single adjustment, try to key the offset at 0 starting at the beginning of a natural change/transition in the motion and completely on (usually a value of one when using constraints) at the end of the natural change. Examples of natural changes in motion/gestures are: standing up, picking a prop up or letting it go, a head turn, a new step forward, etc.
  • Key reducing filters: Automated key reduction = smoother data and easier/faster to control arcs. :D Automated key reduction also = less snap to the data. BE EXTREMELY CAUTIOUS in your use of filters/effects, b/c you can easily over-smooth the data (averaging key positions) and cause problems on other parts of the rig/animation. :(

The Principles of Animation as Applied to Motion Capture

You may be cheating by starting with someone else’s performance, but this is still animation. All the same principles apply: weight, timing, arcs, etc. Some of them happen on stage, some of them in post, but you still need them all.

  • Weight, Follow Through, Overlapping Action, Slow-In and Slow-Out: Realistic animation has to have even more attention paid to subtleties. You have to observe and practice. This is why you really do have to be a good key framer to be a great mocaptionator. Having the video reference of the original, on-stage performance can be useful for this task. Doubly so, if the audio of the director/supervisor’s acting direction was recorded along with the video.

  • Timing: I wanted to include this with the first entry, but it is important to note by itself. The performances that are captured are usually going to need to be re-timed in some fashion or another. They might need to be sped up for a fighting or action game/sequence. The performance might also need to be finessed to hit the main beats of a section of dialog. This is another place where having the the video reference of the original performance can help out.

  • Secondary Action: Also a big one for realistic animation, as you will likely be keying or simulating a bunch of secondary parts of your character. This is all you and your skill. Having the video reference of the original performance can sometimes be useful for this task. If you’re lucky enough to have scripting/rigging experience, implement a system that will allow you to key frame on top of a simulation to help you move along faster. If you aren’t lucky enough to know how to set that up, go start learning! In the long run, you will be able to get a better result faster by freeing your time up to focus on the acting, instead of keying a ton of secondary objects on your character to look like they are moving convincingly.

  • Arcs: You might be surprised to see how much correcting will be needed for the arcs of your character. This can happen because of noisy data or as a result of the re-targeting process. Sometimes, you just need to adjust the arcs to work better for the camera or the game’s style of animation. One of the first things I do to analyze the motion is to select the IK effector for the wrists/arms and view their motion path/ghost/trajectory. [Note: For Motionbuilder, use the trajectory tool in the viewport to quickly visualize and identify trouble shots for the positions of effector.]

  • Staging: If you are lucky enough to be involved with the mocap shoot, you damn well better do your best to see that proper attention is given to planning/coordinating the staging, because it can be annoying incredibly painful to fix in post. By staging, I do mean in the traditional sense. This is vitally important for planning in the pre-capture session phase, so your team has time to accurately prepare the physical props and environments for the capture session. My example here is a drum set. Capturing a performer playing on components of the physical drum set that the final CG drum set couldn’t have, lead to an animator correcting every movement that contacted those extra components. Ever notice how many times a drummer hits the snare? Not fun, fair or efficient. Note: The CG drums were also sent to the artist (who was working remotely) at a different scale than the character. The resulting confusion caused a couple days of delays. Shockingly, that company closed its doors within the same year.

  • Straight Ahead Action and Pose to Pose: Motion capture requires an odd sort of take on this principle. For games, you need to make sure the performer starts and returns relatively close to a specific pose, to make all the post work go faster. Narrative animations can benefit from pose to pose thinking, as well. To be a little more agile for later blending, have the performer hit certain poses during the performance that you can use to blend in and out of the variations you capture. This can be useful, as long takes can tire a performer out. Breaking them up and stitching them together is sometimes a great option to maintain high energy across the duration of the shoot and consequently – the final shot. Of course, this is also more of a time commitment.

  • Exaggeration: Depending on the project’s output, you’ll need to push the acting and motion to really bring out the essence of the character. Performances created for crowds need to be ‘bigger’ or more exaggerated to be understood by the camera who is viewing them from a distance.

  • Silhouette: As mentioned earlier, you’ll still need poses to read clearly. The age old trick of changing the ambient color of your character’s materials/shaders to %100 black or white still helps. In software that supports passes, you can make a special ‘animator’s pass’ to switch back and forth from.

GET BACK TO FACEBOOK WORK!

OK, folks. So hopefully, you now have a better idea of how to use motion capture as animator. Mostly, I hope you’ll find these notes helpful in improving your experience with motion capture and can channel that energy into your projects. If you have anything you’d like to share, errors to angrily point out, or names to call me, please email me at matt [at] lookslikematt [dot] com. I love feedback of all kinds.

Cheers,

Matthew Kapfhammer

PS. I wanted to make sure to thank everyone that helped me over the last few years (in and out of production) and eventually led me to put these notes together. I especially need to thank Megan Hobby, Jason Dexter, Chrystia Siolkowsky, Stephen Wagner, Jared Messenger, Allison K. Shelton, Ben Durbin, Tom Mahady, Bryan Scibelli and Kan Anant. Many additional thanks to Matt Liverman,  Brad Clark, Chad Moore, Morgan Bottner, Alex Lindsay and the PixelCorps, Tony Lomonaco, Alastair Macleod, Demian Gordon and the Motion Capture Society.