Computer Graphics World- Edition 2

CGW.COM | EDITION 2 – 2020 | $8.00 US | $8.25 CAN PLUS

PIXAR SPARKSHORTS
MEDICAL VIS

LIGHT FIELD PRIMER
MYTH: A VR STORY

MAKING
MAGIC

CRAFTING THE LEGENDARY
VFX IN CURSED

1 cgw s u m m e r 2019



CGW features Igniting Passion
COMPUTER GRAPHICS WORLD
16 Pixar’s SparkShorts program
on the cover gives employees the chance to
22 bring their own animated short
8 Making of a Legend 30 films to life.
Netflix turns the Arthurian
myth on its head with 34 Universal Scene
a unique story that is Description
augmented with enchanting
visual effects. Pixar does a deep dive
into USD and its effect on
Visit CGW.com for Web-only features and news. 38 content creation.
40
Plus, stay abreast of the latest industry changes and A Bedtime Story

updates resulting from the COVID-19 epidemic. Filmmakers use VR to extend
the world of the elemental spirits
departments 50 in “Myth: A Frozen Tale.”
56
2 Editor’s Note 60 Left Brain, Right Brain
SIGGRAPH “Thinks Beyond” the norm and revamps its 63
annual conference and exhibition in this very unusual Animators use computer
time – all the while incorporating nearly all the familiar graphics to help explain complex
elements into the virtual rendition. medical concepts.

4 Spotlight Working Together

See it in Post • J.J. Sedelmaier Productions: 30 years For M&E workstation
creating commercials & pop-culture classics performance, it’s different
WHERE TECHNOLOGY AND TALENT MEETSM JULY/AUG 2020 POST strokes for different folks.
www.postmagazine.com • mixing sci-fi & satire for Amazon’s Upload 
• stay in tune with the latest in Music Videos Transcending
PLUS: • Director David Fincher on Boundaries
DAVID FINCHER’S MINDHUNTER
UPLOAD DIRECTOR Netflix’s Mindhunter 41 Dark Days
GREG DANIELS 45 Pre-production
PIXAR’S SPARKSHORTS POSTMAGAZINE.COM
J.J. SEDELMAIER TURNS 30 in Real Time

MAKING Spinning a Yarn
MUSIC VIDEOS
Netflix’s The Willoughbys –
MUSHROOMHEAD, MACHINE GUN KELLY, Not your typical tale.
DIPLO & MORE!
Education

• Classroom Disrupted
• Ahead of the Curve

Hybrid Drama

The Blacklist’s out-of-the-box
production strategy
during the pandemic.

Quest for Reality

A primer on light field
technology.

edition 2, 2020 cgw 1

ADAPTING TO THE NEW NORMAL CGW
COMPUTER GRAPHICS WORLD
B ack in March, when NAB was canceled abruptly due to COVID-19, it was difficult
to envision the new normal beyond the short term. Perhaps it was naivete or THE MAGAZINE FOR DIGITAL CONTENT PROFESSIONALS
ignorance on my part, and while it became obvious soon enough, at the start of
EDITORIAL
the outbreak i never imagined that SIGGRAPH might suffer that same fate.
EDITOR-IN-CHIEF Karen Moltenbrey
However, after 46 years of adapting and serving the needs of the computer graphics e: [email protected] t: 603.432.7568
DIRECTOR OF WEB CONTENT Marc Loftus
industry, SIGGRAPH has proven how truly agile it can be, as the show management and e: [email protected] t: 516.376.1087
CONTRIBUTING EDITORS Kathleen Maher, Jon
volunteers for this amazing conference and exhibition have transitioned it to a virtual plat- Peddie, George Maestri, Barbara Robertson
PUBLISHER / PRESIDENT / CEO William R. Rittwage
form due to the ongoing pandemic. While some other trade shows have gone virtual with COP Communications

an extremely scaled-down offering, not so with SIGGRAPH. Instead, it is offering its exten- ADVERTISING SALES

sive complement of events and programs, from a virtual exhibition area, to its Computer DIRECTOR OF SALES—NATIONAL Mari Kohn
e: [email protected] t: 818.291.1153 c: 818.472.1491
Animation Festival, to its Technical Papers, Panels, Courses, Emerging Technologies, Job CORPORATE SALES EXECUTIVE—
EVENTS, CUSTOM AND INTEGRATED PRINT/
Fair, Immersive Pavilion, Keynote, Production Sessions, Real-Time Live!, Labs, VR Theater, PUBLISHING SERVICES Lisa Neely
e: [email protected] t: 818.660-5828
and more. The various chairs and committees have devised the most optimal methods EDITORIAL OFFICE / LA SALES OFFICE
620 West Elk Ave., Glendale, CA 91204 t: 800.280.6446
of bringing each element of SIGGRAPH to this year’s virtual attendees. It seems that this
ART/PRODUCTION
year’s theme of “Think Beyond” became even more relevant than anyone initially thought.
SENIOR ART DIRECTOR Michelle Villas
Speaking of SIGGRAPH, the Computer Animation Festival’s Best in Show award goes e: [email protected]

to Erica Milsom for “Loop,” part of Pixar Animation Studios’ SparkShorts program. Read ONLINE AND NEW MEDIA Elvis Isagholi
e: [email protected]
more about this program and the making of “Loop” in this issue. Meanwhile, we are also
SUBSCRIPTIONS
presenting to our readers a behind-the-scenes look at the making of the experimental
818.291.1117
Pixar VR short film “Myth: A Frozen Tale.”
CUSTOMER SERVICE
Typically in this issue, CGW covers the year’s summer VFX and animated theatrical
e: [email protected] t: 818.291.1117
blockbusters. This year, however, with film production at a near standstill, we have looked
COMPUTER GRAPHICS WORLD MAGAZINE IS
beyond the big screen and have found no shortage of amazing series made with amazing PUBLISHED BY COMPUTER GRAPHICS WORLD,
A COP COMMUNICATIONS COMPANY.
technology for smaller screens, including the newly released Netflix’s Cursed.
Computer Graphics World does not verify any claims or other
While many studios were establishing remote workflows due to the virus, there are information appearing in any of the advertisements contained in

some animators whose job it is to work intimately with the novel coronavirus and similar the publication, and cannot take any responsibility for any losses or
other damages incurred by readers in reliance on such content.
concepts in the field of medical visualization. We introduce you to some of these unique
Computer Graphics World cannot be held responsible for the safe-
experts whose work straddles the line between science and art. keeping or return of unsolicited articles, manuscripts, photographs,
illustrations or other materials. Address all subscription correspon-
We round out the print issue by examining an important technology: light fields. dence to: Computer Graphics World, 620 West Elk Ave, Glendale, CA
91204. Subscriptions are available free to qualified individuals within
But wait, there’s more. We have had a long lead time since our last issue in early April, the United States. Non-qualified subscription rates:
USA—$68 for 1 year, $98 for 2 years;
made even lengthier since SIGGRAPH moved back a month. As such, we have been Canadian subscriptions —$98 for 1 year and $136 for 2 years;
all other countries—$150 for 1 year and $208 for 2 years.
accumulating stories, stories you will no doubt still enjoy. Therefore, we are publishing an Digital subscriptions are available for $27 per year.
Subscribers can also contact customer service by calling
expanded digital version of this issue, with more great content. You can get it by signing 818-291-1158, or sending an email to [email protected].

up for this and other digital issues on CGW.com; you also will be able to find the stories on Postmaster: Send Address Changes to
Computer Graphics World, 620 W. Elk Ave., Glendale, CA 91204
our website when we make the current print issue available there. Please send customer service inquiries to
620 W. Elk Ave., Glendale, CA 91204
Our bonus content includes coverage of the animated feature The Willoughbys as well
RECENT AWARDS
as the live-action spy drama The Blacklist, which finished out its season with a hybrid

live-action/animated episode when lockdown occurred.

With Emmy Awards on the horizon, we also take a deep dive into two other captivating

series: Amazon’s Carnival Row and HBO’s His Dark Materials, as the film hiatus provided us

with additional time (and space) to cover some well-deserving projects that did not make

it into our last issue. We also take a close look at Universal Scene Description (USD).

As a departing note, I want to applaud all the resourceful technologists in our industry

who, with a moment’s notice, were able to get their studios and remote teams up and

running so they could continue working on projects and keep the industry moving forward

during this tough time. To everyone else, stay positive in this time of our new normal. 

Karen Moltenbrey,
Editor-in-Chief

[email protected]

2 cgw e d i t i o n 2, 2020

FYC.NETFLIX.COM

SPOTLIGHT

Virtual studio in the cloud, simplified its power, scalability, and compatibility with our workflow, it was the
clear solution.
Lucky Post, a full-service post-production studio in Dallas and Aus-
tin, Texas, that combines creative editorial, color, VFX, finish, motion What does your workflow look like?
graphics, and sound design, found itself in the unenviable position of We have four editors, five assistant editors, one graphics designer,
many post facilities amid the COVID-19 crisis. It had to find a way to two sound designers, one colorist, and two Autodesk Flame artists.
keep its employees working remotely. That’s when media systems With SimpleCloud, we’re able to mirror our typical workflow in a
integrators CineSys-Oceana introduced them to SimpleCloud, a remote environment. We utilize shared storage to carry a job from
cloud-based platform for digital content focused on animation, off-line through finish, similar to how we would at our physical studio.
architecture, editorial, game development, and visual effects. Sim-
pleCloud provides management and setup of a virtual studio in the And the results?
cloud, allowing users to be connected to a virtual workstation from This has been an amazing experience. We were able to add some
anywhere, through any thin client or HTML5-supported browser. customizations to workstations to leverage Evercast and rely purely
on the data center network connection, while the clients logged on
Here, Tim Nagle, senior Flame artist at Lucky Post, discusses to supervise. In these times, you can imagine the stress on the local
his team’s experience integrating and using SimpleCloud for network nodes with everyone working from home; this solution has
remote work. mitigated any issues with local upload speeds from artists’ homes.

What initial challenges did you face? What applications do you use in SimpleCloud?
The most obvious is how to give our clients a seamless experience We are using the Adobe Suite, Maxon’s Cinema 4D, along with some
and give the artists the power to collaborate like they did every day customizations we asked for to solve our streaming needs.
prior to this. How are we meant to start or finish a job, and focus on
creativity, when we are removed from our infrastructure? How has it worked so far?
This has been amazing for us to be able to serve our clients seam-
Some of the top requirements we had were collaboration for lessly, as if we physically have the infrastructure of our facility.
artists, speed and efficiency with the tools, and making our clients
successful while we are all in 16 remote locations.

Did SimpleCloud fit those needs? How was your experience with CineSys?
We found that SimpleCloud, combined with our Evercast stream- CineSys-Oceana has been integral to our success in this transition.
ing solution, was the most efficient way to move forward with We rely on them to help solve problems creatively and sustainably.
active projects and spin up new projects in a virtual environment.
We arrived at this solution with Jason Starne from CineSys-Ocea- Do you have any advice for other companies that want to
na. We have had a long relationship with CineSys, and we know work remotely?
and trust Jason to jump in and help us find a solution whenever we We suggest anyone trying to navigate the challenges of working
need one. remotely while giving your clients the same experience, consider
SimpleCloud and don’t be afraid to customize where needed.
We did not try other products prior to SimpleCloud. When we saw

4 cgw e d i t i o n 2, 2020

A note from the CGW team to our readers:

We are aware that the COVID-19 crisis has forced us to
adjust in different ways. In light of studio and office closures,
we’d like you to know you can redirect your subscription to a
residential address – quickly and at no charge. Simply visit
http://www.cgw.com/Subscriptions.aspx to update your
mailing preference. You can also receive our digital edition at
no charge.

We are committed to continuously delivering timely editorial
coverage via all of our platforms today, just as we have done
for the past four decades.

We are in this together. Stay strong, stay healthy,
and stay connected.

facebook.com/CGWmagazine @CGWmagazine

CGWCGW.COM | SUMMER2019 | $8.00 US | $8.25 CAN ANNIVERSARY SPECIAL PLUSCGW.COM | EDITION 1 – 2020 | $8.00 US | $8.25 CAN
COMPUTER GRAPHICS WORLD ..B.FERGOINMNTINHGE!
DWIGKHILTYCAAOUWLSNLO:UYTRAASINDNKNTPDESUIRIE-METEAARSAADGDCTITINIGOINTEORVDDIGNOOEITSASNWRRGCASWTEUTELHAOIILESTTYREHS
ICEBREAKERCNOOMREDIACLMIVYETIHNSFARONZDELNE2GENDS
EL
DS,

SPOTLIGHT

Global PC Gaming Hardware Market Forecast Due to TV broadcasting of sim racing events, we are observing an
Expects Surge in 2020 Due to COVID uptick in racing simulation builds. These include a high performance
computer often with premium audio, racing wheel/shifter/pedals,
Jon Peddie Research (JPR), the industry’s research and consulting sometimes a racing seat, and other elements. Many of the sim
firm for graphics and multimedia, reported in its newest gaming racers are new to the market but have money and spend $2,000 to
market study regarding the PC Gaming Hardware Market, which $5,000 on systems, audio, and accessories.” 
consists of personal computers, upgrades, and peripherals used for
gaming. The market is booming globally due to shelter-in-place or- Jon Peddie, president of JPR, says, “COVID-19 has created a big
ders as gamers upgrade and buy new PCs and accessories. Perhaps spike in sales for PC gaming products in the near term. The total
more importantly, the current situation has actually created millions market is approaching $40 billion in 2020. However, we temper our
of new PC gamers looking for immersive, exciting, and economically forecasts for possible economic issues in 2021 and beyond. Fore-
efficient forms of home entertainment.  casts are heavily dependent on consumer confidence. Nevertheless,
we still predict growth over five years even in the face of a console
Ted Pollak, senior analyst–gaming industry, says, “The PC Gaming cycle. We have improved our accessory analysis this year and added
Hardware Market is in a rare scenario where every segment is going a new category for game recording and streaming products. Studio-­
up. We see a lot of people buying and upgrading personal and com- quality cameras and microphones, capture cards, and personal
pany subsidized computers with better parts, with the intention of lighting have become common items for high-end gamers and are
playing video games. In the Entry-Level, much of this revenue comes making their way down into the mass market.” 
from new gamers.”  
The “Worldwide PC Gaming Hardware Market” report series
The 2020 Entry-Level category is forecast to grow 21.7%, which by Jon Peddie Research – which covers 33 countries, notebooks,
is unprecedented and totally unexpected. The Mid-Range has desktops, DIY, and accessories – comes in three versions: Entry-­
bounced back from a slide; now in positive territory. At the High-End, Level, Mid-Range, and High-End, with a subscription price of
1440p+ display sales (spurred by more affordable offerings) created $27,500, including a global summary. Subscribers receive all re-
a chain reaction of upgrades as gamers configure rigs for 60+ ports twice a year. Reports are also available by segment at $8,250
frames per second.   per issuance.

Collaborate and work from
anywhere

The secure, global, cloud-based
platform for digital content
management and creation,
perfect for Animation, VFX,
Advertising and Games

Powered by:

6 cgw e d i t i o n 2, 2020

Collaborate and work from anywhere
The secure, global, cloud-based platform for
digital content management and creation,

perfect for Animation, VFX,
Advertising and Games

Making of Images ©2020 Netflix
a Legend

NETFLIX’S ‘CURSED’ TURNS THE ARTHURIAN MYTH ON ITS HEAD

BY KAREN MOLTENBREY

8 cgw e d i t i o n 2, 2020

There have been many retellings and in- Wheeler and Miller are co-creators/execu- puter-generated creatures and animals, and
terpretations of the Arthurian legends, tive producers, while Silenn Thomas serves fire/ash/embers/smoke/water. Most of the
particularly those involving Excalibur. as co-executive producer, and Alex Boden, Fey, however, use prosthetics.
And with the magic, drama, romance, myth- a producer. David Houghton is overall visual
ic battles, epic quests, and more, it’s easy to effects supervisor. “One of the decisions we made early on
see why the subject is so spellbinding. And in conversations with Frank and Tom was
indeed, the latest version of this medieval Cursed is set in a mythical medieval version that the magic was going to be visualized
myth, Netflix’s Cursed, follows that tradition, of England, although filming occurred at through elemental forces rather than more
although it places a unique spin on the tale Langley Film Studios, outside of London, with traditional wand/firework-type effects,” says
with familiar characters in very different a good deal of location shooting near Surrey Houghton, “more embedded in elemental
roles, all supported by a wide range of stun- and in North Wales, around an area called forces.” This can be seen in many scenes,
ning visual effects. Betws-y-Coed. VFX were used to transform such as when Nimue summons the Hidden
the soundstage and modern locations. with animated vines representing the
Based on the illustrated novel by Tom mystical, natural forces that give the young
Wheeler and Frank Miller of the same All told, the series contains 2,051 VFX woman some of her powers; or when she
name, Cursed re-imagines the story from shots: Episode 1 (390 shots) and Episode manipulates smoke, flames, and ash to her
the perspective of Nimue, a young woman 10 (360 shots) contain the highest number, advantage. Lightning also becomes a natural
with a mysterious gift who is destined to followed by Episode 4 (282 shots), Episode force that is manipulated by magical char-
become the powerful (and tragic) Lady of 2 (261 shots), and Episode 7 (229 shots), acters in the show.
the Lake. After her mother’s death, she finds with the remaining dispersed among the
an unexpected partner in Arthur, a humble other episodes. The work was shared mainly DNeg: Hawksbridge, Gramaire,
mercenary, in a quest to find Merlin and across five vendors: DNeg, Milk VFX, Mr. X, Village Massacre, Vines, Bear
deliver an ancient sword. Over the course of Goodbye Kansas, and Freefolk.
her journey, Nimue will become a symbol of DNeg’s work across its 822 shots in Cursed
courage and rebellion against the terrifying “We did just about everything,” says spanned the series’ 10 episodes and ran the
Red Paladins – Christian Crusaders whose Houghton. “I’m not sure what we didn’t do; gamut, incorporating just about every type
goal is to rid the land of druids and all mys- we didn’t do spaceships.” There is quite a of work, from environments, to creatures
tical/magical beings – and their complicit bit of blood from battles, massacres, and and digi-doubles, to fire and blood, and far
King Uther. brutality, as well as magical effects, all-CG more. This includes building some of the
environments and digital extensions, com-
Nimue draws strength from the magical
sword she possesses, imbuing her with DNeg built a large set extension for Hawksbridge port, including the sea.
amazing fighting skills. Everyone, however,
wants to possess this weapon, known as the Nimue and Pym approach the CG-augmented town of Hawksbridge.
“Sword of the First Kings,” as it is said that
whoever wields this Sword of Power shall
be the one true king. King Uther Pendrag-
on wants it, as do the Paladins. Cumber
the Ice King covets the sword, as does the
ruby-eyed shadow lord King Rugen, the
Leper King. At one point Merlin possessed
it, or more accurately, it possessed him,
turning the magician into a primal, vengeful
being. Nimue’s mother relieved him of the
sword to save his life, but he lost his magical
powers as a result. Now, believing the sword
is cursed, Merlin intends to melt it using
remnants of the long-ago Fey forge fires
from which it was crafted – fires kept in a
treasure room at Rugen’s lair.

The weapon, it seems, is the ultimate
double-edged sword for Nimue, as well.
It enables her to protect the mystical Fey
beings against the murderous Paladins, but
it also begins to nurture a darker side – just
as Merlin had warned.

Season 1 of Cursed, which comprises
10 one-hour episodes, dropped July 17.

edition 2, 2020 cgw 9

Nimue entangles a Paladin in practical vines interwoven with CG versions. extension was more straightforward but still
required a high level of detail. As was the
DNeg built and animated a demon bear that attacks the young Nimue. case with Hawksbridge, Gramaire required
the artists to remove many trees from the
more extensive environments, such as the layout right for all of the various angles, be- set, but not all of them – “these trees really
town and port of Hawksbridge, where Nimue cause we started off with the widest angle became an issue,” he adds. Similarly, the art-
tries to catch a boat while fleeing from her of Nimue and [her friend] Pym approaching ists built an environment extension beyond
village, but meets Arthur instead. Hawksbridge on horseback, which shows the limits of the physical set, which included
the spread of the town. But then there churches, towers, rooftops, and so forth.
“It involved a lot more work than what you are other scenes that feature that same
see,” says Houghton, describing the environ- environment and were filmed from several The sets for Gramaire and Hawksbridge
mental extension for Hawksbridge as quite different angles,” says Sulé Bryan, DNeg VFX consisted of the large town squares, a
an undertaking. supervisor on the series along with David number of side streets, city gates, and the
Sewell. “We had to make sure we had the port area of Hawksbridge. DNeg extend-
As Houghton explains, first, the locale correct layout to keep all those individual ed the towns above the nearest rooftops;
was surrounded by trees that all had to be shots composed correctly so we didn’t have they also created a vista of the entirety of
removed in comp, with DNeg building out to tweak each one individually.” Hawksbridge as seen when Nimue and Pym
the town and port from the existing set. first approach and for the distant Gramaire
Using art department drawings and period “On one side we had to replace fields as Arthur rides towards the town. “Like
reference, the studio added CG churches, of trees with the city beyond the set wall, Hawksbridge, [Gramaire] is a location that
houses, huts, a bridge, and a plethora of and on the other side, the sea port side, we is used several times across the series, and
objects, such as carts and wagons, barrels, replaced a huge bluescreen with an ocean, had to remain consistent while being used
fish netting, and so forth. As the village boats and ships, and so on,” he adds. again and again,” Bryan notes.
set was built nowhere near water, DNeg
further crafted moving boats and dynamic For the construction, the artists used This build included construction of
sails, as well as a jetty at the port, and even Autodesk’s Maya for modeling, Adobe’s Ector’s castle by DNeg, as well – an all-CG
added the sea itself. Everything within the Substance Painter and Pixologic’s ZBrush for set seen in both the background and in
surrounding harbor wall was a practical texturing, Isotropix’s Clarisse for rendering, close-up. Achieving the right scale for the
set with a large 40-foot-high bluescreen SideFX’s Houdini for effects and simulations, castle was difficult because it was always
where the sea was to be. DNeg added the and Foundry’s Nuke for compositing. visible in shots, which at times required
town beyond the wall, enhanced the set, some slight cheats.
extended the jetty by 90 feet into the sea, DNeg also built the large set extension for
and added the water and majority of the the village of Gramaire, the seat of power An especially complex environment build
ships and boats. for Arthur’s uncle, Sir Ector – a site that ap- was required for Nimue’s village, which is
pears in several sequences throughout the sacked by the Red Paladins, who slay or burn
“The most difficult aspect was getting the series. According to Bryan, this environment the Fey inhabitants, along with the village
itself. “Our brief was to convey an engulfing
sense of death and destruction,” says Bryan.
To this end, DNeg had to greatly enhance
the amount of flames as only minimal SFX
flames where allowed at the location, and
added other effects such as ash and smoke
throughout the sequence, where the shots
ranged from wide to close-up. The artists
also added 2D crowd elements shot against
bluescreen to increase the number of Pala-
dins and villagers seen in some shots.

In a 70-second Steadicam shot, we see
Nimue running through the village during
the chaos, trying to reach a young Fey boy.
“There’s nowhere to hide with a shot like
this. Usually when you have this sort of
shot, you can get away with certain things
in between the cuts. But when the shot
is over a minute long from start to finish,
everything has to work perfectly,” explains
Bryan. “It’s probably the equivalent of about
15 visual effects shots all in one. There are
multiple huts on fire, the added effects of

10 cgw e d i t i o n 2, 2020

ash and embers, and additional people.” live-action, and the vines that writhe on the Fingers of Airimid, a design from leaves and
The artists also added a huge 180-degree Paladin’s chest were animated entirely in thorns that appears on the skin of Nimue
FX with a procedural noise texture, whereas and the Fey elders when they are empow-
mountain range across from the lake that most of the other [vine] animation in the ered with magic or are angry and in danger,
edges the village, which is visible in this se- series was done in Maya, says Houghton. respectively. “Part of the challenge is the
quence as well as earlier ones. “Not only did way the effect comes on through the skin,
we have to create this large mountain range, All her life, Nimue has been taunted for almost like a blush,” says Meg Guidon, exec-
but we also had to get it into the right angle the strength of her magical powers and utive producer at Freefolk. “This is important
for all the sequences, especially the long considered an outcast by her peers and the because it is an emotional response in the
shot, where the camera is constantly panning elders, stemming from a demon-bear attack Fey characters, betraying their special pow-
backward and forward,” explains Bryan. when she was a child. She is saved by the ers. It is magic but has to look organic.”
Hidden, but not before she is “marked” by
Of course, there is a great deal of blood the bear across her back, leading the villag- According to Harin HIrani, Freefolk head
in this scene and others. “We never got to ers to believe she is cursed by dark spirits. of CG, the group explored multiple avenues,
the point where [the client] said, ‘There’s DNeg artists crafted the CG bear based on including an effects approach using Houdini,
too much blood.’ We really had to up the reference from Houghton and Miller, and to create a textural pattern that grew over
violence ante,” Bryan notes. choreographed the attack. As reference, a 3D model of Nimue the studio had made.
the artists examined footage of large brown Once they had settled on an effect they
Following the massacre at the village, bears, studying their gait, stance, and the liked, they streamlined the process in order
Nimue finds her mother, the high priestess way they shift their weight. to deliver the effect across many shots
Lenore, gravely injured inside a cave. She and to easily alter the timing. In the end,
instructs Nimue to take a sword to Mer- But this is no ordinary bear. “Frank wanted they ported the texture growth pattern into
lin. Thus begins Nimue’s true journey. a specific style for this creature, so we Nuke, where they could control the growth
had to obtain the right balance between a and timing on a per-shot basis, and then
Since she was young, Nimue has had the demonic bear and a real-world bear, which generate maps that could be applied in the
ability to call on the Hidden. In the series, ended up being a 20/80 split [respectively],” studio’s 3D software to render displaced ge-
she uses this ability to entangle enemies in says Bryan. Moreover, the animal was 4 feet ometry lit for the scene. These CG elements
thick vines, roots, and branches, some of taller than a typical bear, at about 12 feet were then integrated into the live-action
which are practical and interwoven with CG high. “It had to tower over her and overex- plates using Nuke.
versions animated by DNeg. In one particular tend his jaw, enough to eat her head and
sequence, a Paladin is encased by creeping torso in one.” “The biggest challenge with the facial
vines. He is cut down by his brethren and vines was the tracking process. We used
taken to an abbey to be cared for, where Freefolk: Nimue’s Facial Vines, different techniques on a per-shot basis de-
vines are seen moving in and out, and under, Creatures, Fey Camp pending on Nimue’s actions. Sometimes we
his skin. could purely use vector tracking inside Nuke,
While DNeg created the creeping vines in other times we needed a geometry track in
In the scene, the vines pulled from the the forests, Freefolk generated the so-called
Paladin’s mouth were practical elements
shot separately and combined with the

Freefolk crafted the leafy design that appears on Nimue’s face when she is empowered by magic.

1 1e d i t i o n 2, 2020 c gw

Freefolk extended the cave setting for the Fey refugee camp. with CG Moon Wing creatures that were
Milk built the subterranean environments of the Leper King and his treasures. falling to the ground engulfed in flames,”
says Guidon. “This included animating one
Moon Wing being executed by the Weeping
Monk; all the pleading and dying actions had
to be retrofit to the timing of the existing
greenscreen performance of a sword slash
by the monk.”

Freefolk also created the menacing black
spider creature that enchants Morgana as
she retrieves the sword, which Nimue has
tossed into a vast chasm out of frustration.
Furthermore, the team created a realistic
fawn, shown in close-up. Because the team
was unsure how much movement would be
required, they proceeded to build a muscle
system for the animal in Houdini should it
be needed.

Rounding out the studio’s work were
atmospherics, including a magical storm
summoned by Merlin, a scene requiring a CG
set extension of Pendragon Castle to add
the clifftop location and a wild, crashing sea
at its base.

[The Pixel Farm’s] PFTrack, and other times “We needed to build an entire burning for- Goodbye Kansas: Castles
a combination of both,” says Hirani. est around him, and populate the treetops
One of the more visually stunning all-CG
Freefolk, tasked with 200 shots, also environments was that of Pendragon Castle,
designed and built the convent where the handled by the team at Goodbye Kansas
dying root-wrapped Paladin was taken,
compositing the building to sit seamlessly Goodbye Kansas built Sir Ector’s sprawling castle.
into the wide establishing shot, which was
filmed on location. The studio also built Uther Pendragon’s stylized castle.

In addition, the team created set exten-
sions for the Fey refugee cave, where the
magical folk hid from the pursuing Paladins.
The artists at Freefolk designed, modeled, lit,
and textured the upper levels and wider set
extension, which provide scale to the live-ac-
tion set at ground level. “Having created the
upper stories of the camp, we then populat-
ed the rock faces of the camp with crowds
and added animated CG Moon Wings flying
in the air above the camp,” says Hirani.

In an early scene after the druid village at-
tack, the studio built an all-CG wall of flames
and forest through which strides the feared
Weeping Monk, who is hunting the magical
beings for the Paladins. Meanwhile, Moon
Wings fall from the trees and burn around
him. This was a 40-second continuous take
of the monk walking to the camera, shot
against greenscreen, with extensive effects
elements simulated in Houdini.

12 cgw e d i t i o n 2, 2020

Arthur and the Green Knight escape the CG flames as Paladins burn the windmill, destroyed by Mr. X.

under the supervision of Ditch Doy. This match the designs while also dressing the armies of Uther and the Viking Ice King. In
included exterior shots of the castle perched castle as the story required it. For modeling addition to these two main set pieces, the
on a ledge and the vista surrounding it, as and animation, they used Maya, as well as studio crafted a number of shots scattered
well as lavishly dressed interior set exten- Houdini for simulations and Chaos Group’s throughout the last six episodes, including
sions. As the series is based on the classic V-Ray for rendering. Compositing was done the all-CG character the Widow.
Arthurian legend, albeit from a new angle, in Nuke.
the artists at Goodbye Kansas made a lot The lower half of the windmill was
of nods to the medieval era as well as to the While Pendragon was the studio’s main practical, while Mr. X digitally extended it,
classic story of Arthur in the castle’s design. asset for the series, the team also created adding an extra set of vanes that were later
As such, most of the work was based on art the interior throne rooms for both Sir Ector’s destroyed by CG fire, crafted in Houdini.
developed in-house with Raf Morant, head and Uther’s castles, and extensions to the “It had to be a seamless match,” says Tim
of art department, in conjunction with the exterior of the executioner’s courtyard. “As Stevenson, VFX supervisor at Mr. X, of the
showrunners and Houghton. we had created a lot of parts in the medieval digital build. “There were a lot of drone
style for our Pendragon build, additional top- shots, moving shots, handheld shots, and
“Pinning down the look of Pendragon ups such as those were quick to incorporate, they all had to be tracked to where there
Castle was probably our biggest challenge,” and added some additional grandeur to the was no distinction between the top and
says Doy. “As the seat of Uther’s power in scenes,” says Doy. bottom portions of the windmill.”
the show, it had to evoke his character.”
Initially, the design was based on tradition- Collectively, the studio handled 126 Moreover, the CG fire was also mixed with
al medieval castles in Wales, but as the VFX shots. practical effects and had to seamlessly blend.
concept evolved, it moved more into the As the structure burns in the sequence,
realm of fantasy as the artists worked with Mr. X: Windmill Siege, more and more of the fire becomes CG until
Miller, who contributed a graphic, stylized Battle at the Beach, the Widow the roof collapses, at which point the fire is
aesthetic and influences from German all-digital, as is the windmill itself. “Before
Expressionism. Ultimately, the architecture When it came to Cursed’s big battle scenes, we destroyed it, we had to build a full-digital
returned to a more reality-based look, yet Mr. X was in the thick of it. This included version of the practical windmill – from the
still maintained a lot of Miller’s vision. the windmill siege when the Paladins trap cloth textures to the wood texture, it had to
Arthur, Gawain the Green Knight, and the be a one-to-one match,” says Stevenson.
Goodbye Kansas approached the build in Fey inside a windmill and attempt to burn
a modular fashion, constructing a medie- them out. Among Mr. X’s collective 326 VFX Eventually, the heroes and Fey file out of
val kit of parts that could be modified and shots were those pertaining to the large- the inferno, while Nimue, who is outside,
re-purposed. This enabled the crew to scale battle on the beach between the Fey manipulates the flames and turns them into
and tusked people, against the Paladins and a giant CG vortex that rises and blows back

Nimue battles five CG wolves in a CG environment, crafted by Milk. Culzean Castle’s CG set extension.

on the Paladins, burning them and creating of CG wolves developed from concept built the subterranean environments of the
cover for the Fey group. The vortex had to be through to animation, as well as a forest Leper King, which was mostly filmed on a
believable yet still have a magical component. environment in which Nimue fights with small set and then digitally extended. “These
them; and a dramatic waterfall environment were quite challenging to build, as we had to
“Fire is always difficult; there are scalabil- where a combat sequence takes place at integrate some floating set walls within the
ity issues and dynamic issues, especially the climax of the final episode. Milk’s brief design of the environment,” says Crowley of
when you are combining digital fire with also included a number of environments: the the Leper King caves. “The throne room was
live-action fire; there’s a believability factor CG Culzean Castle and subterranean world imagined as a vast cathedral with a touch
that has to be addressed,” says Stevenson. environments for the Leper Kingdom; the of backlighting – with a direct reference to
“We had some practical fire in the shots underground chasm at Cailleach; digi-dou- Frank Miller’s comic-art style.”
that we could match, but there was very bles of heroine Nimue and the Paladins for
little in terms of the forgiving nature to that the waterfall sequence; the green Fey forge Meanwhile, the artists filled the treasure
sequence because it was filmed in broad fire; and the glow and effects on the sword. room with gold coins and treasure spilling
daylight, one of the more difficult scenarios The studio also created additional environ- out all over the chamber, along with coffins
for doing believable fire.” ments and effects, including chopping off and copious riches. They also crafted the
the hands of the character Bors. green Fey fire, which Merlin steals.
For the epic battle on the beach in the
last episode, Mr. X used Massive crowd In one revealing scene, the sorceress Yeva According to Crowley, the most challeng-
simulation to turn 100 extras into 500 – casts an incantation that enables Nimue ing aspect of the overall job was the wolf
Vikings, Fey, Tusks, royal soldiers – according and Merlin to virtually meet for the first time, scene, wherein the Paladins dispatch five
to Houghton. The sequence had a number at Culzean Castle. Here, Nimue is given a (CG) wolves to hunt down Nimue as she is
of high-angle shots looking down onto the look at the past and at Merlin’s connection carrying the wrapped sword to Merlin. The
beach that were enhanced with digi-double to the sword, leaving her unsure whether to beasts trap her, and during the fight, she
crowd sims generated in Massive. When follow her mother’s dying instructions. climbs atop a rock and uses the sword to
on the same level as the action, Mr. X used dispatch the beasts one by one, behead-
elements of Fey and Vikings fighting, shot The sequence was filmed among the ing three – one in spectacular Frank Miller
against bluescreen, to increase the numbers ruins of Waverley Abbey in Surrey, which Milk style. Like the wolves, the environment in
in the battle sequence. “The choreography developed into a larger, grander build of a the scene is all-CG, with the exception of
was the hardest part because you are deal- castle city ruins. “There was only a little bit the actress and rock, which were filmed on
ing with so many shots of people in close of wall standing [on-location]. Most of the greenscreen.
combat and wide-angle shots, so planning castle was a CG set extension,” says Hough-
became crucial,” adds Stevenson. ton. Therefore, the look and textures of the “The wolf sequence was very tough. It
digital castle needed to match the unusual was shot in the studio, and just prior to the
Aside from the battles, Mr. X also created texture of the existing monastery ruins that scene, Nimue is running through an actual
the all-CG Widow, a reaper-looking charac- appear in the plates. forest and comes into a clearing, which was
ter that Mr. X helped design and build. Ini- essentially a greenscreen studio space with
tially, she was to be an actor with prosthetic The challenge was building an environ- just the rock in the middle,” says Houghton.
make-up, but she wasn’t scary enough for ment that could sit in many different shots “Milk had to then add the whole of the forest
Wheeler and Miller, so the artists crafted a and angles without Milk having to make be- environment around it.”
computer-generated skull-like face that is spoke separate builds, according to Ciaran
less rigid and expressive. Crowley, VFX supervisor at Milk. The Milk team designed the look of the
wolves from scratch, while the lighting style
Milk VFX: Wolves, Culzean Milk additionally built the subterranean and atmosphere they incorporated into
Castle, Caves, Waterfall environment for the Cailleach, an evil- those concepts (which included backlit
filled underground chasm where Morgana lighting, driven from Miller’s signature comic
Milk handled a variety of work on Cursed, encounters the Freefolk-made spider while style) was subsequently used as lighting ref-
crafting 390 shots encompassing a pack retrieving the sword for Nimue. Milk further erence by the DP and director on the shoot.

14 cgw e d i t i o n 2, 2020

Artists at Milk crafted this extensive environment, including the CG waterfall.

The team referenced Frank Miller’s wolf Nimue is shot with an arrow, then another, explains. “We then passed the character
from his comic “300” and created concept eventually tumbling from the bridge into a on to our creature FX team, which handled
designs for several types of wolf, in both pool of water far below. Not only did Milk all types of character deformation, such as
naturalistic and fantasy styles, with the final create the waterfall and the environment cloth and hair simulations; the groom for
creatures having a more realistic look but around it, but also the digital takeovers Nimue was achieved using Yeti.”
are larger than a typical wolf to make them required in the scene.
more menacing. In terms of Nimue’s fall, actress Katherine
“It is a big drama scene in a tiny space, Langford was shot against bluescreen on
The artists began crafting the beasts by which made it more challenging – with most the backlot on a harness with crash mats;
first researching wolf fur textures, coloring, of it completed and delivered in lockdown,” Milk removed the rigs and added the CG
and behavior. Sam Lucas, head of modeling, says Crowley. waterfall and the CG set extension at the
built the wolf models in Maya, then used bottom of the waterfall. In addition, the
ZBrush for the finer details around the paws, The sequence was shot outside on a studio did some digi-double takeovers for
face, and mouth, as well as for the muscle backlot against bluescreen. The set build the dramatic point when Nimue loses her
definition. The group used Peregrine Labs’ consisted of the bridge, with everything grip on Merlin’s hand and plunges down into
Yeti for the groom, giving the artists flexibility else created digitally, including the waterfall, the mist. The artists also made digi-doubles
to play with the look and feel of the fur, and digi-doubles of Nimue and the Paladins, for the Paladins who are killed and fall from
Ziva’s rigging and animation software for the the rain, and Merlin’s magical lightning bolts the bridge during the fight, allowing CG
muscle systems. from the storm he whips up as he slowly takeovers in several of the shots with added
regains his power.  blood splatters in the vein of Frank Miller’s
“The alpha wolf needed to be white; the comic art. 
second in command, black; and the other The artists developed the 3D asset of the
three variations were selected from real-life environment, while the FX team created Ultimate Cliff-hanger
wolves by the client – these were gray and the water simulations and the waterfall (in
two variations of brown,” explains Crowley. just seven weeks during lockdown), as the Season 1 ends with the waterfall sequence,
“Since wolves’ fur color is typically made up latter was broken up into several layers of but is it the end of the series? Many ques-
of multiple shades, we incorporated three simulation, including water surface, foam, tions were answered and issues resolved,
color maps to drive the initial variations for spray, and mist. but others remain. Reviews have been
one wolf’s fur, using root, mid, and tip color positive, leaving the door open to a Season 2
maps that would stipulate how a single hair For the digi-doubles, scans of the actors if Netflix chooses to open it.
strand would be shaded. We also added playing Nimue and the Paladins in the scene
randomized mutant hairs and melanin in the were provided by production and rigged “The way Frank and Tom told this story is
shader to push the individual looks. From and groomed with cloth sims applied to the very interesting. Initially, the characters feel
there, we also had to make two grooms for clothing. These were used in wide shots and unfamiliar to us in terms of the Arthurian
each wolf: a dry groom and a wet groom.” for takeovers when the characters fall into legend, but gradually the pieces come
the water. together as you start to realize who the
As Crowley points out, it was imperative characters are within the existing mythology,
that the fur looked and flowed naturally with “We wanted to cover all bases in terms of and that’s very intriguing and exciting, and
the dynamics. Likewise, it was important character detail, and what helped us mas- opens the door for a much bigger story to be
that the bulk and mass of these huge beasts sively was having a full-3D scan of Nimue told,” says Houghton.
were properly portrayed in the renders with for our assets team to begin modeling her
the multiple lighting setups. digi-double. All of the base mesh was done As for the magic, that was contributed
in Maya, then we moved to ZBrush to get by the visual effects artists, modern-day
Milk’s culminating sequence in Cursed all the extra detail in the skin and clothes. “magic folk.” 
has Nimue, Merlin, and Morgana trying to The textures were made in Mari and were
escape the Paladins. As they are crossing a either hand painted or derived from on-set Karen Moltenbrey is the chief editor of CGW.
narrow footbridge above a huge waterfall, element reference, with look-dev done in
Maya using [Autodesk’s] Arnold,” Crowley

1 5e d i t i o n 2, 2020 c gw

Kitbull Purl
Smash and Grab
Igniting
Passion

PIXAR’S SPARKSHORTS PROGRAM GIVES EMPLOYEES THE
CHANCE TO BRING THEIR OWN ANIMATED SHORT FILMS TO LIFE

BY KAREN MOLTENBREY

16Wind
cgw e d i t i o n 2 , 2 0 2 0

It’s not easy to take creative risks when tunities and see what stories they would radar. As such, not many of their colleagues
there’s so much at stake and so much tell, so we tried to release them from the knew what they were working on. But when
to lose. pressure of theatrical shorts,” explains Col- the fruits of their labor were revealed at a
Take, for example, Pixar Animation Stu- lins. “And every time the temptation is there company meeting, it was to the delight and
dios, known for its animated blockbusters for us to put ‘Kitbull,” or ‘Float,” or another of surprise of the entire staff. Now, everyone
that indeed are very expensive to make but the films in front of a Pixar feature, we have at Pixar is familiar with the program and
have been met with critical acclaim and to talk ourselves out of that. We have been excited about it.
huge box-office success. But, how do you down that path before, and this [Spark-
keep the spirit of taking risks alive in an en- Shorts] is something different.” Careful to avoid labeling or defining the
vironment like this? Such was the question program by its content, the studio released
posed by Lindsey Collins, VP of development Taking Risks the first three SparkShorts – “Purl,” “Smash
at the studio, to the executive team some and Grab,” and “Kitbull,” which are very
time ago as part of a broad conversation. A few years ago, as Collins stepped off different from one another in terms of story
Finding Dory, which she produced, and into and aesthetic – simultaneously to illustrate
Out of this desire to innovate and development, she was able to focus on the the diversity and breadth of the program.
challenge themselves, Pixar’s SparkShorts SparkShorts initiative and begin identifying The three shorts celebrated their world
program was born. those within the studio who might be ideal premiere at the El Capitan Theatre with a
choices to put into the director and produc- limited showing in January 2019, followed
SparkShorts gives people within the Pixar er roles of these shorts. by debut on Pixar’s YouTube channel. Since
community the opportunity to take creative then, the shorts have been played on the
risks that they are unable to take while With SparkShorts, there is no pitch pro- new Disney+ streaming platform.
working on a feature film. “There are people cess or approval process. As Collin explains,
here who we’ve had our eye on and thought, everyone spends a lot of time together in “We wanted to let the SparkShorts pro-
what if we gave them a relatively small the studio, and she and the executive team gram evolve and show us what it wanted to
amount of money and put some incredibly become aware of those individuals on be,” Collins explains.
talented and creative people around them, shows who have risen to the top, or were in-
and have them work with little oversight, formed about others who were working on So far, seven of the short films have been
to see what they would make, what stories concepts of their own. The only rules were released, and as of this writing, two exper-
they would tell?” says Collins, who heads that the short had to be animated, that is a imental filmmakers are in the process of
up the program. “Would we be surprised, narrative story, and that it is in-line with the working on the ninth and 10th SparkShorts.
and would it allow us to take some risks in a values of the company. “For now, [the program] is still doing what
relatively safe way?” we want it to do, which is to give people
Once selected, the director is given six the freedom and the opportunity to tell
Initially, there was no distribution model months and a modest budget in terms of some stories that are continuing to prove
for the shorts in the program. “Maybe that people-hours. “It’s such a different pace to themselves worthy of being told and people
was the safest way for us to really let them what they are all used to,” Collins says of showing they are worthy of the investment,”
do what they want to do and not feel like we the participants. “The creation process here says Collins, who remains hopeful that this
have to be looking over their shoulder all the is usually long, but here they made these program will continue for some time.
time, making sure that it’s perfect,” Collins amazing, great, heartwarming, different
says. “But deep down, we knew that if what stories. That’s been really rewarding for all of Meanwhile, Collins says Pixar is very cog-
they made was great, we wouldn’t be able us here.” nizant of maintaining an equitable balance in
to help ourselves from wanting to get [the terms of male-female directing and produc-
projects] out into the world and would figure The directors/teams are allowed to use ing, and to this end, is trying to have a couple
out how to do that. Nevertheless, it was a studio resources (equipment and talent). shorts in progress at a given time.
leap of faith but within a safe environment Often they rely on their own brain trust
that forced us to take some risks.” – friends and colleagues – to help them The goal of SparkShorts was to come
flesh out their ideas, and when the concept away with some fantastic stories. “That is
Pixar has been long revered for its theatri- becomes solid and official, they are assigned a huge win, having felt like we might have
cal short films, which have been associated a producer and a crew; they interview and identified some of the future storytellers
with a feature release. Alas, under the theat- select their supervising TD, giving those per- in our studio,” Collins adds, which is a big
rical model, the short filmmakers no doubt sons an opportunity for leadership exposure plus, as the current (or traditional) path to
felt pressured to make something that was as well, through the program. Collins points becoming a feature director is a long one
Pixar-branded and worthy of this presen- out that Pixar has had female supervising at Pixar, as the moviemaking process is
tation method, Collins believes. Thus, they technical directors from the SparkShorts typically very lengthy. SparkShorts, however,
felt the short had to look as perfect as the program who have later assumed supervis- shortens that path, making it a win-win for
feature film it was released with, and often ing TD roles on feature films. the studio and these hidden gems: the film-
second-guessed some of their instincts and makers themselves and their projects.
followed less-risky paths, and the produc- Collins acknowledges that the process
tion process became lengthy and expensive. was difficult for the first few filmmakers To date, the following short films have
because the program hadn’t been formal- been released under the SparkShorts
“We wanted to give more people oppor- ized yet, and they were flying under the program: “Purl,” “Smash and Grab,” “Kitbull,”
“Float,” “Wind,” “Loop,” and “Out.” Here

1 7e d i t i o n 2, 2020 c gw

we take a look at some of them, while the was struggling [with the situation,] my wife only dialog in the film, chiding the child for
others are detailed on our website, cgw.com, said, ‘You make comic books, you tell stories. not being normal. Shocked at his own out-
in the In Focus section. Why don’t you make a comic book to help burst, dad hugs Alex, realizing his mistake,
you work through this?’ ” he recalls. He be- and begins pushing Alex on the swings,
Float gan the process of doing just that, but it was bringing happiness to the boy. And then the
an emotional road for him to travel, making boy floats – as his dad smiles.
(7:04 runtime) it very slow going. Determined to complete
Director/writer: Bobby Rubio the journey, he opted to turn the story into “Float” is reflective of the Pixar house style.
Producer: Krissy Cababa an animated short film instead. This became a big advantage when, to save
“Float,” the fourth SparkShorts film released time and budget, the team used some assets
by Pixar, is a heartwarming story of a parent “I was going to do it in my spare time, available at the studio when creating the
and a child who is “different.” It is a story of and I drew a storyboard animatic for it and short. “We went ‘total independent film style’
many parents who are trying to cope with showed it to my friends, who suggested I and looked at what we could get our hands
such a child, although for “Float” director/ show it to Lindsey Collins, executive pro- on. Toy Story 4 was coming out, and I liked the
writer Bobby Rubio, it is a very personal one. ducer of SparkShorts,” says Rubio “I wasn’t look of it. So, we used a bunch of assets, back-
It is his and his son’s story. too sure, as this was a personal project, but ground characters,” says Rubio. In fact, viewers
I know her and thought I might as well get may recognize Alex’s house – it was Bonnie’s
In “Float,” a father discovers that his son her impression of the story. She loved it and house in Toy Story 4. It is just changed digitally
is unlike other kids in the most unusual way. thought it would be a good candidate for to make it look different in “Float.”
To keep them both safe from judgment, the SparkShorts program.”
Dad keeps the boy out of sight – but when According to Rubio, he wanted ‘Float’ to
his son’s ability becomes public, Dad must Rubio’s story unfolds as he, in the role of have a live-action feel and took a more gritty
decide whether to run and hide or accept the father, is playing with his son, Alex, in the approach to reflect the story. “It needed
his son as he is. front yard, but when Alex shows his unique that,” he says. “It’s about a father who is
ability to float, he immediately takes his experiencing depression, so there are a lot
“The inspiration for ‘Float’ came from my child indoors, out of sight from the watchful of dark moments. There are times we get
relationship with my son, Alex, who is on the gaze of neighbors and passersby. He keeps super dark with the backlight, especially
autism spectrum. When he was diagnosed his child hidden from the world, allowing with sequences in the house. We went as
10 years ago, I was trying to deal with it emo- him to float indoors. When he finally does dark as we could. Michael Sparber, director
tionally and had a difficult time processing take the youngster to the park, he tries to of photography, was amazing.”
his diagnosis,” explains Rubio. prevent Alex from floating by weighing him
down with rocks in a pack attached to the To this end, Rubio and the lighters looked
Rubio, a Pixar storyboard artist, is a child’s back. All seems fine until Alex catches to live-action movies for inspiration, and one
storyteller who makes his own comic books the other kids by surprise by exhibiting his scene of the father and son in the house by
outside of work. And this story was one that uniqueness. At this point, dad voices the the window is in full silhouette – dark with
had been gnawing at him for years. “When I black around them. The inspiration for that,

For director Bobby Rubio, “Float” is based on a very personal story/journey.

18 cgw e d i t i o n 2, 2020

according to Rubio, was the feature film E.T. “Float” looked to live-action films like E.T. for lighting inspiration.
Rubio’s team mainly used tools from the
enjoyed it. It was a big learning curve to go tion was to talk her way through a situation
Pixar box to craft the short. The biggest through in just six months, but I am confi- with them; the effect was not a good one,
technical hurdle, according to the director, dent that if I had to do it again, that I could,” she adds.
was the child’s hooded jacket when the says Rubio.
hood was pulled over Alex’s head. “We didn’t “People were not trying to communicate
have a huge budget, so we tried to cheat as Loop with me because I was creating a wall of
much as we could. We showed a little bit of friendly chatter,” the director says. “The
a hand going over the head, but we didn’t go (8:00 runtime) inspiration for the short came from the work
completely over the head. Also, I wanted the Director/writer: Erica Milsom I did as a volunteer.”
dad to take off the jacket completely, and Producer: Krissy Cababa & Michael Warch
that was difficult. We cut corners for that, “Loop” might be a short film but it is long In the film, the boy Marcus tries to com-
too,” says Rubio. “You see dad start to take on emotional content. The animated film, municate with non-verbal Renee by talking
the jacket off and then it cuts to another directed and written by Erica Milsom, tells a lot. Renee becomes agitated; Marcus,
shot, and we come back and the jacket is the story of two teens at summer camp: a frustrated. Eventually, they figure out how to
completely off of Alex.” talkative outgoing boy and a very quiet girl, communicate.
both of whom are paired up on a canoe
Creating the tears Alex sheds after his course. The girl, as we come to learn, is not Milsom also drew on her love of canoeing
father’s outburst was quite the challenge, simply shy. She is non-verbal, autistic. To while crafting the film. “If you are not getting
as well. But, there were just some elements complete their trip, they must learn how along or not in sync with your canoeing
to the short that were too important to skip, the other experiences the world so they can partner, you are kind of stuck. You can really
despite the difficulty they presented. “They communicate. work against each other,” she points out.
were really important pieces of the film, and “So, I thought this would be a good location
I am super grateful that Jane Yen, supervis- Milsom already had been making films on for fun and also some drama.”
ing technical director, and her team were the side and felt participation in the Spark-
able to pull them off,” says Rubio. Shorts program was a natural evolution to “Loop” is indicative of the Pixar aesthet-
her storytelling interests. ic. Using the studio’s pipeline, the group
Completing the short in the given time called on the animators’ talents to find
period required cutting a full minute from Milsom is a documentary filmmaker very specific and different behaviors and
the film. “That was a tough one. We went in Pixar’s Creative Content department, performances from the characters – for the
right down to the wire. But I feel like we were telling the stories behind the stories of the animation and acting, as well as in the voice
successful,” Rubio adds. “I still miss that one studio’s features. So when the SparkShorts performances.
minute. Maybe there will be a director’s cut opportunity came up, she was immediately
some day,” he says wistfully. interested, wanting “to explore working The biggest technical challenges, accord-
inside an animated story versus about an ing to Milsom, were the visual points of view
As for the float animation, the timing animated story,” she says. (“Loop” aired in and the sonic points of view of the girl, Re-
couldn’t have been better to tackle that, as early January, and now Milsom is directing nee. “We had learned that people with au-
many of the animators were just coming off an upcoming series about the studio that tism have a different sensory experience of
Incredibles 2, in which Jack-Jack flies. “We also will be shown on Disney+.) the world, that they might have an amplified
were looking for more of a graceful Peter sense of the world around them,” Milsom
Pan type of gliding across the screen for Prior to working for Pixar (she has been explains. “We only had a short time to make
Alex, as opposed to Superman’s powerful there for approximately 15 years), Milsom the film and had to define an entirely differ-
directional flight or Jack-Jack’s tumbling,” worked within the disabled community, ent look for Renee’s point of view.” Danielle
Rubio points out. later teaching a class for disabled adults Feinberg, cinematographer, then asked
in her neighborhood where she had met Milsom to provide some information on her
Despite the emotional toll the film took, many people who communicated through idea of what this POV was, which Feinberg
Rubio relishes the opportunity to direct. It gestures rather than words. Initially, her reac- then sent out to the lighters, who were not
also gave him the chance to learn the pipe- yet on the project, asking for examples of
line, something he never experienced before different kinds of lighting.
as a storyboard artist. “As a director, you get
to see the pipeline from the beginning to
end, from story to character designs, to the
art department and to the sets – how they
are built – and animation,” he says. “This is
a totally different animal from traditional
animation [which he previously did at Dis-
ney].” Plus, there are the music and lighting
elements, too.

“All the way from beginning to end, it
was a big education for me and I thoroughly

1 9e d i t i o n 2, 2020 c gw

“Loop” filmmakers considered the visual and sonic POVs of the autistic Renee. Milsom. “They are all such superstars in this
industry, and their care and attention, and
“We got five potential looks from people or overwhelmed. Moreover, the camera de- contribution to the story, is very deep.”
around the studio. They were amazing. We partment, led by DP Sylvia Gay Wong, made
shared them with our consultants at the sure that when you’re looking throughout It goes without saying that having the
autistic self-advocacy network to get their Renee’s POV, eye contact is not made. characters talk (most of the other Spark-
opinions. Danielle and I both liked one in Shorts do not have characters who speak)
particular but felt we should choose the one “The details and the careful consider- presented some hurdles. Not only was it
that feels most appropriate to them,” says ation that went into the film are significant,” more work for the animators, but recording
Milsom. The group, in fact, chose a com- Milsom notes. the voice actress for Renee did not follow
pletely different look, almost the opposite of the typical Pixar process. With assistance,
what Milsom and Feinberg had liked. In fact, the complexity of the film is the filmmakers found Madison Bandy, who
apparent throughout, even in the water voices Renee. She is autistic and can be
“The one we chose had a de-saturated and hair simulations. Marcus wears a hat, overwhelmed due to sensory overload.
look and these beautiful bloom lights, and which cuts down on some of the sim work, Therefore, her voice performance was
they chose something that has a deep while Renee was given a hairstyle that was recorded at her home and captured her
saturation in the center and the lights not overly complicated to simulate. “She is performing naturally – when she was happy,
bloomed but in a different way, harsh on the a young woman of color, so on a technical stressed, and so on.
highlights,” Milsom explains. “But they look level, emulating darker hair is a little easier to
beautiful in the film. So, whenever you are do than lighter hair,” Milsom points out. Milsom credits her team and her produc-
with Renee and looking at the world through ers for the project’s success, pointing out
her point of view, the world has this particu- Also, the two characters are on water for that it takes a partnership to make a film like
lar look, this deeper saturation in the center a large part of the film, requiring water sims. this, and producers Krissy Cababa and Mi-
and bigger highlights on the sides. It actually Effects lead Kylie Wijsmuller, who was fairly chael Warch were always right there. “They
amplifies a little if there’s more emotion and new to Pixar, figured out the water cycle, the were very much like the strategists. I was the
the highlights get even hotter.” waves generated by the canoe. “I knew the confidence lady. We’d talk about how great
water would be difficult, but I hadn’t really we were going to make this, but they would
There were other considerations, as well. thought about the water reacting on the strategize with people and make sure we got
Kenny Pickett, sound designer, wanted to sides of the canoe,” says Milsom. things done. They were truly great partners.”
know how Renee would experience sound,
so the sound effects are different from her The simulation team further tackled the Milsom steps away from this project with
perspective than from that of Marcus. It also fit of the life jackets on the characters and a newfound appreciation for the animation
differs depending on whether she is calm moving them in such a way that they did not process, being inside of it and collaborating
look too rigid.. with artists to bring two little somebodies,
as unique people, to life in the animated
“The simulation team was so great,” says form as Renee and Marcus.

“I feel they are really honest and true
and authentic as characters, but they’re
still designed, and their way of being in the
world was carefully considered. It was one
of those amazing artistic experiences that
only comes in animation, because you have

Animators used very specific and unique behaviors and performances for the characters.

20 cgw e d i t i o n 2, 2020

“Out” uses a unique painterly style. When the parents arrive in the short, the “fun” begins.

to design, consider, discuss, and create special attention to dad, while Greg, in Jim’s never really tried to create a moving painting
every single part of your story,” says Milsom. dog body, tries to hide a beloved photo of before at Pixar, and we were still defining
“How does Renee move her hand under the him and Manuel from his mother, with many what that would look like while we were
canoe? How does Marcus flip around that hilarious close calls. Mom eventually pours making the film,” he says. He praises Thomp-
reed when you’re only looking at his hand her heart out to the dog (who is actually her son and Pienaar, along with DP of camera
in certain shots? I had no idea that those son), saying she would like to tell her son that Matt Silas and supervising technical director
would be some of the most challenging she and his father hope that one day he will Gordon Cameron, as the “brains” behind it
shots to get right, because it was such a find someone who loves him as much as they all and keeping the group focused on staying
subtle and emotional gesture.” do, and that he will make their son happy. true to that painted aesthetic.

In addition to coming away with a love Greg and Jim return to their own bodies, “In film, we rely on depth of field to com-
for animation and connecting on a story so and Greg later introduces Manuel to his par- municate where the audience’s focus should
deeply, Milsom also had the opportunity ents, who welcome him. As for the cosmic be, but we didn’t know what that would look
to learn about autism, which she says was cat and dog, they return to the rainbow, their like in a painted render,” Hunter says. “We
a treasured take-away. “It’s hard making work complete. didn’t think we’d be able to figure out how to
films, particularly smaller stories with unique do that in our production’s time frame. But
characters that you haven’t seen before,” The short film is written and directed by Colin came in one day with a big smile and
she says. “It was a wonderful journey.” Steven Clay Hunter, an animator at Pixar. “I said, ‘I think I figured it out.’ And he did!”
wanted to create a story for my closeted gay
Editor’s Note: “Loop” recently was named self, something silly and funny like the car- Working on the experimental short was a
SIGGRAPH 2020 Computer Animation Festi- toons I used to watch growing up,” he says. big adjustment for the team, which is used
val’s Best in Show. “I wanted to create myths for LGBTQ kids, to to working on big-budget feature films,
inspire us and get families talking.” for which they are given ample time and
Out resources to polish their work to perfection.
The story unfolds using a unique painterly “But on an ‘indie film,’ you have to learn to
(9:33 runtime) style that is very atypical from what we get your strongest ideas out in the shortest
Director/writer: Steven Clay Hunter are used to seeing from Pixar – yet, that is amount of time possible, and then let it go.
Producer: Max Sachar exactly the spirit of SparkShorts, to experi- That’s an easy thing to say but an even hard-
The seventh in the series of SparkShorts, ment with story and style. er thing to do,” Hunter says.
“Out,” released in May, tells the tale of two
guys and a dog who are packing up to move, According to Hunter, the Little Golden Nevertheless, the rewards were plentiful.
when the parents of one of the men arrive Books served as a big inspiration, as the Hunter has helped many a director tell his or
unexpectedly to assist. The problem is that “Out” crew were looking to create some- her story but had never sat in the director’s
Greg’s parents are unaware that he is hiding thing that had a bit of childlike wonder to it. chair himself before this. “So when a ques-
a secret: He is gay and in a relationship with Mary Blair’s concept paintings from Alice in tion came up, I suddenly couldn’t turn to
Manuel, and is moving from a small town to Wonderland greatly inspired them as well. the director to find out what to do. I was the
a big city, where he believes his relationship one who had to have the answers!” he says.
will be more accepted. Greg quickly and Hunter guided his team, averaging a “I had to learn what kind of filmmaker I was,
quietly ushers Manuel out the back so his dozen people at any given time. They utilized what kind of storyteller I was. And, I slowly
parents do not see him. Manuel complies Pixar’s in-house pipeline, using Presto for rig- gained that confidence as we made [the
but wants Greg to tell his parents the truth. ging and character animation, and recycling short], but it wasn’t there when we started
as much imagery as they could from the the process.”
Meanwhile, a cosmic purple cat and pink studio’s feature-film backlots, with sets lead
dog emerge from a rainbow to watch the Kristifir Klein filling in the modeling gaps. Hunter especially enjoyed working in the
situation unfold, and they enchant the collar Rendering was done with Foundry’s Nuke, as smaller teams and figuring out how to make
on Greg’s dog, Jim. As the parents make Colin Thompson, look-dev lead, worked with films and still tell great stories, “which is
themselves at home, Jim’s collar magically DP of lighting Andrew Pienaar to achieve the what Pixar is all about,” he adds. 
attaches itself to Greg, and the two switch desired lighting and painted brushstrokes.
bodies. Jim, now in Greg’s human body, pays Karen Moltenbrey is the chief editor of CGW.
The biggest technical hurdle, contends
Hunter, was figuring out the look for the
short and how they would achieve it. “We’ve

2 1e d i t i o n 2, 2020 c gw

Universal Scene
Description

PIXAR DOES A DEEP DIVE INTO USD AND ITS EFFECT ON CONTENT CREATION

BY F. SEBASTIAN GRASSIA AND GEORGE ELKOURA 

22 cgw e d i t i o n 2, 2020

For as long as we have been creating While making Toy Story, an issue of scale required custom toolsets.
digitally synthesized images, we have
needed ways to describe the 3D scenes easier for them to reuse components. in a reasonably modular way, but it was
we are synthesizing in a way that is With these and other organic improve- expensive to maintain and did not help with
mathematical enough for computers to any of our scale problems.
understand, but which is also understand- ments, Toy Story was possible, but also
able and manipulatable by technologists illuminated many ways in which our pipeline Finally, in 2010, the industry received a
and artists. needed to become more scalable and more quantum leap forward on the interchange
artist-friendly. problem, when Sony Pictures and Industri-
This article tells the story of some of the al Light & Magic released its open-source
challenges Pixar faced in describing, sharing, THE INDUSTRY EVOLVES & collaboration, Alembic (see “Share and Share
and transporting 3D scenes as our pipeline PIPELINES GET MORE COMPLEX Alike,” CGW, Summer 2019 issue). Alembic
evolved over 25 years, from creating the first As more studios began making fea- was designed with rigor and deep knowledge
feature-length computer animated movie, ture-length CG animation, more varied and of VFX pipelines to address the interchange
to making it possible for our productions’ ar- powerful software became commercially of “baked” (or time-sampled) geometric
tistic visions to keep growing in richness and available, which was terrific because it ex- data, with vetted schemas for geometric
collaboration. Thus, we present Universal tended artistic reach, but it also made pipe- primitives and transformations, and a data
Scene Description (USD), Pixar’s open- lines more complicated. The three problems model that abstracted “file format.”
source software for describing, composing, of scale only grew (and still do!), but now
interchanging, and interacting with incredi- the industry faced an additional problem of That abstraction was critical because
bly complex 3D scenes. data interchange, since it was rare for any it allowed Alembic to gradually deploy
one vendor’s product to read data in its full formats like Ogawa that addressed some
USD is a cornerstone of Pixar’s pipeline richness from another vendor’s – some- of the important scalability issues for
and has been on a rapid and broad adoption times not even products within a vendor VFX, such as being able to store massive
trajectory, not only in the VFX industry, but family could speak to each other. amounts of animated data in a file, while
also for consumer/Web content and game primarily paying the cost (IO and network
development. Many VFX studios, including Pixar, built traffic) for only the pieces of data a partic-
pipelines around in-house software, but with ular application/node needed for its task.
BEFORE THERE COULD BE TOYS entry and exit points for data from many Alembic arrived (not coincidentally) at a
When Pixar set out to make Toy Story in the vendors. Formats for 3D interchange were time when many studios were lighting in
early ’90s, we had industry-leading, com- available, both open and proprietary, such as different applications than they were rigging
mercially-available products at the front and obj, Collada, and FBX, but none could satisfy and animating, so its “bake out the rigged
back of our 3D pipeline – Alias’ Power- the goal of rich (in terms of schema and animation into time-sampled mesh data”
Animator at the front for modeling, and kinds of data), universal interchange. approach was exactly what was needed.
Pixar’s own RenderMan at the back. But to
handle the scale of making a feature-length Studios developed different strategies to To keep the scope and mission of the
animation, we needed to invent or improve deal with interchange; Pixar built a “many project tight and manageable, Alembic
a suite of custom tools and data formats for to many” conversion system called Mojito stayed focused on interchange of “flat” data,
everything that needed to happen between that defined a core set of data schemas free of any concerns of composing multiple
modeling and rendering. into which you could (in theory) plug any Alembic archives together, proceduralism,
DCC in as an input and get out a data file for execution, rigging, or even a run-time scene
“Scale” here impacts several aspects of any other DCC (or renderer, though we only graph on which such behaviors could be
production that weren’t quite as daunting ever implemented support for RenderMan). layered. Within several years, Alembic had
when making Pixar’s earlier projects (short Mojito allowed us to adapt to new software deeply penetrated the VFX industry, and the
commercials and effects): number of artists
who needed to be simultaneously working
on the same project, complexity of the envi-
ronments, and number of acting characters,
number of assets, sequences, and shots.

We had already figured out that one
important component to successfully work-
ing at feature-scale was to separate out
different kinds of data into different files/
assets. While “appropriate format” was a
consideration, more important was that by
separating out geometry, shading, and ani-
mation into different files, we enabled artists
of various disciplines to work independently
and safely, simultaneously, and made it

2 3e d i t i o n 2, 2020 c gw

lem that a solution was indeed needed.
The solution? USD.

Finding Dory became the first feature film with a USD-based pipeline. WHAT IS USD?
USD began as an experiment in 2012 to see
majority of commercial 3D vendors support later adding simulation workflows. if we could take the low-level data model
Alembic import and/or export. To Presto’s scene graph library, we added and “composition engine” at the heart of
Presto, where it served a rich scene graph
Pixar did not adopt Alembic in its pipeline, and refined all the features that were nec- and anim curve-driven execution system,
primarily because from 2008-2010, we had essary to bring together all the disciplines and build atop it instead a light, highly scal-
been developing our own time-sampled and workflows that we had so far accumu- able scene graph that practiced lazy data
interchange format, called TidScene, with a lated. The most fundamental feature was a access, embraced time-sampled data, and
slightly different approach to the problem… layering system that is often compared to facilitated the creation of “schemas” (think
but we’re getting a little ahead of ourselves. Photoshop’s image layering paradigm. Ref- Mesh, Xform, Material, etc.) to foster robust
erencing, inheritance, and variants complet- scene interchange.
REFINING WORKFLOW AND ed the initial set of composition operators
‘COMPOSITION’ that formed a robust composition engine, Our proof-of-concept came very close to
At around the time Ratatouille entered and which, unbeknownst to us at the time, the performance of TidScene on equivalent
development, Pixar was growing hungry for began the underpinnings of the technology large scenes, while opening the door to the
revamping its toolset, which was beginning that would later become USD. full suite of Presto’s composition operators
to show its age; not only was our anima- (though we chose to leave out a few of USD
tion and lighting system nearly impossible We had initially targeted Toy Story 3 for performance and/or complexity reasons).
to adapt to the multi-threaded future of (and missed it) for early adoption of this However, we believed we needed to be much
performance, but our pipeline had also brand-new ambitious system and knew it more than an order of magnitude better than
grown complicated. Any given shot relied was going to be difficult to achieve. In order TidScene to satisfy the growing needs of the
on dozens of different file formats and to make it more tenable, we stopped short studio for the next 10-plus years.
data abstractions: We had one format for of adding support for the back end of the
“primary” geometry, but another for UV sets pipeline: shading, lighting, rendering, and so We made good progress toward that goal
and other primitive variables that were too on. That was going to be left to the existing by making data structures thread-safe/
big for the primary format, and a third for systems. That meant: interchange. efficient and algorithms multi-threaded at
simulation-generated geometry overrides. all levels of the USD software stack. The
We had one format and language for shad- We now needed a way to translate data development of “native scene graph in-
ers, another for shading networks, and yet a born in Presto to the existing back-end stancing” to augment schema-based point/
third to describe how materials were bound systems. In order to do that, we invented a array instancing allowed us to load scenes
to geometry. It was time to reconsider our new scene transportation system called Tid- containing millions of primitives in seconds
tools, workflows, and pipeline. Scene. TidScene was built using Berkeley DB on modern workstations.
and was primarily targeted toward efficient
The Menv30 project was kicked off, later transport of time-sampled binary data and Performance has consistently been a top
renamed Presto when released. Menv30 rapid preview imaging. As we began deploy- priority for USD, and we continue to make
was an opportunity to take all the learnings ing TidScene in more parts of the pipeline, small and large improvements. But while
of the earlier toolset and develop them we kept encountering the need to add more performance is a key “wow factor” for USD
into a cohesive and well-structured set of and more features to it, such as referencing that facilitates artists’ ability to iterate, there
libraries. Perhaps most importantly, the and inheritance. are several other important reasons why USD
project sought to represent all of our work- has been so quickly and broadly embraced.
flows using a single file format, menva, in Having TidScene meant that we were
a single application that could easily allow implementing things twice, as we kept COMPOSITION ALGEBRA
artists to transition back and forth through wanting to add more features that we had If there is any magic in USD, it is the “com-
a fluid and malleable pipeline. We started already implemented in Presto. Worse, the position engine” that provides the ability to
by targeting rigging, animation, and layout, implementations grew inconsistent and combine “layers” worth of scene description
incompatible with one another. in many useful ways, and lets users make
non-destructive edits. The rules by which
This became a significant enough prob- this powerful combinatorial engine works
have been refined over several different in-
carnations, starting with work pioneered on
A Bug’s Life, and protected by more than a
dozen patents that Pixar released for public
use as part of open-sourcing USD.

Understanding the full implications of
the composition rules can be daunting, but

24 cgw e d i t i o n 2, 2020

simple uses are easy to understand, and References allow us to bring environ- Presto’s imaging technology. We saw a won-
the complete behavior can be described ments into sequences and shots. At all derful opportunity to build a generic imaging
as the recursive LIVRPS (pronounced “liver “levels of referencing,” USD allows you to ex- architecture that we could plug in to the
peas”) algorithm, documented on openusd. press overrides on the referenced scenes in various proprietary tools that needed a 3D
org. We refer you to the website for deeper a uniform way, so once you learn the basics viewport. So, Hydra was born: a multi-head-
exploration of USD’s composition arcs, but of “reference and override,” you can use it to ed beast, each head representing any one of
we will describe several of them here as solve many problems. our scene formats and associated tools.
they pertain to addressing the scalability
problems we discovered way back on Toy One significant source of pipeline Hydra was originally a state-of-the-art
Story. Note that each of the composition complexity is the number of “variations” of OpenGL-based render engine capable of
arcs can be used to solve other problems as published assets that are typically required displaying feature-film scale assets in as
well, and one of the things in which we have to provide the level of visual richness that close to real time as we could get. Supporting
invested heavily is trying to make sure that audiences expect from modern CG films. the idea that Hydra should give high-fidelity
any possible combination of composition USD provides variant sets as a composition feedback to artists, we looked to RenderMan,
arcs behaves in reasonable ways. arc that allows multiple different versions of our production final-frame renderer, for the
an asset to be collected and packaged up ground truth of how certain scene geome-
Layering for Multi-Artist Collabora- together, providing an easily discoverable tries and parameters ought to be interpreted.
tion – Layering in USD is similar to layers in “switch” for users of the asset to select Along with the ability to embed Hydra into
Photoshop, except that in USD each layer is which variation they want. This switch as many applications as needed, we began
an independent asset, and often, each layer (called a Variant Selection) is available in the to fundamentally expand on the goals that
will be “owned” by a different artist or de- USD scene regardless of how “deeply” (via OpenSubdiv pioneered: high fidelity real-time
partment in a pipeline. While the modes by referencing and layering) it was defined in feedback that looked consistent everywhere.
which data in different layers can be merged the scene’s composition structure.
together is much more restricted in USD Today, Hydra has grown into a much rich-
than it is in Photoshop, layer compositing al- One of the most powerful aspects of Vari- er architecture that supports not only mul-
lows multiple artists to work simultaneously ant Sets is that they can vary anything about tiple “heads” (scenes and applications), but
in the same scene without fear of destroying an asset, and an asset can have multiple of also multiple “tails” (renderers). We factored
work that another artist is doing. them, nested or serial. In Pixar’s pipeline, it out the OpenGL renderer into its own Hydra
is common for assets to have a modeling back end that we now call Storm (though
One artist’s work may “override” another’s variant set, one or more shading variant sets, many folks still say “Hydra” when they mean
because their layer has higher precedence and rigging and workflow LOD variant sets. to say “Storm” – and we’ve had to come to
than the other artist’s, but since layer prece- terms with that, well, we’ve mostly come to
dence in a sequence or shot in a CG pipeline And More– USD also provides com- terms with that).
often corresponds to stages of the pipeline, position arcs for broadcasting edits from
this is not often a problem. When it is, the fact one prim to many others (Inherits); special We’ve also implemented a RenderMan
that each artist works non-destructively with references called Payloads for deferring the back end for Hydra. We had already spent
respect to other artists’ work means we can loading of parts of a scene, to allow users to an enormous effort integrating Hydra into
deploy a range of tools to handle situations in craft manageable working sets; scene-graph our various applications, like Presto, and
which something unexpected happens. instancing to not only control the weight now almost for free, we are able to see the
of the scene graph, but be able to reliably scene as rendered by RenderMan. This has
Referencing and Variant Sets for Asset inform renderers and other consumers what the promise to be transformational for our
Complexity – Referencing in USD is not dis- things in the scene can be processed iden- users, but we’d be remiss if we suggested
similar to referencing features in packages tically; and many other features to facilitate that we’re already able to fully benefit from
like Autodesk’s Maya, but more powerful non-destructive editing of large scenes. this technology. Truthfully, we’re not yet able
in that careful consideration has gone into to take full advantage of its full potential for
specifying how multiple references can HYDRA IMAGING ARCHITECTURE several reasons.
combine as we build up assets, aggregate One of the key tools that allowed TidScene
assets, sequences, and shots via chained to succeed as a project was an inspection Chief among them is that some data
and nested application of referencing. tool called tdsview. The tool allowed users required for high-fidelity rendering isn’t
to inspect the contents of a TidScene available in our Presto scenes early enough
References allow us to build simple or file visually in 3D and gave a very fast way in production. Similarly, our scenes are
complex “component” assets out of mod- to debug shots. As we were going to transformed significantly before they hit
ular parts. References also allow us to build replace TidScene with USD, we needed to the renderer for final-frame production. We
up environments out of many references provide a corresponding tool, with perhaps know we’re not done yet, and we’re excited
to individual component assets – some of loftier ambitions. by the future potential workflows we’ll be
the costs will be shared at run time among able to achieve.
many references to the same asset, and The heart of such a tool is a fast 3D
when we enable those references to be viewport, and we wanted to build as good a PRACTITIONER TOOLSET
instanced, that sharing increases substan- viewport as we possibly could. Also by then, USD was built by practitioners, for practi-
tially more. some of us were beginning to itch to replace tioners, and one of the ways in which that

2 5e d i t i o n 2, 2020 c gw

matters is the investment in tools and in some cases, even without any coding), produce better performance than the old.
features whose goal is to make working one can: And the best way to make sure your soft-
with USD data easier and transparent. USD ware stays fast is to measure it continuously.
contains, out of the box, a suite of tools for  Teach USD how to ingest new
working with USD and the various external data sources. As we started getting renders that match
formats that may feed USD, including: perfectly, we grew more confident that we
 Use your own asset management could switch the show over without negative-
 Binary and ASCII encodings. system and “asset resolver.” ly impacting our users. With a flick of an en-
 Fast, powerful preview and vironment variable, we switched Finding Dory
debugging tools.  Add new schemas. to start using USD and never looked back.
 Tools for creating more USD. As an example of how flexible these (and
other) plug-in points make USD, Google was While we were rolling out USD on Finding
OPEN SOURCE, OPEN ARCHITECTURE able to provide Draco mesh compression for Dory, we started reaching out to trusted
Pixar had experimented with open source USD entirely as a plug-in, by combining the partners in the industry, to whom we gave
through its first open-source project: powers of USD references and file format early cuts of the software. Early partners like
OpenSubdiv. With OpenSubdiv, we wanted plug-ins. Luma Pictures, DNeg, and Animal Logic, and
to make sure that our vendors were all using With the various USD technologies a little later DreamWorks, MPC, Rodeo FX,
the same subdivision algorithms that Ren- roughed in, it was time to plan their rollout Weta, and many others provided valuable
derMan was, so that artists saw consistent both internally to Pixar and to the broader insight and use cases beyond our beloved
and reliable results across the pipeline. We community. enclave that ultimately led to making USD
were starting to see the value in changing useful to so many in the industry.
from proprietary, closed software, to em- ROLLING OUT USD INTERNALLY
bracing open source as a way to strengthen AND PUBLICLY We chose SIGGRAPH 2016 as the event
our technology and increase its longevity Finding Dory was targeted as the first project where we would officially push the “make
by turning critical parts of our toolset into to adopt USD at Pixar. We did hit that goal, public” button on the USD repository
shareable, openly available libraries that we and Finding Dory became the world’s first hosted on GitHub. We all gathered around a
hoped would become industry standards. feature-length film with a USD-based pipe- round table set up for one of our USD Birds
line. The production also took on a significant of a Feather-style meetup with our small
With USD, we wanted to bring that same re-thinking of the back end of the pipeline, community at the time. We pushed the
consistency and reliability to the compo- taking on a pathtracing-first version of button right as folks started streaming in
sition of the whole scene. However, unlike RenderMan as well as new lighting workflows. for the meeting. Once everyone was seated,
OpenSubdiv, USD rests upon the lowest lev- Given the rest of the innovations happening we announced that USD is now public on
els of Pixar’s 3D software foundations, but on that show, USD’s rollout was relatively GitHub. It was the culmination of much hard
the studio strongly supported the project by uneventful, exactly as we had hoped. work for many of us on the team, and it was
allowing us to incorporate Pixar’s proprietary overwhelming to see the support and the
software stack into (what we hoped for In order to achieve a quiet rollout, we had enthusiasm that had already been garnered
originally, and then committed to in 2014) an a couple of tricks up our sleeves. The first is by that time.
open-source project. that we generated USD files along with our
old format, TidScene, the latter files being As of 2020, all geometry, shading net-
For USD to succeed in its “universal the ones used for actual early production. works and bindings, rigid-body, hair- and
scene interchange” goals, it is import- Our QA department also built a tool, bcsys- cloth-simulation, and large-scale skeletal
ant that 3D vendors adopt support for it tem, that would render a given shot twice crowd animation travel through Pixar’s pipe-
natively, and it was clear that a well-sup- – once with USD files (not yet made visible line in USD. These different kinds of data will
ported open-source project was the only to production) and once with TidScene files. generally be in different files, for workflow
viable way to foster that kind of adoption. The renders generated by the TidScene files reasons (and there may often be multiple
Additionally, as we will discuss more below, were ground truth and against which the shading USD files for a single asset, again for
sponsoring an open-source project that USD files were compared. workflow). However, being able to encode
literally targets entire pipelines has been a them all as USD brings great simplifica-
fantastic opportunity for productive and As long as the two renders were differ- tion and amplification of the power of our
enjoyable collaboration with many studios ent, we kept fixing bugs quietly behind the toolset – not just for ease of interchange,
and vendors in the industry. scenes. The bcsystem tool not only tested but also for leveraging a common language
the renderer, but the whole pipeline that got and debugging suite between departments
Because USD needs to plug in to most us to the final picture. Along with bcsystem, and artists.
applications in any given pipeline, a critical a fairly big-hammer black-box tool, we also
factor for its success is the degree to which had started amassing a big suite of perfor- Given the advent of MaterialX and USD’s
it can be customized, without impacting mance and regression tests. Our teams are ability to consume and encode MaterialX
the “common definitions” that enable fiendishly fond of regression and perfor- networks, there may be a future where
interchange. Out of the box, USD comes mance testing, keeping track of dozens our shaders themselves will be expressed
with reasonable defaults for all “pluggable” of metrics across thousands of tests. We by standardized nodes in USD vs. custom
behaviors, but by writing a little code (and wanted to make sure that our new pipeline shading nodes driven by in-house OSL and
was going to be built on software that could GLSL code.

26 cgw e d i t i o n 2, 2020



USD IN VFX PIPELINES AND BEYOND The complexity of Coco is apparent. community that has grown around USD
Today, visual effects and animation studios continues to inspire us and make significant
throughout the world have either fully rolled arbitrarily richly-composed scenes and can contributions that drive the technology
out USD or are experimenting with it for be consumed just like any other USD asset, forward. We’ve added features to USD that
rollout within their pipelines. At SIGGRAPH but is suitable for Web/network transmis- were suggested for use cases we never
2018, DreamWorks gave a talk titled “Zero sion and archiving. Apple relies on USDZ for before imagined, and we hope to continue
to USD in 80 days: transitioning feature all of its AR/VR applications. We have also to do so. Of course, we’re driven forward first
production to Universal Scene Description recently been collaborating similarly with and foremost by the needs of the talented
at DreamWorks,” which highlighted how it companies like Nvidia, which are undertak- artists and technical wizards here at Pixar,
is possible to quickly ramp up and adapt ing ambitious projects based on USD that without whom we wouldn’t have anything
an existing, mature pipeline to USD. Animal aim to propel its use in many new industries. worth writing about and for whom we are
Logic, the same year, also presented a talk eternally grateful.
titled “Forging a new animation pipeline with WHERE ARE WE HEADED?
USD.” There was an entire session at SIG- So far, USD has been evolving at a break- Speaking of being eternally grateful,
GRAPH 2018 dedicated to production use of neck pace. We’re a fairly small group of we’d like to take this opportunity to thank
USD, almost all outside of Pixar. developers, getting contributions from an our amazing team: Sunya Boonyatera, Alex
industry for whom we’re grateful. While we Mohr, Corey Revilla, Varun Talwar, Pol Jere-
Animal Logic had taken a different ap- keep adding more and more functionality mias-Vila, David Yu, Kees Rijnen, Matthias
proach very early on to USD interoperability to USD and Hydra, more plug-in points for Goerner, Raja Bala, Tom Cauchois, Florian
within Maya, motivated mainly by the lack site customizations, and more schemas to Zitzelsberger, Adam Fish, Adam Woodbury,
of animation support within the USD plug-in support more domains, occasionally we take Florian Sauer, John Loy, Brett Levin, Matt
provided by Pixar. For the first few years of a step back to look at the big picture and Kuruc, Stephen Gustafson, Jilliene Tongson
USD, integration with Maya meant choosing ask ourselves, where are we headed? Paras, and Jessica Tran. We’d also like to
between the Pixar plug-in or the Animal thank our Infrastructure team led by Cory
Logic plug-in, or using both. Fortunately, There has been a fair amount of pressure Omand, and our Applications team led by
Autodesk, collaborating with developers at for us to add full rigging support, and we’ve Hayley Iben, and all of Pixar’s engineers and
Pixar, Animal Logic, Luma, and many other often held the line that rigging is not within artists. We owe USD to the trust that our
important contributors to these projects USD’s ultimate goal of enabling interopera- VP, Guido Quaroni, put in us and to our CTO,
across several studios, is encapsulating the bility in pipelines. Yet, we added UsdSkel, a Steve May, and the rest of the Executive
two plug-ins under one umbrella with the capable domain of schemas for represent- Team at Pixar who helped create the
goal of combining the best of each into one ing skeletal animation and skinning, includ- wonderful environment that allowed USD
comprehensive plug-in. This project has ing blendshape support geared especially to flourish. Finally, to all of our contributors
picked up momentum in the late half of toward representing large crowds. (look for a list here http://graphics.pixar.
2019 and is well on its way in 2020. com/usd/docs/USD-Contributors.html), a
Along the same lines, USD is famously huge big thank you. 
The Foundry was the first to integrate lacking support for proceduralism, a term
a Hydra viewport directly into its flagship that is abused, misused, and confused. While F. Sebastian Grassia and George ElKoura are lead
lighting and rendering software, Katana, at we’re not ready to commit to any particular software engineers at Pixar Animation Studios.
first to leverage the fast preview renderer. solution quite yet, it’s clear that USD needs to
First-class support for USD is currently address this issue one way or another.
being worked on for Katana and other
Foundry products. How do we choose what to concentrate
on? What to work on next? The amazing
In 2019, Houdini launched a new, USD-
based workflow under the Solaris banner.
Houdini is the first of the major DCCs to ship
with a deep, fundamental integration of USD
into its system.

So many other VFX and game projects
are starting to integrate USD and Hydra in
various forms, from Unity, to Unreal Engine,
to Blender, that we think that within the next
year or two, it should be fairly easy for folks
to get their hands on software that knows
how to manipulate USD.

USD has also penetrated beyond film and
games. In 2018, Pixar collaborated with Ap-
ple to design USDZ, which is a single-asset
packaged form of USD, which can contain

28 cgw e d i t i o n 2, 2020

next issue

FALL ISSUE (Sept., Oct., Nov.)

¢ Animation: Pixar’s latest animated film shows it has Soul.
¢ Gaming: A look at the new gaming technology.
¢ Stylized Animation: Glen Keane’s Over the Moon takes audiences on a moving journey.
¢ VR: We pull back the curtain on some of the latest VR technology.

FOR ADVERTISING INFORMATION CONTACT

MARI KOHN, DIR. OF SALES | 818.291.1153 | [email protected]
LISA NEELY, MARKETING & ADVERTISING SALES EXEC. 818.660.5828 | [email protected]
WILLIAM R. RITTWAGE, PRESIDENT/CEO 818.291.1111 | [email protected]

News happens quickly in
our industry. Keep informed.

Keep up to date.
Anytime. Anywhere.

SIGN UP TODAY!
WWW.CGW.COM

A Bedtime Story

FILMMAKERS USE VR TO EXTEND THE WORLD OF THE ELEMENTAL SPIRITS
IN ‘MYTH: A FROZEN TALE’

BY KAREN MOLTENBREY

Before children close their eyes and 2, was the production designer. In fact, the world was inspired by pop-up books and
drift off to dreamland, parents will same artists, technologists, and teams that graphic silhouettes, even stage elements
regale them with bedtime stories work on the studio’s feature films work on all from vaudeville and music hall productions.
of fantastical characters from wondrous the VR/AR projects, too. That style meshed well with Lee’s aesthet-
worlds. Disney’s third VR film, “Myth: A ic, which Gipson describes as graphic and
Frozen Tale,” uses this ritual to kick off a fairy The impetus for the project began with stylized, almost Eyvind Earle-like, referring to
tale like no other, taking viewers on a visceral Jennifer Lee, Disney Animation’s chief the art director on the 1959 animated Sleep-
journey and immersing them in a stylized creative officer, who wrote and directed the ing Beauty. “She does a lot of cutout paper
world inspired by the environments and Frozen films, after approaching Gipson in art as well. That was one of the reasons I
elemental characters from Frozen 2. late 2018 about exploring the Frozen world gravitated to her artwork and her style,” says
for his next VR project. Gipson of Lee.
“Myth” begins in a forest setting outside
of Arendelle, as a local family sits down to “’Myth’ is a fairy tale, a bedtime story in a As Lee points out, the world of Frozen 2 is
hear a bedtime story. As the mother reads, way of a myth or legend about the ele- already stylized, and with “Myth,” they styl-
the audience is transported to a mystical mental spirits from Frozen 2,” says Gipson, ized it further. “We wanted to be respectful
and stylized world inspired by the elemental who was inspired by bedtime stories from of what was already there, and didn’t want
spirits, themes, and environments introduced his past. He tried to imagine what type of to take the design out of its original realm.
in Frozen 2 come to life, and the myths of the stories he would have been told as a child if It was challenging to walk a fine line that
characters’ past and future are revealed. he were to have grown up in Arendelle. Then elevated it and took it in a new direction
after watching those early screenings of Fro- but still felt like it would fit into the Frozen
The VR short was created in-house at zen 2, he became drawn to those elemental spectrum,” she adds.
Walt Disney Animation Studios by the characters, and pondered what they would
core team who built Disney’s first VR like look in VR. Inspiration for the VR short’s music-driv-
film, “Cycles.” “Myth” was directed by Jeff en narrative came from Disney’s Fantasia
Gipson, who also directed “Cycles” (2018). “I like the idea that if we grew up in Aren- and even Peter and the Wolf, with all the
Jose Luis Gomez Diaz, who also worked on delle, this was something that everyone was characters having their own piece of score. “I
“Cycles,” served as VR technology super- told, a shared story,” he adds. wanted music to play into that, where every
visor on “Myth,” with Michael Anderson as character has protagonistic characteristics,
VR environment lead and Ed Robbins as A Unique Style for a Unique Medium but also antagonistic characteristics as well.
VR character lead. Brittney Lee, a key visual The music can inform each one of those
development artist on Frozen and Frozen In Frozen 2, those elemental characters and then balance itself out as we introduce
assumed elaborate 3D forms. However,
Gipson’s vision for them in his VR storybook

30 cgw e d i t i o n 2, 2020

The monolith at day (top) and at night (bottom).

fire, air, earth, and water,” says Gipson. Like in Fantasia, each character has its Of course, foremost, the stylized charac-
“It’s almost like a visual poem in a way. We own score, color palette, and language. “I ters and world of “Myth” had to work within
harnessed 2D traditional animation from the wanted to bring that element of Disney virtual reality, which meant fleshing out the
artists here in the studio to be incorporated heritage into the medium of VR. Just as whole world and showing the viewer what
into this new medium.” there is a duality to each of the elemental is happening behind them, above them, and
spirits, there is a duality to the music, which over there in the distance, all while telling the
The stylized elemental characters in runs the gamut from sweet to antagonistic,” story. And that was no easy task consider-
“Myth” project a duality that intertwines well Gipson points out. The film’s original score, ing how rich the short film is with effects,
with the music. The Nokk, a mythical water which drives the storytelling experience, which are essential to the narrative and the
spirit that takes the form of a horse that was composed by Joseph Trapanese (Tron: experience, Anderson notes. “It’s a give-and-
harnesses the power of the ocean; the Nokk Legacy, Raid series). take. All that has a cost, and it’s a really big
fiercely guards the secrets of the forest and challenge to balance all of the ideas and the
those wanting to pass must earn its respect. “Myth” producer Nicholas Russell, who cool work coming in, with the performance
Gale, the playful wind spirit whose presence had worked with Gipson on “Cycles,” that we had to hit to make this viable,” he
can take the form of a soft breeze or a raging acknowledges the particular inspiration says. “It was a constant battle.”
tornado. The massive Earth Giants, spirits of Fantasia had on the short. “We wanted to
the earth that take the form of rocky river- combine that type of filmmaking with our The group was intent on obtaining the
banks while sleeping, but when awake and on newest technological tools. We kept imag- same or close to the same quality as the
the move, are capable of massive destruc- ining what it would be like if you could stand feature film, and still maintain the necessary
tion. And, the fire spirit, a tiny fast-moving fire next to Mickey Mouse as the Sorcerer’s frame rate. “Real time is running at way fast-
salamander named Bruni, whose emotions Apprentice when he’s conducting the stars er speed than what we render in the studio,
affect the intensity of its power. or battling the brooms,” says Russell. so there are limitations,” Robbins empha-
sizes. As a result, the team had to be very
“I began thinking of these elemental Myth-ical Characters & Myth-ical Lands clever on how it preserved that quality.
characters as members of a band, and when
they’re all playing together, they help to keep For planning the scenes, the artists used “For the most part, we preserved as much
the world in harmony,” says Gipson. “When Oculus’ Quill, a VR painting tool, to sketch of the pipeline that made sense, and then
even one of these spirits gets out of sync, out scenes for determining scale, the prox- whenever it didn’t make sense, we filled that
the harmony is broken.” imity of characters, and so forth. void with what we needed,” Anderson says.

3 1e d i t i o n 2, 2020 c gw

The fire spirit Bruni, a quick-moving salamander, heats up the action. with Unity instead of Unreal Engine) was a
learning experience, being the first project
For the characters, the team reused as as opposed to a fluid simulation, which was of its kind for the Disney Animation group.
much of the feature film’s production pipe- used in the feature and is far more complex “When we opted to go with a different
line as possible, beginning with the same and elaborate, notes Robbins. engine for this film, that meant starting
assets and animation and simulation rigs, from scratch in a lot of ways,” says Ander-
up until the point of animation – at which From ‘Cycles’ to ‘Myth’ son. “The pipeline changed because the
time they branched off into a customized platform changed. We were able to do a lot
setup. Also, they used Swoop, the in-house As Gipson explains, there are many ways more and iterate faster because of the tool
tool developed to give artists the ability to to make a VR film, though he prefers the set we were using [for ‘Myth’].”
animate the path and timing of the wind narrative, linear approach of “Myth” and his
spirit Gale in Frozen 2, either directly within previous “Cycles,” as opposed to a highly in- There were other differences in the
Autodesk’s Maya or by using VR. Due to its teractive experience where the user is con- processes, as well. For the characters in
VR capabilities, Swoop enabled the “Myth” stantly grabbing at items with controllers. “Cycles,” the artists used Alembic, a cached
team to puppeteer objects and characters, data format. For “Myth,” they used cached
creating the shape of the motion paths and There is a versatility to the way you can FBX rigs. Also, “Cycles” has a more tradi-
the timing for the characters to travel over tell stories in VR, and the way that audienc- tional look, as artists used a toon shader to
that path. es can interact with the world, Lee points flatten out the characters. As for the envi-
out. “It’s very different from what we do in ronment, it is a single setting that remains
For the environments, all the models feature films, and I think there’s something fairly consistent. Conversely, in “Myth,” the
likewise had to be reprocessed so they very fun and exciting about it. You can environments are diverse, and artists visually
could be handled within Epic Games’ really get into that space, and viewers can leaned into the art direction and shape
Unreal Engine, the VR platform used by the experience the elements a bit more closely language familiar to Frozen. Colors change to
filmmakers for “Myth.” and on a personal level,” she says. “I feel like suit the story, with no cuts.
that may be the most enticing aspect of it.
“We take lower-res models and have to There’s a real ‘wow factor’ when you get to “It has to be driven in one continuous
create UVs, then we have to create fully have facetime with the Nokk and he comes piece,” says Anderson. “It’s a technical chal-
unique textures. None of that really transfers right up to you and assesses whether or not lenge to have the sky change and the colors
from our regular pipeline. It’s a completely you’re worthy. You can almost feel his breath change, and cut from the cabin into the
different system,” says Anderson. “So we on you as he approaches. And the scale of forest in a creative way so it’s not just a fade
make new textures for our models using the Earth Giants is so immersive to the point to black. All of these things are simple and el-
[Adobe’s] Substance Designer and various where you feel that they can practically pick egant when you watch them, but underlying,
other tools.” you up. Experiencing the Northern Lights the technology is very complicated, and it’s
as if you’re standing right in front of them is complicated to get it all working in harmony.”
Once the models were brought into the also truly magical.”
Unreal Engine, shaders were created and “It’s amazing to see [Gipson’s] range in
set up so they could be rendered in the In “Cycles,” the filmmakers tried to guide going from a very personal and emotion-
engine. While typically animation is exported the eye of the viewer using color and light, al film that was seen in the confines of a
through Disney’s proprietary format, in this as well as motion, to push the viewer around house like ‘Cycles,’ to this latest project,
case it was exported as cached FBX files the space. They also employed a technique which encompasses an entire forest and
and then brought into Unreal Engine, where whereby the space darkens and desaturates the whole of nature,” says Russell. “This film
the layout was done inside the Unreal En- somewhat if the person is not looking in the is a game-changer in the way that the VR
gine Sequencer Editor tool. intended area. When it came to “Myth,” they technology is being used to tell a story and
again used color, light, and motion to the make the viewer a part of that world.”
It’s important to note that the stylized same effect to direct the viewer’s attention.
characters and world of “Myth” provided the Also, the interactivity is gaze-based. Users Closing the Virtual Chapter
artists with more freedom in making creative have six degrees of freedom, whereby they
choices when it came to their assets. In the can move and view the film as they like. But “Myth,” which is approximately eight min-
VR short, the Nokk, an extremely complex there are also moments where the char- utes in length, took close to eight months to
character, is featured prominently. And while acters will react based on the user’s viewer complete following creative approval, with
the team started off by using the Nokk’s body direction or proximity to the characters. production wrapping last November. All told,
asset from the feature film, they decided to about 80 people worked on the short film,
make Nokk’s mane a cloth simulation sheet Indeed, “Cycles” (which was created though a core team of eight or so worked on
the film for its entirety.

“Myth: A Frozen Tale” is now available on
Facebook’s Oculus Quest.

…The end. 

Karen Moltenbrey is the chief editor of CGW.

32 cgw e d i t i o n 2, 2020

OVERALL CG MARKET TO GROW TO $154B BY 2023

BY JON PEDDIE

The computer graphics market has always for a while. A growing point of exploration has taken a leap, and it shows in increased
been an exciting place since it began in the for VR today is in content creation where it revenues for all companies. And, although
early 1960s. Challenged by hardware speed is used for virtual production, a glamorous unsung, CAD is often involved in the heavy
and costs, visionaries saw the potential, and but tiny market. The real market for VR is in lifting of all those 3D models of cities and
driven by artistic and scientific quests, pushed the design fields for professionals.  spaceships. The beautiful ray-traced images
the envelope seeking imagery and moving pic- of automobiles, gorgeous skyscrapers, and
tures that were indistinguishable from reality, The computer graphics hardware market clothing, bottles, and other consumer goods
as well as fantastic and imaginary. was worth $130 billion in 2018 and is expect- – it’s all CG.
ed to increase to $137 billion by 2023, with
Computer graphics is not a single dis- software growing slightly and a little less The demand for programmers, artists,
cipline, but rather a collection of capabili- than hardware. scientists, and designers continues to be
ties and people: processors, engineers, lots strong, and we’re seeing start-ups arrive in
of memory, artists, displays and XR, tech- The hardware segment of the CG industry emerging and reborn markets such as aug-
nical directors, programmers, and zillions of has seen steady overall growth, with some mented reality, virtual reality, and casual
lines of code. ups and downs due to product cycles and games. Artificial intelligence is becoming
the irregularity of the PC market. increasingly important in image process-
So, the universe of CG has continuously ing, ray tracing, and AR. The arrival of new
grown, exponentially at times. Sometimes The year 2019 witnessed the intro- APIs and platforms are also stimulating
it has followed Moore’s law, sometimes it’s duction of new GPUs and add-in boards development. Firms are actively looking
been boosted by software breakthroughs (AIBs) with ray-tracing capabilities; several for people who can use and exploit these
and eureka moments. Growth in the dollar new CPUs and next-gen game consoles new programs and their associated hard-
value of the market has leveled off a bit, but also were announced for the second half ware accelerators. 
that’s due as much to falling prices as it is of 2020.
to the asymptotic nature of any market. As Given the trends in dropping costs, and
a matter of fact, CG has seen growth for The market for CG software in 2018 was the increasing users and usage of CG tools
over 60 years – not many markets can make worth $14 billion (not counting services, and hardware, we predict that the rate of
such a claim. maintenance, and other aspects) and is growth for the CG industry will remain fairly
expected to grow to $17 billion by 2023 as steady for the foreseeable future.
Growth has slowed to an average (hard- the industry shifts its sales model, moving
ware and software) of 1.2% CAGR, but at more services to the cloud and transitioning Computer graphics is truly a worldwide
$154 billion a year, that’s not trivial. to a subscriber system. industry now.  

The software aspects of the market have Although it doesn’t get discussed much, Dr. Jon Peddie ([email protected]) is a
been slow to flat, but hardware segments the digital graphics market is dominated by recognized author and pioneer in the graphics
have been varied, influenced by consumer CAD/CAM. Digital video is a bigger market industry; president of Jon Peddie Research, a
products such as game consoles and PC than the modeling or rendering segments, Tiburon-CA-based consultancy specializing in
gaming machines. and yet when one reads about CG, it’s all graphics and multimedia that also publishes
about the special effects in games and JPR’s “TechWatch”; and named one of the most
Virtual reality, which has gained so much movies. That’s easy to understand. Everyone influential analysts in the world.
attention, is hardly a factor in either hard- loves VFX and has experienced it. In reality,
ware or software, and despite some glowing truly amazing things are happening in CAD
forecasts, we don’t think it will be a factor as the digital and real get connected. CAD

COMPUTER GRAPHICS HARDWARE MARKET

Monitors used for graphics 2018 2019 2020 2021 2022 2023
Workstations 1.00 1.00 1.08 1.18 1.27 1.32
Game consoles 10.48 12.37 13.08 14.50 15.90 17.16
14.81 12.06 12.47 17.29 17.25 16.45
Total gaming PCs 35.48 34.57 35.91 35.64 37.00 36.95
Mobile devices 68.63 75.35 70.73 65.70 66.70 65.37
130.41 135.35 133.27 134.31 138.12 137.25
TOTAL hardware ($B)

3 3e d i t i o n 2, 2020 c gw

Medical animators Alissa Eckert and Dan Higgins of the CDC created this image of the COVID-19 virus.

Left Brain, Right Brain

ANIMATORS USE COMPUTER GRAPHICS TO HELP EXPLAIN COMPLEX MEDICAL CONCEPTS

BY KAREN MOLTENBREY

In the early part of the year, COVID-19 profession requiring a very specific type of and it was their work that led to the image of
struck, and struck hard and fast. In skill. Here we look at three medical illustra- COVID-19 that is seen everywhere, from the
a matter of a few months, this novel tors and how they are using the same type covers of magazines, to charts and graphs
coronavirus infected close to 4.7 million of 3D tools and techniques employed by used by medical experts, to the nightly news
people in the US alone, killing 160,000 of filmmakers and others to ply their craft. broadcasts.
them as of early August, and those number
continue to rise with each passing day. Alissa Eckert and Eckert initially planned to become a veter-
Dan Higgins, CDC inarian, but has always taken art classes for
Indeed, this microscopic monster has fun. She started out studying biology for a
impacted the lives of people around the Alissa Eckert and Dan Higgins are both few years and then switched to scientific
globe in ways we could never have imagined. medical illustrators at the CDC in Atlanta, illustration, an interdisciplinary study that
And yet, it’s safe to say that most of us have combines science and art, after discovering
never seen this virus, although all of us can that she could combine both those inter-
instantly recognize it on sight, thanks to the ests into one profession through medical
work of medical illustrators and animators. illustration. She looked into it as a career
These specialized artists use illustrations and determined it was for her. She eventu-
and animations to help the medical com- ally earned a master of science degree in
munity as well as scholars and the public ac- medical illustration, having had to com-
curately communicate complex information, plete classes alongside medical students,
and in the case of COVID-19, put a “face” to in addition to art classes that focused on
this invisible infectious agent. illustrating scientific information. Right out
of school, she began working at the Centers
The work requires significant education for Disease Control (CDC), where she has
and training, both in the medical/science been employed for 14 years now.
fields and in the arts – a left brain/right brain

34 cgw e d i t i o n 2, 2020

For her colleague, Dan Higgins, his interest At the CDC, medical animators use a range of DCC software for their work.
in art guided him into this career. He was
working as a graphic designer until an have been utilizing 3D printing in their work, to the CDC’s website to learn more informa-
interest in science led him down the road to output on a 3D Systems’ ProJet 460Plus or tion,” explains Eckert.
a master’s degree in medical illustration. He a FormLabs printer.
began his new career as a medical illustrator In addition, the two medical illustrators
with WebMD before he migrated to the CDC, Eckert recalls making a 3D printed model were putting a face to this virus, taking
where he has been employed for the past a year ago that was used to train physicians something that is unfamiliar and invisible,
18 years. in countries where a specific birth defect and bringing it to life so people would realize
was prevalent, to teach them how to take that it is real and that it does exist. “We
“We’re kind of like translators for the accurate measurements. “It’s a life-size wanted people to take this seriously, so we
scientists when they’re communicating with model, so they can practice measuring tech- created it in a certain way,” adds Higgins. To
the public,” says Eckert. “We take complex niques, which involves a very specific pro- this end, the artists created a gray sphere
scientific information and turn it into visuals cess,” she explains. Other 3D printed models and then added bold red S-proteins –
that people can easily understand.” This of viruses and so forth have been used for spiked proteins (which attach to a cell)
could pertain to a wide range of topics at informational and educational purposes, that are a signature of the coronavirus and
the CDC – from viruses and bacteria, to sometimes for a conference or lecture. enable it to be identified and recognized.
tuberculosis, AIDS, cancer, chronic health
conditions, birth defects, and so much more. “The tactile experience [from a 3D-print- “You would see a similar shape when
ed model] really helps people learn about you look at it on the electron micrograph,
While the work can be geared toward the structures,” Eckert adds. though they [the virus] are always kind of
the general public, the two artists also fuzzy and hard to read. We’re just making
do graphics, illustrations, and animations When the work involves an animation, the it more clear, because they are really, really
that are intended for the scientific com- artists will consult with scientists through- tiny,” Eckert says.
munity, including physicians and scientists out the project, making sure the science is
on staff at the CDC for lectures, journal accurately depicted. The main goal in this case was commu-
articles, and presentations, where they nication, and as such, the artists had to
are conveying the information to fellow Some projects take quite some time to balance the colors so that the image was
scientists and researchers. complete. But, when COVID-19 struck so bold, vibrant, attractive, and noticeable
quickly, Eckert and Higgins had just one without being too playful, so it would be
Just as the subject matter is diverse, week to come up with an image for the virus. taken seriously without scaring people. “We
so, too, are the image formats used: 3D, “We had to drop everything we were work- wanted people to look at the image and
3D with animation, 2D and 2D animations, ing on to focus on this one project,” says become interested in it,” says Eckert, noting
illustrations, 3D printing, and more. Eckert Higgins, noting that in a typical scenario, the she and Higgins created a number of vari-
and Higgins use Autodesk’s 3ds Max as their work would have taken at least three weeks ations of the image before settling on this
primary 3D software, along with Pixologic’s of back and forth with the scientists. “We very popular graphic.
ZBrush. They also utilize Adobe’s Creative were granted access to the scientists much
Suite as well as a number of plug-ins, faster, and they were able to guide us on “We were intending it to be an education-
including those from Red Giant (which now which proteins we needed to include and al opportunity, and it just blew up. I never
merged with Maxon) and Video Copilot. the quantity of the protein.” thought it was going to do that,” says Eckert
of the image’s widespread popularity.
The artists additionally employ special- In this instance, it was important to grab
ized software called Chimera, an extensible the viewer’s attention. “This was a public Higgins agrees, adding, “We’ve seen our
modeling system for the interactive visual- health emergency, and we needed them to work out there before, but this just exploded.
ization and analysis of molecular structures pay attention to this topic and hopefully go I even saw it on SNL, which was pretty crazy.”
and related data, including density maps,
trajectories, and sequence alignments,
from which high-quality images and movies
can be made. Chimera is developed by the
Resource for Biocomputing, Visualization,
and Informatics (RBVI) at the University of
California, San Francisco. 

According to Eckert, she uses 3D
as much as possible, which eases the
workflow. “I find the process much faster,
especially when small changes have to be
made,” she notes, whereas in the past when
they were producing more hand-drawn il-
lustrations, changes were time-consuming.
In addition 3D, these medical illustrators

3 5e d i t i o n 2, 2020 c gw

Veronica Falconieri Hays, the COVID-19 virus, in 3D). Medical illustrator Veronica
Falconieri Visuals Depending on the audience, the imag- Falconieri Hays created this image
of the SARS-CoV-2 virion, the
Veronica Falconieri Hays is a certified med- ery can be scientifically precise, or such COVID-19 virus, in 3D.
ical illustrator specializing in medical, mo- that Falconieri Hays can play with scale or
lecular, cellular, and biological visualization, use metaphor. It just has to be clear and pieces that project out from the surface of
encompassing still media and animation, at understandable to the audience for which it SARS-CoV-2. This work required a great deal
her company, Falconieri Visuals, based in the is intended. of research. In fact, out of the approximately
Washington, DC, area. 27 hours it took for Falconieri Hays to do
“I focus on molecular and cellular visu- the project, at least half of that was spent
Falconieri Hays received her undergrad- alization, using 3D,” Falconieri Hays says. In conducting research, “just making sure I
uate degree in biology from Smith College, particular, she employs Maxon’s Cinema understood the basis of the virus structure
with a minor in art, before earning a master 4D – which she learned in grad school – and the unique details about the spike
of art degree in medical illustration, “which because “it has a plug-in, ePMV, that allows structure, because it’s a target for vaccine
covered a lot of the basics of how to create me to directly pull molecular data. I also use development,” she explains. “It’s an area
accurate illustrations and the general [Pixologic’s] ZBrush for organic modeling.” that an antibody, for instance, could bind
techniques that are used in a lot of the to and block the virus from interacting with
medical and scientific visualizations today,” Once the modeling and animation is human cells.”
she says. completed, Falconieri Hays uses various
Adobe tools “to bring it together.” Render- According to Falconieri Hays, animations
After graduating in 2014, Falconieri Hays ing is done either with the Cinema 4D’s are becoming a larger part of the work she is
worked at the National Cancer Institute as standard renderer or with the Redshift asked to do, often to show how a therapeu-
an illustrator and animator before launching renderer, depending on the look she is tic works.
her own company in 2017. going for. For compositing, she uses Adobe
After Effects. While Falconieri Hays may not be creating
In order to become certified in this visuals for the next blockbuster, the work
field, Falconieri Hays has to meet con- When creating her work, Falconieri Hays she does is extremely important. “I do a lot
tinuing education requirements and pass will spend a great deal of time collaborating of illustrations that are scientifically com-
an exam by illustrating art, medical, and with scientists and researchers, who will municating to other scientists, but I also do
business knowledge. identify the scientific concept and story they projects that are more public-based,” she
want to convey, as well as their audience for says. “It’s really important having amazing
“Everything I do translates scientific the project. They will also identify the format visuals because the subject matter is so
information, research, understanding, and they need, whether it’s video or still images. complicated. You’re asking a lot of your
concepts into visuals. This means doing Then she begins sketching with pencil and viewer to step in and really take the time to
a lot of research and having a fair amount paper. Once the client agrees that all the understand the science you’re communicat-
of literacy in the underlying concepts, and important points are covered and correct, ing. If you attach an enticing visual to it, you
then adapting that visually to the appro- she begins the creation process, whether it’s get a little more buy-in. You can communi-
priate audience, whether to experts or the in 3D or 2D. cate more science to the viewer if you have
public,” she explains. “It’s about making that a visual that hooks them and piques their
connection between complex ideas and As the recent pandemic was developing, interest.” 
translating it visually to the right audience.” Falconieri Hays, along with colleagues at the
Association of Medical Illustrators, created a Karen Moltenbrey is the chief editor of CGW.
Naturally, Falconieri Hays was always in- shared reference document, which included
terested in both science and art, and heard an illustration summarizing the 3D structural
about this field while she was in high school. features of the SARS-CoV-2 Spike (S) pro-
“When I was pondering my future career tein structure (SARS-CoV-2 is the virus that
and college choices, I initially thought that I causes COVID-19). “Research was coming
either had to choose science or art. Fortu- out so fast; it’s a lot to keep track of. So, we
nately, I heard about medical illustration and decided to collaborate and help one another
realized it would enable me to incorporate by crowd-sourcing a lot of our reference to
both into a career,” she notes. help medical illustrators as a whole,” she
explains. “And because I have a particular
Falconieri Hays’ client list comprises expertise in the 3D molecular structure, I
communications agencies that work with wanted to create something to give back to
biotech companies which have complex my colleagues if they were asked to visualize
science they need to explain to investors, for the spike protein, for example. This way,
instance. Others are researchers and scien- they’d have a better understanding of how it
tists themselves who need help with visuals works because it’s pretty complicated.”
to explain the work on a research project.
She also produces imagery for publications, For this particular illustration, Falconieri
such as Scientific American (a recent issue Hays looked closely at the 3D structure of
cover image and main article image contains the spike proteins, which are the hook-like
an illustration the artist did of SARS-CoV-2,

36 cgw e d i t i o n 2, 2020



Working Together

FOR M&E WORKSTATION PERFORMANCE, IT’S DIFFERENT STROKES FOR DIFFERENT FOLKS

BY BOB CRAMBLITT

W hen one talks about workstation process might appear simpler than it is in great detail. A landing gear for a spaceship in
performance related to product CAD, but looks can be deceiving, according a video game, for example, just needs to dis-
development, CAD/CAM/CAE to Alex Shows, chair of the SPEC Worksta- appear into the fuselage, where on an actual
usually comes to mind. It’s fairly easy to link tion Performance Characterization (SPEC­ CAD model of a space shuttle it needs to
design time-savings to greater productivity wpc) subcommittee. “One of the differenc- properly fold up and be stowed.”
within the product development cycle. Yet, es in modeling demands between CAD and
professionals producing graphics for media M&E is the diversity of geometry versus ren- Both CAD and M&E modeling are single-­
and entertainment (M&E) applications such dering complexity,” says Shows. “CAD spans threaded operations that improve with CPU
as gaming, special effects, and 3D animation a wider gamut of geometric complexity, frequency. But as Shows points out, M&E
face very similar challenges when seeking to while M&E applications incorporate some of material complexities place a much greater
optimize their workflows. the most sophisticated interactions among strain on the GPU and VRAM to deliver the
an incredibly diverse palette of materials.” frames per second needed for refresh-
Just as one type of workstation config- ing graphics on the screen when making
uration does not fill the needs for all CAD M&E models also have fewer demands changes to the model and interactively
applications and workflows, neither does it placed on them within the design pipeline, rotating, panning, and zooming around the
for media and entertainment. Performance according to Ross Cunniff, chair of the SPEC scene.
is based on a complex web of interactions Graphics Performance Characterization
associated with the application itself, the (SPECgpc) subcommittee. “CAD modeling Modeling functionality is not monolithic
system configuration, the type of models is typically driven by requirements of the across different M&E packages, of course.
being constructed, rendering complexities, back-end CAM pipeline, leading to design Beyond the differences in end-product
the myriad paths data and storage take, and decisions within the application that can requirements – size of the models, complex-
the destination for the end product. sometimes reduce graphics performance,” ity of textures and effects, resolution of the
says Cunniff. “In M&E applications, the final images, and length of time displayed on
What’s good for Autodesk 3ds Max per- image produced is the product. the screen – there are inherent differences
formance might not be as good for Maya. in how the applications themselves handle
Likewise, what works for gaming might not Unlike CAD, most models for M&E need modeling. There are also the continuous
work as well for special effects, 3D anima- to be only marginally functional. changes within versions of the same appli-
tion, or architectural simulations. cations as new technologies are implement-
“M&E models typically just have to look ed to speed operations and increase realism.
NO TWO MODELERS ARE THE SAME good or be reasonably functional,” says
Modeling in M&E applications is marked by Trey Morton, chair of the SPEC Application “Different implementations between
a series of interactions as the artist makes Performance Characterization (SPECapc) different versions of an application can make
changes to the model and those changes subcommittee. “The time frame for M&E a large difference in the hardware required to
are visualized in the viewport. The modeling objects on the screen can be extremely maximize performance,” says Cunniff. “Older
short and the objects often do not require versions of [Autodesk’s] Maya, for example,
required significant CPU overhead to transfer
Demanding apps such as Blender benefit from massive multi-threading. work to the GPU. Newer versions can store
work directly on the GPU, removing the bot-
tleneck and making higher-end GPUs much
more useful for high-performance modeling.”

SAVING RENDERING TIME
TO REINVEST
For both interactive and non-interactive
rendering in M&E applications, the focus
shifts from single threads to multi-threading,
where more cores working in parallel can
significantly accelerate performance. This is
especially the case when dealing with very

38 cgw e d i t i o n 2, 2020

Heavy use of textures, like in this model, puts a strain on GPUs and VRAM. workloads, the M&E applications they
use, and the functionality they rely on the
large, realistic models produced in high-res- invest in a high-core-count CPU and a high- most. But, this is difficult, and the terrain is
olution for television and movie production, end GPU with enough memory to cache full constantly shifting, as new versions of appli-
and architectural models incorporating real-­ animation sequences,” says Cunniff. cations introduce new ways of maximizing
world shadows, reflections, and textures. efficiency.
WHAT’S THE FINAL PRODUCT?
“The critical components for increasing As with CAD, the end result of the M&E The next best thing to homegrown perfor-
rendering performance in an M&E applica- production cycle has an impact on what mance measurement is to use standardized
tion are high core count processors, whether configurations will provide the best ROI benchmarks that most closely align with
they are general-purpose CPUs or massively for hardware investments. Film and TV your application and provide the granularity
wide vector processing engines like the sequences are rendered at 4K or sometimes to measure the myriad paths that an appli-
compute cores found in a GPU,” says Shows. 8K resolution per frame at 24 or 60 frames cation takes. In that vein, there are SPECapc
“Which of these is more important depends per second. The large number of high-reso- benchmarks based on 3ds Max and Maya
on your application and workflow.” lution, uncompressed frames places a pre- that run on top of those applications. If you
mium on network or local storage to address want a smaller, self-contained benchmark,
Interactive rendering, when an artist makes the high data rate demands. the current version of SPECworkstation
a change to a scene and wants to quickly see includes workloads for 3ds Max, Maya,
the newly applied textures and lighting in the Game development has dual require- Blender, and open-source applications such
viewport, is heavily reliant on the massive ments: creating the game in applications as the Handbrake media encoder and the
computing cores found in the GPU. such as Maya, 3ds Max, or Blender, then LuxCore physically-based renderer.
integrating the assets into game engines
The speed of both interactive and non-in- such as Unreal or Unity. The emphasis in If you are interested primarily in graphics
teractive rendering strike at the core of pro- this process is optimizing animation and performance, the SPECviewperf benchmark
ductivity: A fast GPU with enough memory effects such as lighting and shadows to (available in Windows and Linux versions)
will calculate changes on the fly, eliminating create a seamless gaming experience. A provides workloads (called viewsets) based
waiting time for the artist. A powerful GPU modern GPU with multiple cores and a large on 3ds Max, Maya, and Showcase. All of
can also cut down overall render times for a VRAM buffer gives game developers the these benchmarks are available to users for
scene and save a studio hours of time that speed they need to create scenes and make free download. Vendors of computer-related
can be reinvested into the artistic process or changes interactively. A fast CPU then allows products and services that are not mem-
later production phases. quick transfer of files into the game engine. bers of SPEC/GWPG are required to pay a
licensing fee.
PARALLEL PROCESSING Other functionality within M&E appli-
SPEEDS ANIMATION cations can stress computing resources THE EVOLUTION NEVER QUITS
The animation process – creating or as well. Transcoding, the act of converting Users don’t need the biggest or fastest of
changing motion for characters and objects, a media file or object from one format to every component, but will benefit from
then seeing the scene played back in real another within an application, puts a tax on upgrades that address their most problem-
time – is heavily reliant on multi-core CPUs a workstation and can benefit from GPU atic bottlenecks. Benchmarks that deliver
and GPUs. Maya is particularly optimized for acceleration, according to Cunniff. alignment with an artist’s day-to-day work
animation operations, incorporating parallel can help organizations make the decisions
processing and playback caching to stream- MEASURE TO GET BETTER that will guarantee the best price/perfor-
line performance. Beyond ascertaining how different M&E mance ratio.
applications interact with workstation com-
Parallel processing assigns a computa- ponents, professionals concerned about Whatever way you decide to benchmark,
tional task to the best available core, while performance should be aware of their own be aware that it is a commitment no different
cached playback saves calculations to usage patterns. from dedicating yourself to continuous edu-
system or GPU memory so the application cation about technologies that can contrib-
doesn’t need to load the changes to every Ideally, workstation users would devel- ute to the quality and speed of your work.
frame, enabling smooth playback. “Animators op their own benchmarks specific to their
looking for the best performance need to “Applications are continually evolving,”
says Cunniff. “A GPU that might be big
enough for yesterday’s application might
well be a bottleneck once the next version of
the application is released.” 

Bob Cramblitt is communications director for
SPEC. He writes frequently about performance
issues and digital design, engineering, and manu-
facturing technologies.

3 9e d i t i o n 2, 2020 c gw

Transcending
Boundaries

It used to be that cutting-edge visual effects were the domain of feature films. But now,
similar work can be found on the small screen as streaming services (Hulu, Amazon, Netflix,
Disney+, and more) and premium channels (HBO, Showtime, Starz, Epix, and others) are
growing in popularity and offering extraordinary content boosted by CGI. Indeed, VFX in
entertainment appears to have no bounds. Here, we look at the complex digital work playing
out on television, computer, and mobile-device screens. This includes the VFX used to create
the enchanted world of the faeries in Amazon’s Carnival Row and the real-time design and vi-
sualization technology that has brought the amazing world of HBO’s His Dark Materials to life.
With film and television production stalled due to COVID, audiences are rediscovering these
series. To this end, CGW decided to backstep, too, and highlight this compelling work.
– Karen Moltenbrey

40 cgw e d i t i o n 2, 2020

Dark Days

CREATING THE THRILLING VFX OF CARNIVAL ROW’S
FLASHBACK EPISODE

BY KAREN MOLTENBREY

C arnival Row is an immigrant’s tale, Artists added digital faeries (below) to shots of actors on rigs (above).
centered on various races who were
forced to flee their homelands when Prime Video. Created by Travis Beacham and Important Looking Pirates, or ILPvfx
war and invaders arrived on their doorsteps and Rene Echevarria, the series is based on (unique environments and more in Episode
to plunder their resources. In this story, these Beacham’s film script “A Killing on Carnival 3), plus others.
venerable beings are collectively known as Row.” Betsy Paterson, who served as the
the Fae, and they come from ancient, myth- visual effects supervisor on Season 1, says Season 1, which dropped late last summer,
ical lands, and have settled in Carnival Row, the work needed to feel grounded and gritty, features eight episodes. No release date has
the seedy immigrant ghetto section of the part of a world that isn’t ours but still feels been announced yet for Season 2 due to
sprawling human city of the Burgue. real and physical. interruptions resulting from COVID-19.

There are two primary types of Fae who Without question, it’s a big show with lots Most of the series thus far has taken
have settled here: the Pix, full-size pixies/ of VFX, which mainly involve creating char- place in Burgue – and especially at Carnival
faeries with wings; and the Pucks, satyr-like acters and sets, built by a number of studios, Row. A great deal of set extension work is
beings with horns and hooves. There are such as Image Engine (creature animation used for shots from Carnival Row, as the
glimpses of others, too: fauns, centaurs, and including the central monster), Pixomondo main drag consists of approximately three
the gremlin-like sprites known as Kobolds. (faerie wings), Rhythm & Hues (Kobolds), blocks of constructed set that extends
The Burgue is a class-based society that vertically for only about a level and a half.
resembles a dreary, grungy, steampunk-in-
spired Victorian England, and the Fae are
considered the lowest class of all here, as
the creatures – many of them working as
indentured servants – are stripped of their
dignity and their rights.

To make matters worse, a serial killer is
terrorizing Carnival Row, threatening the del-
icate balance of life in the city, where peace
among the Fae and humans is tenuous at
best. Keeping the peace in the area closest
to Carnival Row are constables who care
not a whit about those living here, with the
exception of human detective Rycroft Phi-
lostrate (Orlando Bloom), the only one who
is respectful of the Fae and tries to solve the
murders and keep the fragile peace. But,
Philostrate (Philo) has a secret of his own:
He lives as a human but was born a faerie.
Just as damning, he has rekindled his love
affair with Vignette Stonemoss, a refuge
faerie, within this intolerant world.

Such is the premise of Carnival Row, a
web television sci-fi/fantasy drama pro-
duced by Amazon Studios and Legendary
Television Studios, and airing on Amazon

4 1e d i t i o n 2, 2020 c gw

Film plate before digital faerie wings were added. there is mass confusion and Philo and
Vignette become separated. Told Vignette
would die for him, Philo agrees to have the
message relayed to her that he was killed in
battle so she would escape to safety.

“It was a great opportunity to build a
universe unto itself within this one episode,”
says Paterson. “We’re seeing the home world
of the faeries and the war that led to the situ-
ation we see in the city during the series.”

This particular episode was submitted for
this year’s Emmys, in the Outstanding Visual
Effects category, but did not make the final
cut. It is written by Travis Beacham and
directed by Anna Foerster.

Wings with gray lighting. History in the Making

Final comp of the CG wing added to the shot. While the typical Carnival Row episode
contains around 200 VFX shots, “King-
At either end of the street and off the side into a Fae refugee village in the kingdom of doms of the Moon” has approximately 450,
streets is greenscreen. The bridge spanning Anoun, in the Tirnanese Highlands, where encompassing partial- and all-CG charac-
the river, seen at one end of the street, was they have come in peace and plan to ters; large, expansive CG environments and
created in post, along with the train crossing establish a defense post. The Mimasery, a digitally-augmented sets; as well as a big
the bridge. And, uphill from this street lies holy place, is a preserve for the religious and airship battle. Image Engine and ILP were
the rest of the city, which is CG as well. educated, and now serves as a shelter for the two lead vendors on this flashback
the Fae escaping from the war. episode: Image Engine was tapped for the
However, Episode 3, “Kingdoms of the creature work, while ILP handled the big
Moon,” is a stand-alone flashback that pro- The soldiers are wary of the magical digital environments, the CG faeries and
vides history and context on Philo and Vig­ beings, but not Philo. He and Vignette, the their 3D wings when the faeries are practical,
nette’s relationship, and how the refugees steward of the sacred library there, become and the battle.
have come to be in their current situation. friendly and have a short-lived affair, during
The story line takes viewers out of Carnival which time he shares his secret with her – According to Paterson, the VFX team
Row and back 10 years in time, to when Philo that his wings were shorn off when he was a used practical prosthetics as much as possi-
and Vignette first met. The episode begins child so that he could live a better life. When ble throughout the season. This was true for
with Philo leading fellow Burgue soldiers an invasion by the evil Pact soldiers begins, the faeries’ wings as well as the horns and
hooves of the Pucks as they mingle together
in the crowded streets.

In Episode 3, the village is filled with
faeries, whose wings were a combination of
practical and visual effects by ILP, a midsize
VFX facility in Sweden. The wings are
practical when the characters are walking
around and the appendages are tucked
to their bodies. Sometimes the wings had
to be refined digitally, but as soon as the
wings begin to move, they are transitioned
to CG replacements. (Nick Dudman, special
effects designer, devised a system for easily
removing the practical wings in the series,
and then a green tracking cube was placed
on the back of the costume.)

According to Niklas Jacobson, visual
effects supervisor at ILP, the CG wings had
to look very specific, with an oily, semi-trans-
parent feel. And, they had to blend seam-
lessly with their practical counterparts. Even
the animation was delicately balanced so

42 cgw e d i t i o n 2, 2020

the CG versions would not stand apart, yet The film plate with a stand-in for the marrok. Note Vignette without her wings.
the animation had to make them feel as if
they are alive and part of the character. Artists added the CG marrok to the scene, as well as CG wings for Vignette.

“It’s an incredible amount of work – the themselves with a serum, which causes artists began the worldbuilding, carefully
roto alone that’s needed to remove the them to transition into the beasts. constructing and then incorporating the
practical wings and add the CG wings was same type of trees and the specific type of
unbelievable,” Paterson notes. “We had to Epic Worldbuilding rocks from the live-action plates into the
roto around clothing and other characters, digital environments. SpeedTree helped
and there are lots of crowd scenes in the The landscape in this episode is stunning, grow the trees. “The stones and mountains
episode – we had to get those wings in there with breathtaking hills, towering mountains, are very specific to this area, so we had to do
between every faerie.” and deep ravines. The episode was filmed custom texturing work to create the specific
in Prague and the nearby towns, where stones and so forth,” notes Jacobson.
For the mid- to near-ground faeries filmmakers captured the natural beauty of
who fly, the actresses were filmed on wires the environment. For instance, the open- Into this landscape, the ILP artists built
against greenscreen. For one scene in par- ing sequence with Philo and the soldiers the village, including a massive cathedral-like
ticular, however, a giant rig was constructed marching to the village was shot practically library holding sacred texts, maps, and
over the main courtyard set that enabled with CG extension work. The snow was real, technology research. The bottom two levels
16 to 20-some faeries to fly at a given thanks to a well-timed blizzard, which negat- of the library were built practically and then
time, making the transition to flight more ed the need for digital effects. For the most digitally extended to a soaring height, allow-
seamless. “We weren’t separating them out part, though, the plate photography in the ing Vignette to fly upward while locating a
on greenscreen [for this sequence], even episode was used for extreme close-ups. particular book. “You don’t know how high it
though it led to a lot of rotoscoping. But, it actually is. You never quite see the top,” says
felt more realistic to have them walk through The wide establishing shots were a Paterson. A spinning rig lifted up the actress
the set and then lift up and fly out of there,” result of worldbuilding by ILP artists, who from the set piece, and the effects were
Paterson adds. constructed the area near the walled-off added later.
Mimasery village including the forests and
When the faeries appear midground to mountains around it. “One of the more excit- While ILP had been using SideFX’s
background, full-CG characters are used. ing challenges for us was creating a believable Houdini for many years to create effects,
world and planning it out. We received some this episode marks the first time the studio
Pixomondo was the main “wingman” for fantastic artwork from the art department, migrated fully to the software as its primary
the majority of Season 1, and did quite a few but we also did some concept work our- package for worldbuilding and environment
shots in this episode, too. selves,” says Jacobson. The artists generated work, enabling the team to take advan-
almost all of the environments with geometry tage of all the tools that exist within that
Technically more complex than the due to the large number of shots taking place program.
faeries in the episode were the marroks within the environment; there were only a
(werewolves), changelings that transition few matte paintings, used in the establishing “We have been using Maya for many years,
from human to beast form. shots as the soldiers arrive. and we love it and still use it primarily for
modeling, animation, and so forth. But, it was
“We scanned all the actors who had their Once ILP received plate photography, the hard to assemble scenery, and it became
own marrok, and we worked hard to ensure
that the design of each was closely related
to the actor playing that particular charac-
ter through the use of skin tones and hair
placements,” says Paterson. Then, it was a
matter of building out the creature. Paterson
praises Image Engine’s incredible work with
the muscle systems and the skin, as these
creatures have much more visible skin than
the typically furred creature. Instead, these
have mostly human skin covering an animal
body shape.

In one particularly scene, Vignette is flying
back and forth across a ravine to help the
soldiers secure an electrical line – which
required a lot of transitioning between prac-
tical and CGI character work. Then, a CG
marrok attacks Philo on the same cliff until
Vignette rescues him.

The marroks appear in other scenes,
as well, including one wherein three Pact
soldiers strip naked in a forest and inject

4 3e d i t i o n 2, 2020 c gw

culminating in what he calls “a milestone in
an environment build for us” on Carnival Row.

Pushing Boundaries

Some recent streaming and television

series, including Carnival Row, are pushing

A partial practical set at the Mimasery. the quality and quantity of visual effects like
never before – a trend mastered by Game

of Thrones, also known for its the massive

scope of work. And, Paterson is aware of the

comparisons being made between the two

fantasy series. “It’s rare that you get a show

where you’re doing this type of creature

work in addition to these environments,” she

says of Carnival Row. “And the environments

themselves are much more complex than

you usually do for television.”

Amazon Prime Video dropped the entire

first season of Carnival Row simultaneously,

though the VFX teams work on a staggered

ILP crafted an extensive digital environment, including rock, snow, and smoke. delivery for the episodes. Still, all the teams
had what Paterson describes as a fairly

long post on the series, and even did a bit of

added editorial work at the very end. Overall,

more difficult as we started doing more and effects work such as fires and explosions, the visual effects teams had an average of

more big-scale environment work,” Jacobson as well as digital doubles and the blimps 16 weeks per episode, and since they are

explains. “Houdini is a great tool for assem- themselves, plus the destruction,” Jacobson being done concurrently, some of the shots

bling very complex digital builds. Also, the points out. were completed quickly while others were in

integration is so smooth, having everything in Whenever we see the action from the process nearly the entire time. For Episode

one package [for effects and worldbuilding]. airship’s point of view, the shots are com- 3, however, ILP was in production for nearly

Our effects TDs can do setups and provide pletely digital. “We’re seeing the top of the nine months. And, the results show the

them to our lighters to refine certain aspects. mountain and the Mimasery, with all the effort that was taken.

It’s a lean approach, getting everyone using faeries and the Burgue soldiers in the court- The episode is a visually stunning piece,

the same software.” yard below, and it’s all-CG,” says Paterson. but it also gives audiences a much-ap-

In addition to Maya, the artists at ILP “There’s a tiny postage stamp of practical preciated backstory for these two main

use Autodesk’s Mudbox and occasionally people in the courtyard, but everything else characters, as well as context for the

Blender, as well as Foundry’s Mari for textur- – all the faeries flying out, the mountain, faeries’ struggle in the series’ present day.

ing. They also employ asset libraries, such the trees, the snow – it’s all created by ILP. The Season 1 finale solves a number of the

as Quixel’s Megascans for high-detailed They built the whole mountaintop, really. It’s series’ central mysteries, but it also sets the

scanned models. Compositing is done in beautiful.” stage for the growing inequality and tension

Foundry’s Nuke, and rendering is achieved As Jacobson points out, ILP has tackled among the humans and the creatures as we

in Arnold. “It’s integrated into Houdini, so we some amazing creature and other type of enter Season 2. 

didn’t have to switch to a new render engine work since it was founded 12 years ago, but

while we were making this big [software] for the past three to four years has been

transition,” Jacobson points out. called on to do larger-scale environments, Karen Moltenbrey is the chief editor of CGW.

As stunning as the Mimasery envi-

ronments are, the most difficult set, in

Paterson’s opinion, is toward the end of ILP orchestrated a complex battle scene using CGI.

the episode, when the Pact soldiers arrive

in zeppelins and begin to shoot the faeries

from the sky as they hurl Molotov cocktails

at the soldiers. An all-out battle rages as the

sky fills with airships, which rain destruction

down on the monastery.

“We built the airships that are wreaking

havoc in the Mimasery. The fight includes

44 cgw e d i t i o n 2, 2020

Pre-production
in Real Time

PAINTING PRACTICE USES UNREAL ENGINE FOR DESIGN
AND VISUALIZATION ON HIS DARK MATERIALS

BY KAREN MOLTENBREY

S ome might consider the pre-produc- parallel universe where science, theology, show like His Dark Materials, where you’re
tion work Dan May and his Painting and magic are entwined, and where every- building an entire town for one episode,
Practice company do as “gameplay.” one has a personal daemon, a manifestation you’ve got to make really good decisions
And, in a sense, it is. of his or her soul in the form of an animal.  that are practical, financial, and creative.
What Painting Practice’s team created gave
Painting Practice, a West London design The feature film was ambitious, yet the us an ability to make critical judgments that
studio May founded more than a decade television series is even more so in terms of meant we could get it right on the edge of
ago, can be described as a previsualization/ its scope and scale, with a plethora of crea- affordable and, absolutely, on the edge of
VFX facility, offering services such as previs, tures from witches to armored polar bears, shootable, in the sense there was no fat on
animation, VFX, production design, concept and mythical cities and vast landscapes what we built.”
art, and digital matte painting. It has recently created by CG artists at Framestore (see
garnered attention for its design and visual- “Crafting Daemons,” at the end of this fea- Painting Practice originally was set up
ization work for the large sets and complex ture). Before that happened, though, Paint- to facilitate production design, whether
action sequences on HBO’s His Dark Mate- ing Practice provided real-time mock-ups of physical or digital, but as more greenscreen
rials. And they did so using real-time tech- various sequences that enabled the director sets were being incorporated, the compa-
nology by implementing Unreal Technology’s and producers to make decisions concern- ny’s work on previs became more plentiful,
Unreal Engine. ing the series’ complicated sequences. though it still does quite a bit of animation
as well. “Primarily we’re first-in-the-door,
His Dark Materials is an HBO epic fantasy “Unreal was a great way of getting people working for producers, directors, or show-
drama series produced by Bad Wolf Studios to physically be in a space that was not even runners on the vision of the show,” says May.
and based on the book series of the same conceived,” says May, creative director at “We often help get them off the ground,
name by Philip Pullman, the first of which, Painting Practice and one of the company’s and then we work through the traditional
“Northern Lights,” was the basis for the fea- co-founders. pre-production art department space, pre-
ture film The Golden Compass (see “Animal vis, and then in the last few years, with Black
Control,” CGW, December 2007). The story As Joel Collins, executive producer and Mirror and now with His Dark Materials, we’re
introduces Lyra, an orphan, who lives in a production designer on the show, notes,
“These are very expensive sets, and on a

A look at scene lighting for Trollesund.

4 5e d i t i o n 2, 2020 c gw

Painting Practice uses real-time tech to aid the process easier and more automated.
design and vis for His Dark Materials. For the most part, Maxon’s Cinema 4D

serves as Painting Practice’s backbone –
“It’s great for previs and postvis, for effects
work,” notes May. In addition, the facility
employs the Adobe Creative Suite, along
with Adobe After Effects, and Premiere for
editing and Pixologic’s ZBrush for complex
environment and character designs. The
group also uses Frame.io for collaboration
as well as delivery, archiving, and referenc-
ing, along with Quixel products (Bridge and
Mixer) and PureRef, freeware for making
giant art boards.

A look at a smoke-filled scene of Bolvangar. Worlds and Characters

very much hand-in-hand with editorial and and white-card modeling, I went the extra In essence, Painting Practice’s Concept
the effects-room postvis, which is obviously mile and built a VR experience that the team for His Dark Materials on Season 1
a quite complicated and involved process director and producers were able to look at (Season 2’s release date is still pending)
because it involves spending money on and experience firsthand,” he says. worked with the environments and crea-
things that might not even make it into the tures teams as well as with the production
cut, but has to be good enough for people to May began exploring this medium further, designer to help determine what the worlds
make informed decisions about.” examining the use of game engines beyond and characters would look like. They would
gaming applications. then review that with Framestore and Rus-
In some instances, the company will do sell Dodgson, VFX supervisor.
final shots, especially if it is an environment “His Dark Materials is full of complicated
and within the company’s strengths, where- challenges. The show has a good budget, but “Sometimes the directors wouldn’t come
by it will do final matte paintings, as they with the ambition of the show, that budget on board until late in the process, and we’d
did on the Netflix dystopian sci-fi anthology is quickly stretched. So, you have to come need to plan a lot of the sequences. So, it
series Black Mirror. up with creative solutions and find ways to was my job, really, to have a go at making
resolve situations, and this is often the tech- those sequences, with various outcomes
“We’re not looking to be a post house,” nology that can do it for you,” May says. and different tone levels,” May explains.
says May. “We’re purely there to serve and “Then when the directors would start, we
help facilitate productions or post houses to While filmmakers like James Cameron would show them a bunch of things. Some-
get them through design or tricky prob- and Jon Favreau have been using virtual times they would like the ideas or the prin-
lems, whether it’s an animation problem or cameras to scout locations for some time ciple, or they wouldn’t necessarily like all the
[involving] previs storytelling.” now, the process up to now has been out of cameras or the speed or tone. It would be
the cost range for television. But, Unreal has a hit or miss, but a lot of the time we got it
Embracing Real Time opened up that access to far more people. quite close, and then we’d make the chang-
es. But, it is really important and essential
According to May, he had been looking at Nevertheless, the integration of Unreal that the directors have creative ownership
VR and game engines for some time as way Engine into the process called for a retool- of the sequences so they can shoot some of
to explore a space. He used the real-time ing of Painting Practice’s pipeline. When the that previs; otherwise, [the work] becomes
technology to build the night club for “San company began its work with the engine, it null and void.”
Junipero” (Season 3 Episode 4) of Black had difficulty getting the different software
Mirror. “Rather than just doing a few visuals to communicate, whereas now there are As May points out, there were many
more scripts and plug-ins available to make instances on Season 1 where the work paid
off and was used shot-for-shot, and other
times when it was used but was filmed a bit
differently although the elements and beats
were the same; alas, there were even times
when everything changed, particularly when
there were script alterations.

One of the particularly challenging sets
Painting Practice had to design for His Dark
Materials is Trollesund, the main port of
Lapland in the world of Lyra Belacqua, the
story’s heroine. For this, the team made a

46 cgw e d i t i o n 2, 2020

scan of a quarry using photogrammetry Using Painting Practice’s Plan V.
drones and used it to create a high-reso- A visual comparison of work in Unreal (left) vs. broadcast footage (right).
lution model to import into Unreal Engine.
To this, the group added additional models Unreal Engine enables the director and showrunners to plan out scenes.
and textures to complete the real-time A sample of a video plan involving the characters in the popular series.
mock-up.
4 7e d i t i o n 2, 2020 c gw
The previs enabled the director and pro-
ducers to explore the conceptual environ-
ment and make key decisions concerning
the scale of the buildings and the set. The
previs also aided in the storytelling aspect,
so even though the area was called Trolle-
sund Port, not all the action had to take
place dockside, providing more flexibility
for constructing the set, which was then
extended digitally.

The environments are complicated, but
a good deal of the series’ budget from the
start went toward the complexity of the
daemons, which were photoreal, not styl-
ized. After all, the characters are a big part
of the story.

“His Dark Materials and Black Mirror
have an air of fantasy and science fiction
[respectively], but there’s also a reality, a
grounding, and that’s the space where we
fit well,” says May. “And, they have a raft
of very complicated, weird, and wonderful
challenges that we aim to solve.”

Another critical piece of previs the com-
pany provided was for a fight between two
of the series’ armored polar bears. Episode
7, called “The Fight to the Death,” involves
mortal combat between Lyra’s friend,
Iorek Byrnison, and the usurping king Iofur
Raknison. According to Jamie Childs, who
directed the episode, being able to exper-
iment with a virtual camera on a set that
did not yet exist enabled him to explore
the best camera angles to maximize the
scene’s believability.

“I wanted to walk around and look at the
bear fight because I wanted that fight to
feel like it was a real fight being shot, not
a CG fight,” says Childs. “I could actually
do that with Unreal. I could move around
that room physically, get the camera in the
position I wanted, and see on my camera
monitor what was going on, and record
those shots and cut it together – and that
was really freeing for a director.”

According to Childs, the sensation
reminded him of his early days learning his
craft, when he would go out and just film
things to try them out, letting his creativity
find the essence of the story. “I didn’t really

An overview of Trollesund by Painting Practice. gine setup. The company received funding
that helped push further development of
The previs saves valuable shot-planning time on the complex production. the tools, and after some fits and starts, the
visualization/effects company released Plan
think that previs would help me do that side ful sequences occurred in Season 1 Episode V. A new software app based on Unreal En-
of things,” he says. “I thought it might help 4, “Armour,” as Iorek retrieves his armor. gine, Plan V is a bespoke studio environment
me with technically [determining] where “We planned out this town and built it in that enables artists, producers, directors,
to put a camera and things like that, but it the game engine, and could visit it virtually and others involved in a production crew to
actually made me go, ‘Right, I don’t need to inside the studio, rather than having to drive experiment with lenses, storyboards, previs,
worry about any of the noise; I can actually an hour and a half into the Welsh moun- and more.
just go and create something.’” tains,” says May. “And we were able to make
complicated and very cool animations of the With a simplified workflow and a user-­
In the last episode of Season 1, “Betray- bear running around and trashing the town. friendly interface, Plan V is designed to
al,” a great battle plays out with exploding We were able to get in there and discover enable less technically inclined users to
airships firing at the bears and Lyra. The shots with the director [Otto Bathurst] in interact in a high-quality 3D environment
Painting Practice team dropped the director the various sessions. We would make multi- so they can design worlds, sets, and scenes
and DP into the virtual environment, some ple versions of the sequence, do an edit, and for films, television, advertising, and games.
of which “never saw the light of day due to then have different versions of the edit, with The tool also supports local and remote
flooding [at the real location],” recalls May. different endings. We ended up with some- collaboration.
Initially, that scene “blew the VFX budget thing that could be shot, was achievable,
out of the water because we were having so on budget, and Framestore was happy with “You can get an amazing visual look out
much fun,” he adds. “Then we pared it back the volume of shots. It was a collaboration of [Unreal Engine] very quickly, and there’s
to what we could then shoot, although it among all the partners that pulled off well.” an incredible amount of assets that are
was a painful process to create something now available [for it], so it makes building
that was really interesting and fun and quite Plan V environments super quick. But expecting
beyond our wildest dreams, then smash it directors and writers and others who are not
into pieces in a reality check.” Soon interest piqued concerning Painting necessarily technically-minded to pick it up
Practice’s proprietary tools and Unreal En- is a bit of a tall task,” says May. Plan V makes
One of May’s favorite and most success- viewing the Unreal Engine scenes simple
and affordable.

Moreover, as crews continue to work
remotely for whatever reason – be it pan-
demics, travel costs, personal or professional
obligation – tools like this will become even
more valuable.

As May points out, this is such a fast-
paced industry, and it is more important
than ever to keep relevant and keep mov-
ing forward.

As of this writing, work is in progress for
Season 2 of His Dark Materials. Season 1
was a learning phase of sorts, as the group
became familiar with the virtual production
tools and cameras as well as the operation
of the engine itself and the resulting collab-
oration with the vendors. On the upcoming
Season 2, the group pushed the usage
further, creating a virtual space where the
actors could better understand the sight
lines and so forth. And as the seasons
progress, the technology will become even
more useful, May predicts, in handling the
increased fantasy elements and pushing
the storytelling.

“A lot of Season 1 was just trying to
get things to work and discovering cool
ways of working and making some great
sequences. In many ways, we made great

48 cgw e d i t i o n 2, 2020