So how does it work and where did it come from?

The answer partly begins with a 2003 Aussie detergent ad and ends with a 2018 bounty hunter in a shiny suit of armour.

Lost in depthless green

For years, green screens have dominated big-budget movie production.

Actors run about in front of bright green backdrops, emoting with anonymous figures in bright green bodysuits. Then later, months after this filming stage has been completed, an army of computer-generated imagery (CGI) artists painstakingly create a fantastical world pixel-by-pixel.

The end results can be amazing, but the process is a nightmare. The lack of visual information means actors struggle to give their best performance, while cinematographers sometimes choose the wrong kind of lighting.

"All the actors see is green," says Noah Kander, who wrote the Virtual Production Field Guide for Epic Games book. 

"The camera people don't necessarily love it because they're lighting to something that isn't complete."

Plus, those green walls create their own problems. Green-tinged light shows up as green reflections or "spill" on uniforms and props, which have to be removed in post-production.

"I don't know anybody who loves working on green screen because it's a compromise," Noah says.

"Sometimes it looks good and sometimes it looks silly."

Bringing computer games to cinema

Back in the early noughties, director Clayton Jacobson was one of many in the industry wondering if there was an alternative.

As it turned out, help was at hand. In fact, it was in the living room.

The Australian director, who would go on to make Kenny (2006) and other films, was busy filming a detergent ad that, annoyingly, required a green screen.

Every time he moved the camera, the virtual backdrop had to be re-rendered, which took eight hours.

"And yet I'd come home and see my son moving around in a 3D world, with his thumb, in Grand Theft Auto."

The idea bubbled away for years.

Meanwhile, game consoles were getting better and better at rendering 3D graphics in real-time. A big reason for this was the use of specialised processors known as graphics processing units or GPUs.

GPUs now do everything from powering ChatGPT to mine cryptocurrency, but back then, their primary application was gaming. The heated competition between console makers drove down the price of GPUs, with the performance-per-dollar doubling about every two years.

In 2016, Clayton put his idea into action. He and his son cobbled together a set of LED screens to make a small virtual production studio in the backyard shed.

It was one of the first of its kind in the world. Unfortunately, few took notice.

"I couldn't get anyone interested in the technology — they didn't really understand what we were talking about," Clayton says.

He soon abandoned the shed-size experiment.

But, meanwhile, many other filmmakers had the same idea.

In fact, just two years later, an Australian cinematographer would film a TV series that finally showed the world the potential for virtual production, and triggered a sudden change in how films are made.

Tearing down the green screens

In 2018, Greig Fraser began filming the Star Wars spin-off The Mandalorian, which would be released the following year.

"It was the most stressful experience I've ever had," Greig says.

Greig, born and raised in Melbourne, was part of a team taking a giant risk.

They would ditch the green screen and collapse post-production into the production stage.

Around the soundstage, they installed giant LED walls hooked up to powerful computers running the Unreal Engine gaming software (the same engine used for games like Fortnite).

They called this space the "volume", which was the pre-existing term for a stage where visual effects techniques like motion capture and compositing took place.

Loading

The wraparound display could be programmed to show any kind of virtual backdrop in photorealistic resolution. Crucially, the graphics on the walls responded to the perspective of the camera in real-time.

For instance, when a camera is in motion, parallax makes the foreground and background appear to move at different speeds. For a two-dimensional backdrop to appear three-dimensional, it needs to account for this.

It has to be done in real-time, at many frames per second.

Space to play or pause, M to mute, left and right arrows to seek, up and down arrows for volume.

Watch

Duration: 1 minute 39 seconds1m 39s

"All the technology that existed didn't exist for filmmakers," Greig says of these early attempts.

"The [backdrop rendering] was for gamers and the LED panels were for exhibitions or billboards."

Films such as Gravity (2013) and Rogue One (2016) had experimented with aspects of this technology, but none had brought all the parts together.

"We weren't taking just one step ahead, but 10 steps ahead. There was so much room for failure."

But virtual production also opened up possibilities, including dressing the main character in a very shiny suit of armour.

With a green screen, this would have created a nightmare of green spill. 

With virtual production, the armour would reflect the graphics on the LED screens, making the setting appear more realistic.

At the start of filming, Greig had to inform the five directors involved in the series that they would be trialling virtual production.

The directors were horrified.

"They all went pale. They thought, 'What are we committing to?'"

Enter the 'LED volume'

Five years on, the filming of The Mandalorian has become a celebrated story of innovation.

"They took a big leap of faith," Noah Kadner says.

"They threw a lot of money at it and it worked.

"I would put virtual production on the level of when Star Wars did computer-aided motion-control cameras, or when Jurassic Park brought in CG animation."

Since The Mandalorian, LED volumes have been built around the world. There are now more than 200.

Almost all were built post-COVID, as lockdowns spurred production studios to find new ways of filming without travelling to different locations.

US-based Vu Technologies, which built a small experimental volume at the start of the pandemic, now owns and operates four volumes, and has helped build dozens more.

Tim Moore, Vu Technologies co-founder and CEO, says he heard about virtual production through The Mandalorian.

"And I thought it was a really good way to do commercials," he says.

"The day we opened, our volume was booked out for next two months."

Clayton Jacobson has a similar story. In 2020, during the pandemic, producers asked if they they could film in Melbourne. He replied that they'd need a set-up like they had on The Mandalorian.

"I heard myself say that out loud and thought 'Geez I should jump on this now.'"

He built a second virtual production studio — this one much larger than his backyard shed, at 17.5 metres wide and 4.5m high. 

Until recently, it was the only permanent volume in the country outside of a university or film school.

Loading

Last week, US-based NantStudios unveiled the world's largest permanent volume at a facility in Melbourne's Docklands. The 12.19m-high wall of 6,000 panels runs 88.1m long.

A second LED stage is smaller, with a screen 7.9m high, but still larger than the one used to film The Mandalorian.

"It's the most beautiful volume I've ever seen," Greig says.

"It's the best one in the world."

Rendering graphics in real-time for an LED screen this size requires a lot of computing power.

According to Nathan Bazley, director of Virtual Production Concepts, the main LED wall (about 1,000 square metres) has about 130 million pixels.

"A computer has to tell every single one of those pixels what to do 24 to 30 times per second.

"That's a lot of data."

'Better than the real world'

Virtual production is not only a "green screen replacement", but can also be better than filming in the real world, Greig says.

On The Batman (2022), they shot a scene on top of a tower at sunset. Instead of having only minutes to get the shot right, a volume gave them as many hours as they liked.

"We got really good consistent light for 12 hours," he says.

It can also work out much cheaper.

"If you're doing a commercial on the Moon, it's probably cheaper to do virtual production," Vu Technologies' Tim Moore says.

"But if you're doing a commercial in the park, it's probably cheaper in the park."

Partly for this reason, he and others in the industry predict virtual production will become the dominant filming method.

"In the next year or two you're going to see mass adoption."

Nathan Bazley agrees: "The majority of productions in the next five to 10 years will be done this way."

The type of film and TV that gets made will change too, Greig says.

Exactly how is hard to say, but we're likely to see lots of visual effects, with heavy doses of fantasy and science fiction.

"Filmmakers are going to be writing material that's really good in the volume," he predicts.

Then he pauses and rephrases: "That's even better in the volume than in the real world."