Virtual Production for Commercial AV Applications

What is virtual production?

In the world of AV, technology is virtually limitless. The use of networks and other IT-centric technologies and practices have ushered in an unprecedented range of robust, flexible and scalable media applications across the commercial AV integration market. Additionally, as a result of the AV over IP (AVoIP) revolution, network connections are open nearly anywhere customers want to access content.

Ever-increasing computing power is also introducing another specialized technology platform to use in AV setups: virtual production. As the capacity of media servers increases, and display hardware costs decrease, virtual production technology is being used more and more for corporate, house of worship and higher-education applications.

If you've heard of the term “virtual production”, but aren't sure what the term means, you're not alone. So, what is virtual production and how can AV integrators leverage it to deliver more dynamic content experiences? Read more to find out.

Virtual production meaning in the context of commercial AV

With a rising interest from tech-savvy end users and a growing range of AV-ready hardware and software offerings as well as new training opportunities available, now is a great time for integrators to expand their skill set with virtual production.

Furthermore, the demand for more dynamic content is continuing to grow, so there is a strong business opportunity for AV designers and integrators to differentiate themselves by offering their customers with the ability to add high-quality graphics and visual effects to meetings, marketing content, classes and worship services.

Virtual production technology: How it works

Virtual production technology varies according to application and output. Within television and film production, where the focus is on what is captured by the camera, it's often referred to as in-camera visual effects (ICVFX). If you combine physical sets and augmented reality (AR), it's regarded as extended reality (XR) or mixed reality (MR), which can enhance the immersive and interactive elements of hybrid streaming and live events.

Virtual production technology blends physical video production techniques with numerous virtual capabilities to create dynamic media. In the highest-end cases, video game engines are used to design 3D photorealistic environments and render them in real-time so they can be explored by tracked cameras moving through the virtual world. It's also possible to use 2D content. The virtual environment is displayed in real-time on large-scale video walls or projection screens that are used as backdrops for stages of all sizes and scales.

Basically, it's like a much more advanced and potentially expensive green-screen setup — with the main difference being that the computer-generated, interactive background is viewed and captured live, in real-time, instead of added in post-production. So, anyone presenting inside the space actually feels surrounded by the virtual scenic design, instead of being asked to imagine that a giant green wall is a mountain range or a seascape or a spaceship.

Because of the immersive, live, interactive surroundings they create, these virtual stages can feel more “real-life” than virtual. This is helpful for presenters, performers and producers who can see something close to the final product right there, live, all around them. Except in cases where a layer of graphics or AR content is being added to the final product — those extra details will be seen only in the livestream or recorded output, unless the live-composited image is shown on Image Magnification (IMAG) screens or other monitors in the space, allowing the presenters to see and interact with all the layers as they are added in real-time.

There are many different ways virtual production sets can be configured, which will appeal to AV integrators. Let's look at some of the variables:

The volume and lighting

First, your customer will need to set the scene with a large backdrop that will display the virtual environment. This is called “the volume.” The size, shape and media of this volume will vary according to budget, space, and most importantly, your customers' goals. What are they trying to achieve using virtual production technology? And what type of media will help tell the story that delivers that result?

If they're looking for atmospheric imagery that will always be in the deep background of a camera shot or a live stage setup, projection might work. But if they want flexibility in how they're shooting, or they want to add interactivity to their content, LED might be the way to go.

With LED, the budget and scale of the stage will determine pixel-pitch. Remember, there is a big difference between how a camera sees LEDs compared to the human eye. The camera will pick up nuances and color problems that we can't see. One of the biggest issues related to this is the moiré effect, where the tight pattern of lines between pixels on LED walls creates wavy lines in a video image. An example of this could be when you point a camera at someone wearing a checked pattern, and the video image ends up looking distorted because the camera's refresh rate struggles with the tiny squares on the fabric.

There are things your customers can do to avoid moiré. But it's good to check with the camera and LED screen manufacturers first to find out if their setup is compatible. Those conversations will be helpful too, because soon there might be more options - many LED display manufacturers are developing products that will work better with cameras. Also, make sure your customer's video wall processor is powerful enough to keep latency low, to avoid sync problems between the content on the screen and the movements of the camera. LEDs also have the added benefit of providing some lighting support, and ceiling and floor displays can also help create realistic reflections on physical set pieces and presenters.

Cameras and sensors

Whether your customer uses a manually operated camera or a production-quality PTZ, they'll need to calibrate color reproduction between the wall and camera. Cameras carry their own set of considerations, as they may only be compatible with certain LED screens. If the refresh rate of the camera is different than the screen, visual artifacts or dropped frames can be a result.

Cameras are tracked using infrared markers or sensors, which follow the shot and shift the virtual background based on the position and movement of the capture device. Tracking technologies are continuing to improve, making it easier to present realistic and dynamic content, especially in virtual production environments.

Media servers and real-time rendering

Media servers enable real-time content changes, which enhances interactivity and makes it easier to make certain adjustments quickly. This makes the whole content pipeline faster, making it feel more like a live performance instead of standard media playback.

Real-time rendering plays a critical role in virtual production workflows, transforming how content is created and delivered during production. Instead of waiting hours or days for final output, teams can visualize and adjust scenes instantly. This means directors, designers and operatives can see changes as they happen - from lighting shifts to environmental transitions - and make creative decisions on the spot.

The flexibility real-time rendering offers makes it ideal for fast-paced settings such as live broadcasts, interactive product demos and virtual events. With real-time rendering, it's easy to adapt visuals mid-performance, respond to unexpected needs or iterate quickly, all without derailing the production timeline.

When it comes to choosing between 2D and 3D, keep in mind that the answer isn't always to do the most technically intense 3D workflow for the ultimate cool factor. Sometimes a 2D background is more than enough. Several video production systems on the market will readily handle 2D content, including virtual backgrounds and on-screen graphics.

If your customer is interested in 3D, then a game engine needs to be added to the workflow. These are used to create backgrounds and 3D environments that can be navigated — just the way a player would move through a game's world. The background can move in any direction or velocity needed to allow the presenter to “move” through a space without physically stepping beyond the volume's edges.

The size of the volume will vary — for film and television production, virtual production sets are gigantic. But even a small space can also be used, due to the miracle of set extension. That's when a media server is used to stitch a virtual background in real-time outside the LED walls. So, if a presenter is standing in front of a 12' wide x 8' tall LED screen, your customer can make it look like they're standing in a volume that's twice that size, or more.

Remember, the real-life image captured in-camera is composited in real-time with the virtual scene for live streaming, broadcast or recording. So green-screen rules still apply — people in the room won't see the extended set. And if someone steps beyond the volume's edges, they will disappear in the video output. But wherever the audience sees the stitched-together image, either on an IMAG screen somewhere else in the venue, or on any other video platform, they'll get the full experience.

Keep in mind, media servers are becoming more powerful all the time, so real-time content changes are much faster now. And those media server companies are also working diligently to make it easier to use these advanced systems. They're creating accessible interfaces that merge screens, mapping and compositing together for ease of use.

Control systems and workflow integration

Once the content assets are loaded into a content management system, they can be modified for any presentation, such as swapping out logos and rearranging pieces to tell stories in a narrative that suits the purpose of a meeting, classroom exercise or message. For smaller streamlined setups where there is only one presenter with a single microphone the control of the full audio and video system is handled via presets on an iPad or a simple video production push-button interface.

Virtual production in AV applications

For AV integrators looking to dive into virtual production, there are many potential applications within a variety of verticals. This powerful technology is reshaping how a wide range of industries create content, communicate ideas and engage with their audience - all with fewer logistical constraints and more creative freedom.

From immersive corporate communications to real-time simulations in education, virtual production technology allows integrators to deliver scalable, high-impact solutions that align with their customers' evolving needs. Let's explore how various industries can benefit from virtual production.

Corporate and enterprise communication

Since the earliest days of virtual production, corporate and enterprise customers have seen the technology's potential to distinguish their brand using marketing, sales, social media, meetings, live streaming and training content. Lots of companies already have advanced, broadcast-level studios in-house. So, it was only natural for them to make the leap to a production process that would make it possible for them to do even more within those dedicated spaces.

But the good news is, it's not just the big companies that can do this. Virtual production spaces don't necessarily have to be gigantic. There's a lot your customers can do even in a 10' x 10' setup, thanks to set extension and all those in-camera visual effects. For a compact setup, use a basic 2D media server for playback, and opt for mostly stock backgrounds.

In terms of location, sometimes all you need is an empty office or conference room. Depending on the amount of space and budget, an array of volume options are available. It's possible to do a one-screen backdrop, a two-walled corner setup or a three-sided LED setup. If you have a space as large as an actual broadcast studio, go for the full 360-degree surround with LED ceilings and floors. Install some production racks, switchers and editing bays, and now you're repurposing underutilized office space for your customers to produce content that they would have outsourced or paid for studio time to create.

With virtual production technology, your customer can make meetings, product launches, training and recruitment, so much better by helping them to:

• Product high-quality content without the need for location changes or elaborate sets
• Create lifestyle videos or a whole year's worth of social media
• Save money on post-production
• Make product demos feel more interactive
• Customize content live

Houses of worship

Houses of worship are an interesting application, because much of the production will actually be closer to a live performance. Rather than simply creating content that is built for the camera and/or the streaming audience, there will also be people in the room, seeing the LED walls and sets. Fortunately, to make the most of the in-camera visual effects, simply use the IMAG screens to show the composited images. All that's needed is a tracked camera and then you're most of the way there for a virtual production setup. And of course, the augmented content will be great for online viewers.

Higher education and training environments

Colleges, schools and universities can use virtual production technology as a helpful resource for different subjects. It can be used as an instructional tool by theater, film and communications departments to help prepare students for the evolving workplace. Many universities are also building LED volumes to use for new creative technology and immersive entertainment departments.

And there are many additional possibilities, including:

• Interactive content that combines physical materials with 3D models that can be explored in real-time
• Immersive simulation and training environments
• Sports media, broadcasts, press conferences and team promotions
• Recruitment and retention efforts
• Collaborative and commercial opportunities for local industry, which might borrow or rent the space
• Create a simulation environment for training and development

Conclusion: The future of virtual production for commercial AV

In this limitless era, the creative technologies of video game developers, mixed reality producers, animators, interactive programmers and generative media artists are playing a more significant role in traditional AV applications. This physical-meets-virtual video production process, once the domain for the filmmaking industry and massive-scale esports multimedia events that blend live audiences with streamed content worldwide is now being applied more and more by C-suite executives, professors and worship leaders.

Want to learn more about virtual production and how to bring this approach to your customers? Contact our Systems Design team today to explore the opportunities virtual production technology offers and learn how ADI can help you with your next virtual production project.