A Mind-Bending VR Concert
The day before Camila Cabello released her third album Familia in April 2022, the Cuban-born recording star announced a special online event. Six songs from her album appeared, streaming in a phone-friendly vertical video on TikTok, billed as Familia: Welcome to The Family. Camila, TikTok, and Epic Records produced the mini-concert in collaboration with creative director Charlotte Rutherford and XR Studios as part of her promotional tour for her new album, presenting her fans with a psychedelic virtual reality journey through half the content of the Familia album.
Songs were linked by Camila wandering through a corridor of crooked doorways resembling a rainbow-hued Cabinet of Dr. Caligari. Each door led to a new song, including the flamenco-beat La Buena Vida in a crumbling cantina in the clouds, the up-tempo Bam Bam housed in a rainforest temple, the abstract zebra-stripe zone of Psychofreak, and Hasta Los Dientes on a futuristic disco-floor highway.
To conjure the trippy and immersive environments, XR Studios used a suite of immersive technology tools, created in collaboration with Megapixel, which enabled the choreography of animated backgrounds captured in real-time in XR Studios’ LED environment. The combination of live performance in extended reality and augmented reality were an XR Studios specialty that drew on the facility’s strengths of live-event musical productions and virtual reality.
The mini-concert was streamed in a phone-friendly vertical video format on TikTok.
“Camila had already been making music videos as promotions to promote her new album Familia,” notes XR Studios President J.T. Rooney. “She had been on the Jimmy Fallon talk show, and she had a tour coming up. But she wanted another way to promote her album. At the same time, TikTok was looking for a way to feature artists more directly to their audience. And because people are scrolling through their phones all day long, they were looking for a match with Camila. It was quite a long process talking about the concept of the performance before deciding to move forward. All collaborators had to be involved, and we wanted to make sure that it was the best fit for what Camila wanted to achieve.”
Joining XR Studios were champions of the studio’s technology from previous productions. “A creative director named Paul Caslin, and a camera director named Sam Wrench had both worked with us in previous years,” added Rooney. “That included the MTV Video Music Awards and a couple of other projects in our early days. Paul reached out to us to help make it happen and introduced us to the team. The overall creative director for the project was Charlotte Rutherford, who is a fantastic director from England who’s done some awesome music videos and commercial work. Camila had been working with her on a couple of music videos, so Charlotte applied her creative voice from the music videos she had created with Camila and brought that into the virtual world.
Silent Partners Studio rounded out the team as the production’s visual effects and motion design company. “Silent Partners created all the virtual environments,” Rooney explained. “They were creative partners with Charlotte to make something special. Working with all those players, the filmmakers found a way to explore the themes from Camila’s album in visual ways that would not have been possible in a standard live performance. And that’s why they decided to use us at XR Studios. They wanted to create a Hispanic Alice in Wonderland.”
The Techniques Behind the Virtual Concert Experience
The Tec
To bring Camila’s animated worlds to life, XR Studios used Megapixel’s powerful LED processing platform, HELIOS®, to manage virtual content in real-time for a VR shoot in XR Studios’ Hollywood studio location. The technology enabled Rutherford and Wrench to use multi-camera set-ups to capture Camila’s exuberant performances in six virtual set pieces, occasionally incorporating nearly two dozen dancers with Camila in interactive digital environments.
The client’s production designer Liam Moore conceptualized each of the six songs with Silent Partners Studio to create visual aesthetics that complemented Camila and her dancers in their wardrobe and makeup stylings. Silent Partners Studio then designed and created 3D graphic set extensions and effects, which XR Studios incorporated into XR environments that engulfed performers with digital animation streamed across a cove and floor of light-emitting diodes. “Our stage and technology are based around what we call ‘set extension,’” explains XR Studios Chief Technology Officer Scott Millar. “The concept is that you have a live LED volume, but rather than restrict a camera to only seeing the LEDs inside the volume, the camera can go anywhere. Whenever the edge of the LED comes into the shot, we fill that in with virtual content. This means you can have a tiny practical set, a six-foot by six-foot square in a corner of the stage, and your camera can be 200 feet away. As long as the talent is inside the LED volume, we can pin the rest of the environment to the shot from the camera’s perspective.”
XR Studios used a technique called ‘set extension’ to merge 3D graphics with the LED volume.
TikTok’s portrait-style mobile app format set the proscenium for a 9×16 aspect ratio, which XR Studios captured with 6K RED Komodo cameras in native 2:1 format. “We planned to shoot with cameras horizontally,” notes Millar. “One of the biggest uses of our virtual production setup is that it enables us to build environments very quickly and change on a dime, at whatever scale is required. That’s why they wanted to use our studio – because it is great for creating magical worlds and scenes.” The production weighed the benefits of using physical props on set for objects that Camila touched, such as chairs or doors. In those instances, XR Studios planned to incorporate XR rear-projection style, dropping in animated backgrounds behind foreground objects as a real-time in-camera composite. “A.I. has gotten very clever at doing two-dimensional background rotoscoping. But for us to step into a fully virtual world where items appear in front and behind Camila or other people dancing, we would have had to three-dimensionally rotoscope people in real time. That’s where we brought in physical set pieces, where we need that level of interaction and detail.”
Physical props included the corridor of doors, seen in Bam Bam where Camila pushes open a door into a cantina setting. The production design team led by Liam Moore physically built the door. As the camera pulled away from Camila, seated, XR Studios projected animation of crumbling walls, a glowing floor, and a vertiginous sky. Another prop served as a linking device, seen in Psychofreak where a giant pair of red lips offers Camila a portal into an abstract realm of dancing geometric shapes. “The lips were real,” Millar explains. “That way, we didn’t have to worry about occlusion, as Camila crossed in front of and behind that object. It would be very hard to do that trick in XR. Pointing a camera at a real object and lighting it with real lights allows a level of interaction that isn’t currently possible with a virtual production world.”
How XR Studios Created Camila Cabello’s Seamless Virtual Concert Experience
To generate virtual screen content, Silent Partners Studio in Montreal spent approximately two months before the shoot digitally modeling buildings, props, vegetation, atmospheric clouds, debris, and twinkling pixie dust effects in Unreal Engine and Notch. XR Studios then ingested material into its disguise media servers in preparation for real-time playback on set, via Megapixel HELIOS. “Disguise tells Unreal or Notch Playback to render a scene from a certain camera perspective,” relates Millar. “It takes that rendered image and calculates the positions of the camera and the LED wall. Everything is calibrated. The software then puts the content on the screen in the correct perspective for each camera, for each frame. The camera then captures the talent and environment and moves on to the next frame. And that happens at 24, 30, or 60 frames a second.”
The workflow allowed XR Studios and Megapixel technology to feed XR playback to the LED wall using Megapixel GhostFrame™ camera technology to generate correct perspectives of backgrounds for multiple cameras without visual anomalies associated with frame remapping of images. “Frame remapping creates a jumble of superimposed backgrounds on set,” explains Megapixel Product and Project Manager Scott Blair. “That can be distracting for actors, and hard on the crew to stare at all day. With GhostFrame, we can have multiple feeds feeding the screen for up to four camera perspectives. Each camera is tuned to pick up a slice of what it needs to see, while GhostFrame hides other frames. It works the same way as noise-canceling headphones – it catches background noise, creates the inverse of that noise, and that cancels out the noise. GhostFrame allows us to select which imagery we want to hide, and then through a patented process we dynamically create an inversion of that feed. We tune the camera to see a very narrow slice of time when the correct content is displayed. People on set only see one source playing.”
Megapixel’s GhostFrame technology works the same way as noise canceling headphones – it catches background noise, creates the inverse of that noise, and that cancels out the noise.
While the directors captured multiple angles of Camila’s performance, dances played out in real-time, and Megapixel’s LED processor calculated light and color values for backgrounds. “We fed HELIOS the 4K signal from our processor and it sent that to the LED wall,” says Scott Millar. “Where HELIOS comes into play is it understands the key relationships between the look of the LEDs, the talent, and what the cameras are seeing. We light the talent using color charts. The camera looks at the LED and, from that, we calculate the nature of the LED, the pixels, and how much light the LEDs produce. There have been a thousand studies about those relationships, but it’s not a pure light like the sun. We tune our LED image using tools in Parameter Value Language software to make sure the LEDs look as good as possible in camera. The HELIOS processor has a very consistent web-based, multi-user tool for editing on the fly to get the LED processor up and running.”
HELIOS provided dexterous solutions for managing on-set imagery. “This specialized workflow wouldn’t be possible without HELIOS’ pedigree in color accuracy,” comments Jeremy Hochman, founder of Megapixel. “Our team has developed LED processing and display fixtures for use on camera since 2003, and it’s this inherent knowledge that helps us give users the right handles to control their systems quickly and accurately. The combination of XR Studios’ workflow with HELIOS has resulted in an incredibly powerful stage setup.”
For the Familia shoot, XR Studios used Stype RedSpy camera tracking to calculate camera positions. This delivered virtual backgrounds to four cameras – including Technocrane and Steadicam set-ups – within XR’s 54-foot-wide, 35-foot-deep, 20-foot-tall LED volume. XR imagery streamed to the wall and floor sampled 8K outputs, which HELIOS divided into manageable segments. “We split the wall and floor into four 4K outputs,” says Scott Millar. “We had 8K pixels going to the wall, but we only generally rendered 4K in the camera. So, if the camera was zoomed in tight, we rendered 4K concentrated in a small part of the LED wall. When the camera went wide to a 14mm lens revealing the entire wall, we stretched that 4K in the correct perspective across the wall. In person, it looked like less resolution on the wall, but in-camera it was captured at 4K, in pieces.”
To generate imagery for each camera perspective, XR Studios and Megapixel used HELIOS’ Application Programming Interface [API] to enable GhostFrame inside the frustum – or field of view – of each camera. “This patented process allows for a comfortable experience in person while enabling crossfades between multiple cameras,” adds Jeremy Hochman. “That has never been achieved before when shooting in an LED volume.”
Taking cues from background imagery, directors and their lighting teams made use of XR Studio’s grip and lighting package of ARRI SkyPanels, moving lights, and ultraviolet lighting – occasionally revealing Día de Muertos-style u/v pigment makeup on performers, in La Buena Vida. “We had to make sure that the spotlight on the talent was never hitting the back wall,” says Scott Millar. “If you have light hitting the LED, that’s going to be picked up in-camera. The size of the volume is determined by pixel pitch – you can only get the camera so close to the LEDs until you get moiré interference, or start seeing pixels. We used classic stage and theatre lighting – you don’t want a shadow on the back wall, you want the shadows on the floor. That required a scale of lighting to make sure the talent was lit correctly without lighting the LED. When we had twenty dancers on set, that also dictated the scale of the stage.”
Behind the Scenes of a Successful Innovation
Camila’s hardest-hitting song, Psychofreak – which featured the performer in a lemon yellow catsuit with four black-and-white-striped dancers in a kaleidoscope of shifting geometric forms – gave the XR Studios and Silent Partners Studio teams their most creative freedom. “We brought out all the tricks,” comments Millar. “Set extensions only work when the talent is surrounded, there has to be LED underneath the talent on the floor and behind them. We used that to do a trick where the dancers were on stage and we put augmented reality on top of them, leaving Camila open. As Camila was dancing, we placed content over the top of the LED content. When we removed that content, we revealed the dancers, who were on stage with her the whole time.”
Camila’s Psychofreak crawl through a red-lip portal into a cylinder of swirling graphics was achieved with the performer on a steel deck, while LED projections made her appear to be enclosed. “That was quite simple,” notes Millar. “Camila crawled on the deck. We had content on the screen behind her. We did some other clever tricks where she was entering and exiting the lips. When she climbed in, we had the inside of the tunnel play behind the lips. We put the 3D tunnel on the LED and, as the camera tracked around, the tunnel interior was visible in the correct perspective. As the camera looked straight down the lips, it saw a tunnel inside. As it came around, the tunnel interior appeared to disappear.”
From the first meeting through to our continued working relationship, working with Megapixel has been an extremely rewarding and productive experience.
Scott Millar, XR Studios Chief Technology Officer
For the performing artists, virtual elements were an invisible part of the production. “When the technology works perfectly it is simply in the background,” observes Scott Millar. “Camila was fantastic to work with and very patient. She turned up with her team and expected it to work. The clients were very happy. For me, this was one of the best shows we’ve been involved with, in terms of visuals and changing between the songs.”
Dexterity was a key feature of the production for the filmmakers and the XR Studios team. “We’re working in a new world now where the camera is affected by the LEDs,” Millar affirms. “Everything is connected in a way it hasn’t been before.” The connections were made possible by close cooperation with technical support. “From the first meeting through to our continued working relationship, working with Megapixel has been an extremely rewarding and productive experience. The HELIOS platform is the backbone of our virtual production environment, so a solid working partnership is crucial for innovation in this space. Megapixel understood from the start what we needed, they were honest and proactive in development and responded to our requests for both suitable features and the support we needed. The flexibility of the system, the API integration, and the robustness have allowed XR Studios to build our tools to integrate with HELIOS with confidence and support at all times.”
“Camila’s Familia: Welcome to The Family was a really interesting project,” concludes J.T. Rooney. “The client could see the work we and all our collaborators did at XR Studios for this livestream helped her album sales. It was also different from Camila’s music videos, and her promos for her tour. This was something special for her TikTok audience. And it’s important to note that this was not a replacement for a concert or replacement of a music video. This was something new.”
About
Megapixel is an innovative technology partner dedicated to delivering fast-tracked, customized, state-of-the-art LED displays and processing to the world’s leading entertainment, film & TV, and architectural applications.
XR Studios is a cutting-edge digital production company specializing in immersive technology for entertainment. Known for producing Extended Reality (XR) and Augmented Reality (AR) workflow solutions, XR Studios executes innovative experiences for some of the most renowned artists and brands across the globe.
