From “Post-Production Rescue” to “On-Set Color Grading”: Breakthrough Practices of ACES and Color Management in XR Virtual Production

In traditional film and television production workflows, “fix it in post” is often the go-to phrase for production crews dealing with unexpected lighting and color inconsistencies.

However, on an XR virtual production (ICVFX) set, this phrase becomes an unbearable cost.

The core value of virtual production lies in on-set final cut and “Final Pixel.” When real actors stand in front of a glowing LED screen, the physical reflected light on the actors' bodies, the RGB narrow-spectrum light emitted by the LED screen, the digital colors output by Unreal Engine's real-time rendering, and the photoelectric signals recorded by the cinema camera sensor—all converge within milliseconds.

If even one link in the chain has a color space that is not calibrated and aligned, the image will exhibit a fatal “plastic look”: virtual and real elements disconnect like stickers, skin tones turn eerily green or gray, shadow details are lost, and highlights blow out to a sickly white.

To seamlessly blend virtual and real, we must abandon temporary color grading that relies on human visual experience and apply color science throughout the entire pipeline. ACES (Academy Color Encoding System) is the core key that bridges this gap.

I. The Root Cause of Virtual-Real Mismatch: Spectral Conflict Between LED Screens and Physical Cinema Cameras

Many teams trying virtual production for the first time often harbor an intuitive misconception: as long as the screen colors look realistic to the naked eye in the studio, the camera should capture them without issue.

Reality often teaches newcomers a lesson. The human eye has extremely strong dynamic adaptation and color constancy, while a cinema camera's CMOS sensor records photon energy strictly according to a fixed quantum efficiency curve.

There are three main underlying causes of color deviation:

1. The Narrowband Spectrum Trap: Commercial-grade LED chips are typically composed of red, green, and blue light-emitting chips with extremely narrow wavelengths. The resulting spectrum is discontinuous, consisting of discrete peaks. In contrast, natural daylight or tungsten filament lamps produce smooth, continuous spectra. When an actor's skin is illuminated by this narrowband spectrum, its color rendering (CRI/TLCI) becomes distorted, and the reflected light captured by the camera cannot reproduce realistic skin texture.
2. Cross-Device Gamut Tearing: Unreal Engine's internal physical lighting calculations are based on a linear wide gamut; the LED processor may receive a custom wide gamut or Rec.709 signal; the screen's own panel is limited by physical hardware and has a color saturation ceiling; and finally the camera records using a proprietary Log curve (such as ARRI LogC4 or REDWideGamutRGB). If precise mathematical conversion matrices are missing between these multiple conversions, color information will completely collapse through clipping and compression.
3. Parallax Color Shift and Viewing-Angle Falloff: The light-emitting surface of an LED screen is constrained by physical lenses and packaging processes. As the angle between the camera and the screen changes, the screen's color temperature and brightness drift non-linearly. When shooting off-axis, the background may imperceptibly drift from warm white to a cyan-gray.

II. Building a Full-Chain Virtual Production Pipeline Based on ACES

To break this chaos, the entire workflow must be unified under a mathematical reference system that is device-independent and has an extremely wide dynamic range. This is precisely the original design intent of ACES (Academy Color Encoding System).

In a mature virtual production workflow, ACES is not merely a post-production color grading tool—it is the full-pipeline language from 3D asset generation to final shot output:

1. Asset Generation (Texture & Material): 3D scene textures created in Substance Painter or Maya must have their color attributes standardized before being imported into the real-time rendering engine. BaseColor textures need to be linearized under the ACEScg color space, while normal and roughness maps must strictly remain as Raw data, ensuring that materials exhibit diffuse reflection and specular highlights consistent with real-world physical laws under physical lighting calculations.
2. Engine Rendering (OpenColorIO / OCIO): Unreal Engine uses the embedded OCIO (OpenColorIO) plugin to set the computational space to ACEScg. When the engine outputs signals to the LED screen, a specific Display Transform is applied. This means that before the image is sent to the screen, the screen's color reproduction capabilities have already been mathematically anticipated, avoiding uncontrollable hard color clipping.
3. On-Set Synchronization and 3D LUT Calibration: This is the most critical “closed-loop calibration” in virtual production. The technical team must use a spectroradiometer to perform actual measurements and calibration for the specific LED screen model, specific camera model, and main lens set on set. By generating a specific 3D LUT, when the camera films a white patch on the LED screen, the color values reproduced by the sensor strictly match the readings of a real physical gray card under the same illuminance.

III. From “Rescuing” to “Creating”: A Qualitative Shift in On-Set Color Decision-Making

When ACES color management runs reliably on set, the collaboration paradigm across all departments of the crew undergoes a dramatic transformation:

First, the DP and colorist truly gain “on-set control.” In the past, the cinematographer could not accurately judge the final light and shadow layers in front of a green screen; now, the cinematographer can use a grading panel on the on-set monitor to directly fine-tune the color of the virtual background and its correlation with the color temperature of physical lighting. What you see is what will ultimately be delivered to the audience.

Second, post-production delivery cycles are significantly shortened. In traditional VFX shoots, one of the most time-consuming steps is the complex and painful “despill” and virtual-real edge matching for green screen keying footage. In a precisely calibrated virtual production studio, the camera captures fully matched diffuse reflection and real physical ambient light in one pass. The post-production VFX department receives footage with over 85% of light and color integration already completed, and can proceed directly to finishing and final output.

Third, cross-project fidelity of digital assets. Digital scenes built under the ACES standard have their lighting information fully preserved in a high-dynamic-range color model. Whether this production uses an ARRI camera or the next one switches to a Sony Venice, the digital assets do not need to be rebuilt from scratch—they only need to be re-rendered with the new camera's Input Device Transform (IDT) for seamless reuse.

Conclusion: Color Science Is the Soul of Virtual Production

Virtual production is often labeled with visually striking technical tags like “big screen,” “compute power,” and “game engine.” But the subtle boundary that determines whether a shot looks like a “premium blockbuster” or a “cheap browser game ad” often hides within those spectral curves and color mapping matrices that are imperceptible to the naked eye.

Virtual production without scientific color management is just blindly gambling in front of an expensive big screen; only teams that embrace ACES and rigorous color pipelines truly hold the key to the next era of film and television industrialization.

Give color back to science, and give time back to art—this is the most fascinating industrial charm of contemporary virtual production.