Aximmetry: 3D Dynamic Mask Reconstruction

In television broadcasts, large-scale galas, and esports live streams, the breakout applications of AR (Augmented Reality) virtual elements have long been commonplace. However, when a virtual dragon coils above a real stadium, or a virtual skyscraper rises from the center of a real stage, technical teams inevitably encounter a fundamental pain point concerning spatial order:“Depth Occlusion Failure between Virtual and Real Spaces”。

The most classic blunder scene is: When a singer walks forward on stage, a virtual AR sculpture that should be behind the singer is still forcibly superimposed in front of the singer's body due to the lack of a precise physical occlusion mechanism. This “perspective disorder” instantly strips the AR element of its sense of physicality, degrading the entire performance into a crude sticker-like graffiti.

在native Unreal Engine 5 (UE5)In live broadcasts, solving virtual-real occlusion is extremely tricky:

• If traditional 2D Garbage Mattes are used, once the camera pans or zooms, the static 2D matte will quickly misalign and reveal the trick due to Parallax Error.
• If a complex “Opacity Mask” material is used inside UE5 to forcibly clip the 3D scene, the high-frequency material updates will directly interfere with the engine's rendering pipeline, causing severe frame rate jitter.

Aximmetry With its unique“3D Garbage Matte & Occlusion Compositing Pipeline”, it reconstructs the virtual-real depth clipping at the GPU level, granting AR elements absolutely correct spatial order on complex physical stages.

I. Digitizing Physical Space: 3D Garbage Matte Technology Based on Tracking Data Binding

In a real green screen studio or live-action studio, the ceiling lights, surrounding light stands, and physical dead zones at the edge of the green screen are collectively called “physical reveal areas.” Traditional keying software requires manual post-production matte adjustments, which is impossible in real-time live broadcasts.

Aximmetry employs“3D Garbage Matte”technology geometrically digitizes the physical world:

Stage Geometry Import

During the system initialization phase, calibration engineers import a simple 3D CAD model of the physical studio (including the physical boundaries of the green screen, the 3D dimensions of light stands and load-bearing columns) into Aximmetry.

Real-time Tracking Matrix Synchronization

In Aximmetry's flow graph, this virtual 3D space model does not participate in visual rendering but is instead bound 1:1 with the real camera tracking system viaProjection Matrix AlignmentWhen the real camera sweeps an arc on the jib arm, the virtual matte camera inside Aximmetry rotates in perfect sync.

GPU-level Real-time Geometry Clipping

At the very front of Aximmetry's compositing pipeline on the GPU, this synchronized 3D model is rendered as a real-time binary matte. Any area outside the entity range of the 3D model (i.e., physical reveal areas) has its pixels forcibly removed and replaced with the virtual 3D background before entering the keyer. Because this geometry clipping is performed directly at Aximmetry's low level using lightweight pixel shaders, Unreal Engine bears no geometric computation for the physical space, successfully eliminating physical reveals at the source.

II. Depth Fracturing for Dynamic Actors: Pixel-level Comparison of Scene Depth

After solving background reveals, the more challenging task is “bidirectional traversal occlusion between dynamic actors and AR elements”—for example, a singer walking a full 360-degree circle around a virtual ancient Greek stone pillar.

This requires the system to make pixel-level depth tests within milliseconds per frame.

Aximmetry avoids the traditional cumbersome approach of mesh reconstruction inside UE and has developed a“Per-Pixel Depth Reprojection”pipeline:

Extracting the UE5 Scene Depth Pass

Through its high-bandwidth data channel, Aximmetry extracts the uncompressedScene Depth Mapfrom Unreal Engine's rendering pipeline in real-time. This map records the absolute physical distance of virtual stone pillars and virtual walls from the camera.

Reconstructing the “Depth Block” for the Live-Action Character”

Using the dynamic billboard positioning mentioned earlier, Aximmetry assigns a dynamic depth value to the singer in 3D space based on the physical distance between the camera and the singer transmitted by the tracking system.

Pixel-by-Pixel Depth Comparison

In Aximmetry's 2.5D compositor, the GPU shader adjudicates each pixel:

• For areas where the singer's pixel depth isless thanthe virtual scene depth (singer is in front), the live-action footage of the singer is output directly, occluding the virtual stone pillar;
• For areas where the singer's pixel depth isFor areas where it isgreater than

the virtual scene depth (singer has moved behind the pillar), the pixel from the virtual stone pillar overwrites it.

This GPU-based hardware-level pixel adjudication has a response time of less than 0.1 milliseconds and can handle any complex cross-traversal in real-time with perfect resolution, completely eliminating the awkward “clipping” in AR compositing.

III. Eliminating "Hard Edges": Dynamic Matte Smoothing Based on Motion Vectors and Penumbra Feathering

During fast motion, rigid 3D clipping produces extremely unsightly aliasing and high-frequency flickering at the virtual-real boundary. Especially when the camera pans quickly or the singer waves their hand rapidly, hard edges instantly expose the artificial compositing.“At the very end of its compositing pipeline, Aximmetry deploys a”"Penumbra Feathering Based on Motion Vectors"

algorithm:

Motion Blur ReconstructionAximmetry extracts theMotion Vector Map

rendered by Unreal Engine, which records the movement direction and speed of each pixel in the virtual scene between consecutive frames. When compositing the edge of the depth occlusion, the algorithm applies adaptive directional blur to the matte boundary based on the motion vector magnitude at that point. If the camera is panning rapidly to the left, the occlusion edge automatically generates a motion trail to the right that conforms to the physical camera's shutter angle.

Sub-pixel Softening

At the junction of depth transitions, Aximmetry uses a high-order interpolation algorithm to perform sub-pixel feathering contraction on the Alpha matte. It perfectly preserves the semi-transparent transition of the singer's flying hair or fluttering clothing in front of and behind virtual elements, making the boundary between the virtual and real worlds as smooth as silk.

Conclusion: Rebuilding the Geometric Laws of Virtual-Real Space

In Augmented Reality (AR) production, "being visible" is only the first step; "being touchable and occludable" is the golden ratio for achieving ultimate immersion. Without strict physical space occlusion, no matter how dazzling the virtual elements are, they are just posters floating in the air.

Aximmetry Unreal Engine 5 outlines magnificent digital wonders, but when faced with real actors who change unpredictably and lack mesh topology, it cannot independently rebuild order in the 3D depth world.

With its precise flow graph design and powerful GPU 2D/3D hybrid compositing engine, it acts as the strictest "geometric judge" between the virtual and real worlds.