Aximmetry: Physical Stitching for Set Extension

On the set of Extended Reality (XR) and Mixed Reality (MR) productions, when the camera performs a wide pan and the line of sight extends beyond the physical LED wall boundary, the system must activate“Set Extension”technology.

This requires the system to respond at extreme speed, using virtual backgrounds rendered in Unreal Engine 5 (UE5) to seamlessly fill the blank area beyond the LED screen in real time.

However, when a Technical Director (TD) attempts to achieve this “virtual-real stitching” under the native UE5 architecture, two forces will violently erupt at the edge of the physical LED wall, causing“Spatial Tearing”：

1. Parallax & Nodal Slip: Due to micro-latency in the tracking system and the physical offset (Nodal Offset) between the optical nodal point of the physical camera and the tracker sensor, when the camera moves, the junction between the “live-shot + background” inside the LED wall and the “purely virtual extension” outside will produce extremely noticeableDrifting。
2. Edge Distortion Breakage: Real lenses inevitably have optical distortion, which becomes more severe toward the lens edges. When a virtual column spans the LED boundary, the portion inside the LED wall is curved due to the live-shot lens distortion, while the extension portion, as a purely digital rendered signal, is perfectly straight, causing the column to “break” directly at the junction.
3. Photometric Color Temperature Discrepancy: Due to the light-emitting material and viewing angle attenuation, the physical LED screen produces a yellowish, darker color cast at the edges; whereas the digital extension is a perfect RGB signal, resulting in a glaring brightness discontinuity on either side of the seam.

Aximmetry With its specially designed spatial geometry correction“3D Reprojection and Distortion Unification Pipeline”, it performs a physical-level stitching of the virtual-real boundary at the GPU memory level.

I. Parallax Elimination: Spatial Alignment Based on Bidirectional Reprojection Matrix

To achieve absolute perspective consistency between the extended virtual background and the interior of the LED wall, Aximmetry refuses to use simple 2D image stitching. Instead, it reconstructsHomography Reprojection"Penumbra Feathering Based on Motion Vectors"

Digitization of Physical LED Geometry Model

In its flow graph space, Aximmetry reconstructs the spatial geometric topology of the physical LED wall (including curvature, tilt angle, and physical anchor coordinates) at sub-millimeter precision, 1:1.

Real-Time Optical Node Compensation

The system captures 6DOF tracking data in real time and applies a3D Transform Matrixat the underlying level, accurately converting the tracker's origin to the lens's physical optical node. This algorithm eliminates the lever arm error caused by the “tracker orbiting the lens” during physical camera rotation, ensuring that the Inner Frustum and Set Extension share the same absolutely determined visual axis center.

3D Depth Reprojection

Aximmetry performs real-time pixel-level fusion of the 3D depth map of the extension scene with the live video footage. By projecting both images into the same Standard Frustum, it eliminates any “perspective slip” caused by differences in network data arrival order, keeping the virtual-real junction rigidly bonded in spatial geometry.

II. Distortion Unification: GPU-Level Undistortion and Redistortion Cascade Pipeline

To solve the optical disaster of “line breakage” at the junction, the purely digitally rendered extension background must be “infected” in real time with the same optical distortion properties as the physical camera lens.

Aximmetry deploys a“Undistort - Stitch - Redistort”cascade pixel shader:

Dynamic Reading of High-Order Distortion Parameters

Aximmetry extracts the calibrated lens radial distortion (Radial Distortion, K1/K2/K3) and tangential distortion (Tangential Distortion, P1/P2) parameters in real time.

Inverse Undistortion and Merge

One millisecond before compositing, Aximmetry's GPU shader first performsUndistorton the input live camera image (including the scene inside the LED wall), stretching it back to a perfectly linear space. With both sides absolutely straight and distortion-free, Aximmetry performs pixel-level stitching of the internal and external images in GPU memory.

Global Redistortion

The complete stitched image is then processed through Aximmetry's high-order distortion shader forRedistortcalculation. The entire image (including both the physical LED portion and the virtual extension portion) is uniformly endowed with the physical curvature and chromatic aberration consistent with the real lens edge. This ensures that any lines or objects spanning the virtual-real boundary have absolutely continuous curvature at the pixel level, completely curing the visual “spatial fracture.”

III. Photometric Concealment: Edge Physical Light Falloff Compensation and Adaptive Feathering

In the physical world, LED screens exhibit inherent vignetting and color shift at side viewing angles. At the final stage of compositing, Aximmetry uses a“Photometric Equalizer”to smooth out this cold physical boundary:

Establishing LED Edge Light Falloff Curve

Aximmetry's color engine pre-scans and records theVignetting Profileand color temperature offset values of the physical LED wall at different camera shooting angles.

Dynamic Chromatic Compensation

When the camera moves to a side angle, Aximmetry calculates the angle between the camera's line of sight and the normal of the LED wall in real time. The system automatically and dynamically injects subtle color temperature compensation and gamma attenuation into the digital extension background pixels, actively making the extension image conform to and match the “grayness and yellowness” of the physical LED screen at side viewing angles.

Sub-Pixel Adaptive Soft Blending

At the final seam edge, Aximmetry employs a high-frequency luminance-basedAdaptive Blend MaskIt applies a non-linear alpha gradient at the junction, breaking up and dissolving the hard edge within a tiny width of a few pixels. This makes the physical LED wall boundary line completely “physically invisible” to the audience.

Conclusion: Recasting the Fractured Space

Set extension is the key for XR stages to break through physical space limitations and move toward infinite vastness. But if the geometric, distortion, and photometric discontinuities at the virtual-real stitching point cannot be overcome, this key will only become glaring evidence of a flaw.

Unreal Engine 5 outlines a boundless digital territory, but it cannot perceive the glass refraction of a physical lens, nor can it predict the light-emitting physical defects of a physical LED screen.

Aximmetry Here, it demonstrates the rigid capability of an industrial-grade central control.

It uses spatial reprojection to eliminate parallax slip, a GPU-level distortion cascade pipeline to smooth out broken geometry, and photometric compensation to dissolve color cast boundaries. Aximmetry recasts the fractured spatial order at the source of photons and pixels, condensing the limited physical stage and the infinite digital universe into the same unbreakable physical reality within the rapidly moving lens.