TY - JOUR
T1 - A GPU-Enabled Framework for Light Field Efficient Compression and Real-Time Rendering
AU - Zhao, Mingyuan
AU - Sheng, Hao
AU - Chen, Rongshan
AU - Cong, Ruixuan
AU - Wang, Tun
AU - Cui, Zhenglong
AU - Yang, Da
AU - Wang, Shuai
AU - Ke, Wei
N1 - Publisher Copyright:
© 1968-2012 IEEE.
PY - 2024
Y1 - 2024
N2 - Real-time rendering offers instantaneous visual feedback, making it crucial for mixed-reality applications. The light field captures both light intensity and direction in a 3D environment, serving as a data-rich medium to enhance mixed-reality experiences. However, two major challenges remain: 1) current light field rendering techniques are unsuitable for real-time computation, and 2) existing real-time methods cannot efficiently process high-dimensional light field data on GPU platforms. To overcome these challenges, we propose an framework utilizing a compact neural representation of light field data, implemented on a GPU platform for real-time rendering. This framework provides both compact storage and high-fidelity real-time computation. Specifically, we introduce a ray global alignment strategy to simplify the framework and improve practicality. This strategy enables the learning of an optimal embedding for all local rays in a globally consistent way, removing the need for camera pose calculations. To achieve effective compression, the neural light field is employed to map each embedded ray to its corresponding color. To enable real-time rendering, we design a novel super-resolution network to enhance rendering speed. Extensive experiments demonstrate that our framework significantly enhances compression efficiency and real-time rendering performance, achieving nearly 50× compression ratio and 100 FPS rendering.
AB - Real-time rendering offers instantaneous visual feedback, making it crucial for mixed-reality applications. The light field captures both light intensity and direction in a 3D environment, serving as a data-rich medium to enhance mixed-reality experiences. However, two major challenges remain: 1) current light field rendering techniques are unsuitable for real-time computation, and 2) existing real-time methods cannot efficiently process high-dimensional light field data on GPU platforms. To overcome these challenges, we propose an framework utilizing a compact neural representation of light field data, implemented on a GPU platform for real-time rendering. This framework provides both compact storage and high-fidelity real-time computation. Specifically, we introduce a ray global alignment strategy to simplify the framework and improve practicality. This strategy enables the learning of an optimal embedding for all local rays in a globally consistent way, removing the need for camera pose calculations. To achieve effective compression, the neural light field is employed to map each embedded ray to its corresponding color. To enable real-time rendering, we design a novel super-resolution network to enhance rendering speed. Extensive experiments demonstrate that our framework significantly enhances compression efficiency and real-time rendering performance, achieving nearly 50× compression ratio and 100 FPS rendering.
KW - GPU-Enabled
KW - Light Field Compression
KW - Light Field Rendering
KW - Neural Light Field
KW - Real-Time
UR - http://www.scopus.com/inward/record.url?scp=85212830279&partnerID=8YFLogxK
U2 - 10.1109/TC.2024.3517743
DO - 10.1109/TC.2024.3517743
M3 - Article
AN - SCOPUS:85212830279
SN - 0018-9340
JO - IEEE Transactions on Computers
JF - IEEE Transactions on Computers
ER -