MFSRNet: spatial-angular correlation retaining for light field super-resolution

Sizhe Wang, Hao Sheng, Da Yang, Zhenglong Cui, Ruixuan Cong, Wei Ke

Research output: Contribution to journalArticlepeer-review

Abstract

Light field (LF) images acquired by hand-held devices suffer from a trade-off between spatial and angular resolutions. To solve this problem, super-resolution (SR) in the spatial and angular domains is studied separately in previous works. However, spatial-angular correlation can not be reconstructed effectively by the separate SR methods. In this paper, a multi-scale feature-assisted synchronous SR network (MFSRNet) is presented to retain spatial-angular correlation for spatial and angular super-resolution, which consists of four modules: multi-scale feature extraction (MFE), view relation reconstruction (VRR), SR information acquisition (SIA) and up-sampling. The MFE module is used to acquire multi-scale angular SR features from low-resolution LF. In the VRR module, these multi-scale features are concatenated with two original adjacent low-resolution view images to reconstruct the angular relation among original and new views. Then, a continuous fusion mechanism is proposed in the SIA module to obtain spatial SR information from four surrounding views and reconstruct the spatial-angular correlation in LF. Finally, super-resolved LF is generated by allocating the sub-pixel information in the up-sampling module. Furthermore, a combined loss is proposed to provide constraints on both angular feature extraction and spatial and angular synchronous SR, and train MFSRNet in an end-to-end fashion. On synthetic and real-world datasets, experimental results show that our algorithm outperforms other state-of-the-art methods in both visual and numerical evaluations. Especially, our method brings significant improvements for sparse LFs from the dataset STFgantry using MFSRNet. Our method improves PSNR/SSIM while preserving the inherent epipolar property in LF.

Original languageEnglish
Pages (from-to)20327-20345
Number of pages19
JournalApplied Intelligence
Volume53
Issue number17
DOIs
Publication statusPublished - Sept 2023

Keywords

  • Light field super-resolution
  • MFSRNet
  • Multi-scale features
  • Spatial and angular synchronous SR
  • Spatial-angular correlation retaining

Fingerprint

Dive into the research topics of 'MFSRNet: spatial-angular correlation retaining for light field super-resolution'. Together they form a unique fingerprint.

Cite this