MFSRNet: spatial-angular correlation retaining for light field super-resolution

Light field (LF) images acquired by hand-held devices suffer from a trade-off between spatial and angular resolutions. To solve this problem, super-resolution (SR) in the spatial and angular domains is studied separately in previous works. However, spatial-angular correlation can not be reconstructed effectively by the separate SR methods. In this paper, a multi-scale feature-assisted synchronous SR network (MFSRNet) is presented to retain spatial-angular correlation for spatial and angular super-resolution, which consists of four modules: multi-scale feature extraction (MFE), view relation reconstruction (VRR), SR information acquisition (SIA) and up-sampling. The MFE module is used to acquire multi-scale angular SR features from low-resolution LF. In the VRR module, these multi-scale features are concatenated with two original adjacent low-resolution view images to reconstruct the angular relation among original and new views. Then, a continuous fusion mechanism is proposed in the SIA module to obtain spatial SR information from four surrounding views and reconstruct the spatial-angular correlation in LF. Finally, super-resolved LF is generated by allocating the sub-pixel information in the up-sampling module. Furthermore, a combined loss is proposed to provide constraints on both angular feature extraction and spatial and angular synchronous SR, and train MFSRNet in an end-to-end fashion. On synthetic and real-world datasets, experimental results show that our algorithm outperforms other state-of-the-art methods in both visual and numerical evaluations. Especially, our method brings significant improvements for sparse LFs from the dataset STFgantry using MFSRNet. Our method improves PSNR/SSIM while preserving the inherent epipolar property in LF.