StyleNeRF-Med: A hybrid neural framework for robust 3D image reconstruction from sparse 2D views

Reconstructing high-quality 3D images from fewer input images is a fundamental challenge in computer vision and medical imaging, which has significant implications for data-efficient artificial intelligence. Traditional methods such as computed tomography and magnetic resonance imaging typically require multiple images for accurate image reconstruction, making them costly and data-intensive. Recent advances in the field of neural radiance fields (NeRF) have shown strong performance in the development of novel views. However, such methods are highly dependent on precise camera settings and scene-specific optimization, which limits their generalizability and usefulness. To overcome these limitations, we leverage advanced AI techniques, such as Generative Adversarial Networks (GANs) for generating realistic images. We propose StyleNeRF-Med, a hybrid framework that innovatively integrates StyleGAN2 and NeRF to generate high-quality 3D reconstructions from fewer input images. StyleGAN2 encodes rich image features to synthesize detailed views, while NeRF processes 3D spatial coordinates and camera positions to compute accurate pixel colors and intensities. A synthetically generated dataset of digitally reconstructed radiographs (DRR) is used to validate the model. StyleNeRF-Med demonstrates superior performance in reconstructing high-quality 3D medical images from limited 2D input. On the chest dataset, it achieves the lowest FID score of 101.24 and KID score of 0.072, outperforming GRAF and PixelNeRF. Similarly, on the knee dataset, StyleNeRF-Med again shows better image quality with the lowest FID score of 132.64 and the KID score of 0.098. These results indicate improved alignment with real data distributions and improved reconstruction quality over existing methods.