Application of the ResNet50 Neural Network for RGB and IR image fusion

The aim of this study is to develop a method for fusing image data from a daylight (RGB) camera and a thermal (IR) camera using deep learning with a ResNet-50 network. The research defines the conditions for fusing RGB and IR images in terms of the required level of resolution alignment between RGB and IR images, and resistance to scaling and rotation to ensure the desired/required object recognition accuracy. The image fusion is carried out using a data-level fusion approach by appropriately aligning image resolutions to achieve the desired accuracy, followed by their combination into an extended input representation. The prepared data is then used to train a ResNet-50 convolutional neural network on an annotated image dataset. The task of the network is to classify objects present in the images. Experiments conducted on datasets containing both RGB and IR images demonstrated that combining information from both sensors significantly improves the model’s performance compared to networks trained solely on a single type of input. The results of field experiments confirm that image fusion combined with the ResNet-50 architecture is an effective approach for vision tasks performed under varying lighting conditions and reduced visibility. Fusion procedure: https://home.mycloud.com/action/share/d14c53ac-9ce6-4001-897c-89336d08acdf.