Reconstruction algorithm of invisible sides of a 3D object for depth scanning systems of a 3D object for cost effective truncation of point cloud data

With the advent of consumer grade depth-sensing cameras there exists an ever-increasing demand for real-time algorithms capable of reconstructing three-dimensional scenes. However, currently existing environmental 3D scanning solutions are still a very limited in regards of developing full 3D video streams of real-life vision. These solutions either require you to position the depth camera around areas of interest with pristine accuracy and lose a real-time transmission capability or you can continuously increase the amount of depth cameras positioned around the area of interest to in order for us to extract the entirety of point-cloud data for any given scene, which is an even more expensive and time consuming process. One of the solutions that might solve all or some of these issues and make the technology more accessible is the reconstruction of occluded regions. In our solution we offer an algorithm for an arbitrary 3D point-cloud/voxel reconstruction using a modular hybrid neural network architecture. Our modular hybrid neural network architecture aims to create an accurate representation of an object being captured by a depth sensor with as little overhead as possible in order to achieve real-time interactability with as little barrier of entry as possible. This would allow for holographic technologies to become more widespread in both commercial and home user applications. Furthermore, this solution would also provide numerous economic benefits. According to published market data for Hologram technologies by Technavio marketing researchers in 2018 in their Global Holographic Display Market 2018-2022 report, Holographic technology market is expected to reach market worth of 120 million US dollars by the end of 2023, which equals roughly 27.3% of growth per year every year, from the initial value of 29 million US dollars estimated in 2017. Therefore, our method would allow to substantially reduce the cost of producing holograms by removing the need of having multiple sensors in exchange for reduced quality in the reconstructed areas that have been occluded by other objects or caused by self-occlusion i.e. backside of a person. Additionally, our method also allows for reduced network costs due to greatly lowered bandwidth requirements when streaming data over the network as we are able to reconstruct the missing data. Thus, allowing to cut the bandwidth requirements by almost half. While naturally there already exist similar methods such as 3D-R2N2 that is capable of object reconstruction from a single colour image, these methods are limited by their reliance on environment lightning, while our method allows for almost complete indifference to lightning conditions due to the use of infrared depth sensors data for object reconstruction. This robustness against to lightning conditions allows us to use the system in low-light or even pitch-black environments. Therefore, our solutions have a wide variety of commercial applications such as virtual reality simulators, augmented reality, indoor mapping, digital forensics, security, etc. Our method consists of modular hybrid neural networks which consist of a classifier and n-modules of reconstruction networks each capable of reconstructing a specific subset of objects. This allows for much faster object training and expansion as the reconstruction modules are generally much more difficult and time consuming to train than image classifiers. Allowing for much greater flexibility for users to expand the viable reconstruction dataset. Our classifier consists of a neural network taking depth image as an input, whose received output is the class of the recognized object. Once the class is known we are able to pick the most appropriate neural network architecture for object reconstruction that was previously trained on similar types of objects thus giving us a 32x32x32 voxel cloud. However, displaying voxel cloud in real-time graphics is not ideal, therefore we convert it into triangular mesh with the help of marching cubes in addition to dual contouring of hermite data which gives us a smooth mesh that can be used for real-time graphics. Our method has shown to have high recall rates for object classifier, despite only using depth data as opposed to normally used fully RGB sensor data and a good reconstruction performance. The method has managed to achieve these results while also fulfilling one of the most important criteria - the ability to run in interactable framerates on both high-end and relatively low-end devices. Therefore, with our research we have managed to achieve our set out goal which allows us the users not only to reduce the initial setup costs when using holographic technology, but also to reduce the upkeep costs required to stream due to a significant reduction of data that is required to be transferred over the wire. The later giving the ability to stream on lower speed networks such as 3G/4G. With the additional benefit of modularity which allows for easy extension of the object set to be used in reconstruction with the downside that the reconstruction cannot account for all unknowns of the occluded object regions i.e. it's impossible to know the gesture that a persons hand is making when it's behind it's back without having an additional camera behind the person. And the additional downside of consumer grade depth sensors having relatively low quality of depth reconstruction which can cause speckling artefacts to appear and cause major defects in frames that cannot be accounted for.