M.Sc. Dissertation Proposal

Vision and Motivation

Surface reconstruction is a challenging problem in computer graphics with a wide range of applications in computer-aided design, reverse-engineering, prototyping, molecular biology, medical imaging, visualization, and computer vision [2017-Boltcheva], largely because a growing variety of scanning devices (e.g., optical laser-based range scanners, structured light scanners, LiDAR scanners, and commodity real-time scanners as Kinect) are available nowadays to acquire point sets from single 3D objects and massive 3D scenes (e.g., urban scenes). As the diversity, ease of use, and popularity of 3D acquisition methods continue to increase, so does the need for the development of new surface reconstruction techniques, particularly those supported by deep learning.

State-of-the-Art

Surface reconstruction aims to reconstruct a given surface from its samples; these samples constitute a point set previously acquired by a 3D scanner [2013-Kazhdan]. While surface reconstruction algorithms have matured for the last three decades, their fundamental issues remain, despite the surge of geometric deep learning methods to reconstruct surfaces from point clouds [2020-Cao] [2020-Hanocka] [2023-Ge] [2023-Yang]. The main remaining issues are the following: noise sensitivity, shape incompletion, deterioration of sharp features, shape drifting, and ill-posedness. Noise sensitivity can be tackled using deep learning, but it is much difficult to solve this issue using classical methods. Shape incompletion can be also solved through deep learning in the latent space, but there is no classical method to approach it. Sharp features are difficult to preserve using both classical and neural methods because sharp features are singularities; that is, the derivatives behave badly at them. Recall that neural networks are not designed to deal with derivatives that behave badly. Shape drifting can be solved using classical methods, but its resolution has never been accomplished properly. At last, the surface reconstruction is an ill-posed problem because there are an infinite number of solutions for the same input point set (or point cloud). This problem can be a priori solved by reformulating it in terms of shape classes in the latent space. In fact, given the increasing scale of acquired data, we no longer deal exclusively with individual shapes, but with classes of shapes, possibly at the scale of cities with many objects defined as structured shapes [2019-Lan]. Recovering the structure of such large-scale scenes, like the one shown in Fig. 2, is a stimulating scientific challenge because it is mandatory to distinguish among distinct objects, i.e., we a priori need a specific sort of object segmentation for point clouds.

Research Methodology

To deal with combinatorial surface reconstruction of scenes with multiple objects, we need to develop new neural reconstruction algorithms (and segmentation algorithms), i.e., new combinatorial reconstruction methods that combine geometric data and deep learning. More specifically, we intend to investigate how to learn classes of shapes that enable high-quality shape representation, interpolation and completion from partial and noisy 3D input data. Recall that classical methods only model a single shape rather than a shape dataset.

References

[2013-Kazhdan] Kazhdan, M. and Hoppe, H. (2013): Screened Poisson Surface Reconstruction. ACM Transactions on Graphics 32(3) Art.29:1-13.
https://doi.org/10.1145/2487228.2487237
[2017- Boltcheva] Boltcheva, D. and Lévy, B. (2017): Surface reconstruction by computing restricted Voronoi cells in parallel. Computer-Aided Design 90(2017):123-134.
https://doi.org/10.1016/j.cad.2017.05.011
[2017-Hackel] Hackel, T., Savinov, N., Ladicky, L., Wegner, J., Schindler, K. and Pollefeys, M. (2017): SEMANTIC3D.NET: A new large-scale point cloud classification benchmark. In Proceedings of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. IV-1-W1, pp.91-98.
https://doi.org/10.5194/isprs-annals-IV-1-W1-91-2017
[2019-Lan] Lan, Z., Yie, Z. and Lee, G. (2019): Robust Point Cloud Based Reconstruction of Large-Scale Outdoor Scenes. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19), IEEE Press.
https://doi.org/10.1109/CVPR.2019.00992
[2020-Cao] Cao, W., Yan, Z., He, Z. and He, Z. (2020): A Comprehensive Survey on Geometric Deep Learning. IEEE Access 8(2020):35929-35949.
https://doi.org/10.1109/ACCESS.2020.2975067
[2020-Hanocka] Hanocka, R., Metzer, G., Giryes, R., and Cohen-Or, D. (2020): Point2Mesh: a self-prior for deformable meshes. ACM Transactions on Graphics 39(4) Art.126:1-12.
https://doi.org/10.1145/3386569.3392415
[2023-Ge] Ge, M., Yao, J., Yang, B., Wang, N., Chen, Z., and Guo, X. (2023): Point2MM: Learning medial mesh from point clouds. Computers & Graphics 115(2023):511-521.
https://doi.org/10.1016/j.cag.2023.07.020
[2023-Yang] Yang, L., Yang, C., Xie, R., Liu, J., Zhang, H., and Tan, W. (2023): 3D Reconstruction from Traditional Methods to Deep Learning. In Proceedings of the 2023 IEEE 10th International Conference on Cyber Security and Cloud Computing (CSCloud’23)/2023 IEEE 9th International Conference on Edge Computing and Scalable Cloud (EdgeCom’23), Xiangtan, Hunan, China, IEEE Press, pp.387-392.
https://doi.org/10.1109/CSCloud-EdgeCom58631.2023.00072