3D Scene Reconstruction from Comic Books through Deep Learning
2024, Apr. 27
Vision and Motivation
3D scene reconstruction from images is a fundamental and popular research topic in computer vision and graphics [2010-Moons]. However, 3D scene reconstruction algorithms and frameworks have been only developed for realistic scenes (i.e., photos and 3D range scanners). At our best knowledge, 3D scene reconstruction from 2D non-photorealistic scenes like those of comic books (e.g., Spider Man book series in Fig. 3) has been never tried before. The ultimate objective behind this challenge is to build and animate a 3D comic movie from an entire comic book.
State-of-the-Art
There is nothing relevant to mention in the literature about 2D-to-3D reconstruction for comics, unless a very few references related to the opposite pipeline, i.e., generating 2D comics from 3D realistic scenes; see, for example, [2006-Shamir], [2016-Correia], and [2018-Nguyen].
Research Methodology
Starting from the in-house knowledge concerning image analysis and processing of comics, we intend to design and build up a 2D-to-3D comics pipeline from the scratch. For that purpose, we will take advantage of well-known reconstruction techniques in computer vision and graphics, as well as multidimensional scaling techniques (i.e., 3D character pose prediction in shape space given 2D poses in comics), pose interpolation in shape space (latent space) for animation, and deep learning to identify characters independently of their pose in comics books. If we succeed in this challenge, we certainly will revolutionize the production of animated 3D movies.
References
[2006-Shamir] Shamir, A., Rubinstein, M., and Levinboim, T. (2006): Generating Comics from 3D Interactive Computer Graphics. IEEE Computer Graphics & Applications 26(4): 53-61.
https://doi.org/10.1109/MCG.2006.58
[2010-Moons] Moons, T., Van Gool, L., and Vergauwen, M. (2010): 3D Reconstruction from Multiple Images Part 1: Principles. Foundations & Trends® in Computer Graphics & Vision 4(4): 287-404.
https://doi.org/10.1561/0600000007
[2016-Correia] Correia, J. and Gomes, A. (2016): Balloon Extraction from Complex Comic Books using Edge Detection and Histogram Scoring. Multimedia Tools and Applications 75(8): 11367-11390.
https://doi.org/10.1007/s11042-015-2858-0
[2018-Nguyen] Nguyen, N.-V., Rigaud, C., and Burie,J.-C. (2018): Digital Comics Image Indexing Based on Deep Learning. Journal of Imaging, 4(7) Art.89:1-34.
https://doi.org/10.3390/jimaging4070089
