Typically, to create a lifelike 3D digital model of a face, you need to use a 3D scanner or multiple cameras. But a team of researchers at Carnegie Mellon University in Pittsburgh has created a new technology that requires only one smartphone to create realistic 3D facial models based on video. It is understood that the technology was developed by Assoc and condenses the work of Professor Simon Lucey and master’s students Shubham Agrawal and Anuj Pahuja.
3D modeling vs. original video (pictured from Carnegie Mellon University)
First, the researchers used their smartphones to capture short video skits of 15 to 20 seconds around the object’s head. To collect as much data as possible (number of image frames), the Lucey team also used the slow photography capabilities of the iPhone X.
It then uses an existing technique called visual simultaneous positioning and mapping (SLAM) to determine the basic geometry of the face. It triangulats the points on the face characterization to calculate the corresponding shape and the distance of the phone relative to the face.
The deep learning-based algorithm can then be used to identify the facial contours of objects, as well as the relative position of “landmarks” such as the eyes, ears, noses and noses. Traditional “grid fit” computer vision techniques are also needed to fill in other “blank” data.
Although the entire process takes 30 to 40 minutes, its biggest feature is that it can be done simply by relying on a smartphone. The 3D model not only looks realistic, but also has submillimeter scale, a significant improvement over similar devices previously developed.
Finally, the team hopes the technology will be used in a variety of areas, such as the creation of player avatars in games, medical, or biometric synods.
Details of the team’s work were presented at the IEEE Winter Computer Vision Applicationconference in Colorado earlier this month.