Nvidia researchers propose ways to generate a rendering frame for 3D objects with 2D images

Media reported that computers are very good at calculating 3D models and putting them on 2D screens for display. But to push a 2D image back into a 3D model, the computer is a little out of the force. The good news is that Nvidia researchers have come up with a similar rendering framework, with the addition of machine learning technology. It gets 2D information through AI and accurately converts it to a 3D object — a system called DIB-R.

Nvidia researchers propose ways to generate a rendering frame for 3D objects with 2D images

(Instagram via TechSpot)

The full name of DIB-R is “renderer based on differential interpolation” and is built under the PyTorch Machine Learning Framework. At this week’s annual Neuroinformation Processing System Conference in Vancouver, The Nvidia team presented their latest developments.

The framework works almost in turn with the DAILY work of the GPU. It needs to analyze the 2D image and then form a highly fidelity 3D object, including shapes, textures, colors, and lighting.

The architecture of the codec starts with a variable sphere and deforms it using the information given in a 2D image. It’s worth noting that the process takes only 1/10 seconds.

Training with a single Nvita V100 GPU requires a 2-day training of the neural network. Training with other GPUs can take you weeks.

Nvidia researchers propose ways to generate a rendering frame for 3D objects with 2D images

After feeding it multiple datasets containing images of birds, DIB-R was able to accurately create a 3D model when giving a single image.

But jun Gao, co-author of the paper, says the system can also render any 2D image as a 3D model: “Actually, for the first time ever, you can shoot almost any 2D image and predict the relevant 3D properties.”

The researchers believe that the system can be used in the depth perception application of autonomous robots, thereby enhancing the safety and accuracy of their work in real-world environments. With this three-dimensional processing, robots are better able to navigate and manipulate the objects they need to handle.

Nvidia is known to have added DIB-R to its 3D deep learning PyTorch GitHub library (Kaolin) to help researchers accelerate 3D deep learning experiments.

Add a Comment

Your email address will not be published. Required fields are marked *