This Neural Net Reconstructs the Missing Parts of Images

Saturday, April 28, 2018


⯀ NVIDIA researchers have developed a state-of-the-art deep learning method called "image inpainting" that can edit images or reconstruct a corrupted image, like one that has holes or is missing pixels. The method can also be used to edit images by removing content and filling in the resulting holes.


Researchers from NVIDIA, have introduced a state-of-the-art deep learning method that can edit images or reconstruct a corrupted image, one that has holes or is missing pixels.

The method can also be used to edit images by removing content and filling in the resulting holes.


Called “image inpainting”, this algorithm could be implemented in photo editing software to remove unwanted content; filling it with a realistic computer-generated alternative. In one example in the demo video below for instance, a bridge is easily and seamlessly removed from a landscape, and unwanted architectural features are easily blended away.

“Our model can robustly handle holes of any shape, size location, or distance from the image borders. Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing,” the NVIDIA researchers stated in their research paper. “Further, our model gracefully handles holes of increasing size.”


The team generated 55,116 masks of random streaks and holes of arbitrary shapes and sizes training the neural network. They also generated nearly 25,000 for testing. These were further categorized into six categories based on sizes relative to the input image, in order to improve reconstruction accuracy.

This Neural Net Reconstructs the Missing Parts of Images

The researchers claim existing deep learning based image inpainting methods suffer because the outputs for missing pixels necessarily depend on the value of the input that must be supplied to the neural network for those missing pixels. 

Related articles
This currently leads to image artifacts such as color discrepancy and blurriness in the images. To fix this problem, the NVIDIA team led by Guilin Liu developed a method that guarantees the output for missing pixels does not depend on the input value supplied for those pixels. 

Their method uses a “partial convolution” layer that renormalizes each output depending on the validity of its corresponding receptive field. This renormalization ensures that the value of the output is independent of the values of the missing pixels in each receptive field. 

Check out the video below to see the impressive results in action.




SOURCE  NVIDIA


By  33rd Square