🤣LOLNeRF: Learn from One Look

Abstract

We present a method for learning a generative 3D model based on neural radiance fields, trained solely from data with only single views of each object. While generating realistic images is no longer a difficult task, producing the corresponding 3D structure such that they can be rendered from different views is non-trivial. We show that, unlike existing methods, one does not need multi-view data to achieve this goal. Specifically, we show that by reconstructing many images aligned to an approximate canonical pose with a single network conditioned on a shared latent space, you can learn a space of radiance fields that models shape and appearance for a class of objects. We demonstrate this by training models to reconstruct object categories using datasets that contain only one view of each subject without depth or geometry information. Our experiments show that we achieve state-of-the-art results in novel view synthesis and competitive results for monocular depth prediction.

Acknowledgements

We thank Matthew Brown, Kevin Swersky, David Fleet, and Viral Carpenter for their helpful insights and feedback. Also Danica Matthews for her assistance with feline data collection. This work was partly supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) and by Compute Canada.