Computational cameras

I attended a fascinating talk this morning by Shree Nayar of Columbia University. The main point was that the combination of computers with cameras allows us to create devices that can do much more than a conventional camera could, which he demonstrated by describing

• Cameras fitted with anamorphic mirrors to take distorted very-wide-angle views which could then be un-distorted to 360-degree panoramas, or to take views that include multiple reflections of each object, allowing three-dimensional distance information to be reconstructed stereographically.
• A similar anamorphic approach to images reflected on the surfaces of people's eyes, allowing both the reconstruction of the image they're seeing and of their point of focus within that image.
• Single-image high dynamic range imaging without significant loss of spatial resolution by incorporating different levels of pixel-by-pixel neutral density filtering, similar in principle to Bayer filtering, on the sensor elements of a digital camera.
• Automated multi-image mosaicing similar to the polaroid mosaics of David Hockney.
• Cheap flexible arrays of many small video cameras to use with this mosaicing technique to create video streams with wide and highly dynamic perspectives.
• The use of computer projectors as programmable light sources to generate coarse single-image depth maps (by measuring the defocus of projected points of light) and then selectively blur the image to emphasize features at different depths.
• Combining multiple images with different high-resolution light patterns from a projector in order to recover different components of illumination of a scene. For instance, he showed a pink flower which, under direct illumination (light bouncing from a light source off the flower and back to the camera) looked more like gray paper; the pink coloring came from a combination of that lighting with indirect light paths by which light bounced around and through the petals picking up more of a red color with each bounce.

Overall, very inspiring. It led me to wonder how many other of the tools we use have been fixed in their form by pre-computer technology, and could now use a deep rethinking?

Also seen at the talk: one of our graphics grad students wearing a t-shirt with the mystic inscription: OBEY THE GEOMETRY. I approve.

Comments: