New generative AI technique allows for better photo editing—in 3D

Clay, glass, wood, and marble are among the materials that can be sculpted, turning ordinary, three-dimensional objects into works of art. Researchers at NYU’s Courant Institute of Mathematical Sciences are working to bring this approach to images—through a pioneering form of sculpture that transforms 2D pictures into 3D.

Saining Xie and Daniele Panozzo, professors in Courant’s Department of Computer Science, have created a prototype technology that manipulates geometry and deploys generative AI to allow users to see not only an image’s front or back, but also its sides, top, and bottom—and to move it around based on one’s preferences. For instance, let’s say you wish to add a picture of cherries to the top of a cake image, by turning a 2D image of cherries (and cake) into 3D, you can more accurately place the cherries because the extra dimension affords a better match to reality.