A method for using structured light in a handheld projection device is provided that includes projecting a structured light pattern in at least one portion of a frame being projected by the handheld projection device, wherein the at least one portion of the frame is a subset of the frame, capturing an image of the projected frame, computing scene depth information based on the structured light pattern in the captured image, and using the scene depth information in processing of a subsequent frame of the video stream.