A virtual user interface in a 3D environment is projected onto a display object, such as an open hand. An operating object, such as a stylus or a second hand with an index finger extended, may be used to select input features of the projected user interface. In one scenario, for example, a telephone keypad may be projected onto the open hand and the index finger may be used to select a particular number on the keypad. Images of the hands may be segmented to produce contours of the finger and open hand, and these contours are used to determine the location of the finger relative to the open hand.