A head-mounted device may have a gaze tracking system and an optical sensor. The gaze tracking system may include a camera, ambient light sensor, or other optical sensor to determine an angle/direction of a user's gaze. The optical sensor may be a camera, directional color ambient light sensor, or other optical sensor. After determining the user's gaze, the optical sensor may be used to determine the color of an external object in the direction of the determined angle/direction. For example, the color of the external object may be determined in response to an input from the user, such as a button press, a voice command, or a blink. The determined color may be used as an input color in a virtual space or on another device.