A method and system of performing interactive object segmentation from streaming surfaces is disclosed. An environment data stream, including correlated image and depth data, is received from a set of sensors collocated with a robot. A virtualized representation of a physical environment is displayed and updated in accordance with the environment data stream in real-time. A marking input is received from a haptic-enabled input device. A position in the virtualized representation of the physical environment is determined in accordance with the marking input and is constrained by a first virtualized surface in the virtualized representation of the physical environment. Object segmentation is performed from the position of the marking input on the correlated image and depth data.