Systems and methods for providing geometric interactions via three-dimensional mapping. A method includes determining a plurality of first descriptors for a plurality of key points in a plurality of first images, wherein each first image shows a portion of a 3D environment in which a robotic device and a visual sensor are deployed; generating a 3D map of the 3D environment based on the plurality of key points and the plurality of descriptors; determining a pose of the visual sensor based on at least one second descriptor and the plurality of first descriptors, wherein the second image is captured by the visual sensor; and determining a target action location based on at least one user input made with respect to a display of the second image and the pose of the visual sensor, wherein the target action location is a location within the 3D environment.