A system for physical-virtual environment fusion includes a sensor with a computing system and a memory in communication with the computing system, the memory storing a plurality of endpoints. The computing system is configured to select a semantic identity of an object from among a group of objects indicated by a user pointer indicator by determining, based on inputs from the sensor, the approximate orientation of the user pointer indicator towards a first endpoint from among the plurality of endpoints, the first endpoint having the group of objects located within the first endpoint. It further renders, on a display surface, the group of objects, receives from the user an indication regarding the semantic identity of the object and presents, on the display surface, an indication of the selection of the object from among the group of objects based on the further indications from the user.