A system configured to determine a six-degree of freedom pose of a physical object in a physical environment and to utilize the six-degree of freedom pose as an within a virtual environment or mixed reality environment. In some cases, the system may utilize one or more cameras on a headset device to track the pose of a controller or other objects and one or more cameras on the controller itself to track the pose of the headset device or the user. In one example, the system may capture image data of a physical object having a constellation or pattern on the external source. The system may analyze the image data to identify image points associated with the constellation or pattern and to determine the pose of the object based on a location of the points in the image.