Systems and methods for providing a real-world object interface in virtual, augmented, and mixed reality (xR) applications. In some embodiments, an Information Handling System (IHS) may include one or more processors and a memory coupled to the one or more processors, the memory including program instructions stored thereon that, upon execution by the one or more processors, cause the IHS to: receive a video frame during execution of an xR application; instruct a user wearing a Head-Mounted Display (HMD) to perform a manipulation of a real-world object detected in the video frame; receive additional video frames; determine whether the user has performed the manipulation by tracking the object in the additional video frames; and execute an operation in response to the determination.