Systems and methods for scalable gesture and eye-gaze tracking in virtual, augmented, and mixed reality (xR) applications are described. In some embodiments, an Information Handling System (IHS) may include a processor and a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution, cause the IHS to: monitor utilization of an IHS resource; capture an image of a gesture performed by a user wearing a Head-Mounted Device (HMD) during an xR application; and generate a feature vector usable to identify the gesture based upon the image, wherein the feature vector has a maximum number of features selected based upon the IHS resource utilization.