The disclosed computer-implemented method for offloading image-based tracking operations from a general processing unit to a hardware accelerator unit may include (1) sending imaging data from an imaging device to a hardware accelerator unit, and (2) directing the hardware accelerator unit to generate a multi-scale representation of the imaging data sent from the imaging device, (3) preparing a set of input data for a set of image-based tracking operations, and (4) directing the hardware accelerator unit to execute the set of image-based tracking operations using the generated multi-scale representation of the imaging data and the prepared set of input data. Various other methods, systems, and computer-readable media are also disclosed.