In an embodiment, a system provides object tracking and 6D pose estimations to a robot that performs different tasks such as manipulation and navigation. In an embodiment the 6D object pose is determined using a Rao-Blackwellized particle filtering framework, where the 3-D rotation and the 3-D translation of the object is decoupled. In an embodiment, the system provides the 3-D translation of an object along with a full distribution over the 3-D rotation. In an embodiment, the 3-D rotation is determined by discretizing the rotation space, and training an autoencoder network to construct a codebook of feature embeddings for the discretized rotations. In an embodiment, the system is able to track objects with arbitrary symmetries while also maintaining adequate posterior distributions.