A tunable representation for tracking that simultaneously encodes appearance and geometry in a manner that enables the use of mean-shift iterations for tracking is provided. The solution to the tracking problem is articulated into a method that encodes the spatial configuration of features along with their density and yet retains robustness to spatial deformations and feature density variations. The method of encoding of spatial configuration is provided using a set of kernels whose parameters can be optimized for a given class of objects off-line. The method enables the use of mean-shift iterations and runs in real-time. Better tracking results by the novel tracking method as compared to the original mean-shift tracker are demonstrated.