A moving object detected and tracked within a field of view environment of a 2D data feed of a calibrated video camera is represented by a 3D model through localizing a centroid of the object and determining an intersection with a ground-plane within the field of view environment. An appropriate 3D mesh-based volumetric model for the object is initialized by using a back-projection of a corresponding 2D image as a function of the centroid and the determined ground-plane intersection. Nonlinear dynamics of a tracked motion path of the object are represented as a collection of different local linear models. A texture of the object is projected onto the 3D model, and 2D tracks of the object are upgraded to 3D motion to drive the 3D model by learning a weighted combination of the different local linear models that minimizes an image re-projection error of model movement.