The disclosure provides an approach for estimating a state-space controller from a set of video frames depicting a motion of an entity. The approach includes incrementally optimizing parameters of the state-space controller and changing a structure of the state-space controller based on expanding subsets of the set of video frames. In one embodiment, a controller-estimation application greedily selects, at every stage of the incremental optimization, structure and parameters of the controller which minimize an objective function. In another embodiment, the controller-estimation application re-optimizes, after the incremental optimization, all parameters of the state-space controller based on all of the video frames. In yet a further embodiment, the controller-estimation application alters the structure of the state-space controller for robustness and compactness by adding cycles in the state-space controller and enforcing constraints on the structure of the state-space controller and adding and modifying state transition types, as appropriate.