A method and system tracks models and predict object movement in an environment. Sequences of temporally and spatially adjacent events sensed by the set of sensors are linked to form a set of tracklets. Each tracklet has an associated starting and terminating location. The tracklets are used to construct a directed graph including starting nodes, terminating nodes, and, intermediate nodes connected by edges. The intermediate nodes can be split nodes where tracklets diverge onto different tracks, and join nodes where multiple tracklets converge onto a single path. Probabilities are assigned to the edges to model and predict movement of the objects in the environment.