Images of an environment that are captured from two or more imaging devices may be captured and evaluated in order to identify a state of the environment, or an interaction that placed the environment in the state. The content of the images may be analyzed in order to recognize observed information or data expressed therein. The information or data may be associated with a given state according to one or more observation functions, and the state may be used to identify an action according to one or more transition functions. The observation function uses conditional probabilities to transfer the probability of making an observation by one imaging device to the observation made by the other imaging device. The observation functions and the transition functions may be derived based on historical training data including clips that are labeled to identify states or interactions expressed therein.