Systems and methods are described for generating a three-dimensional track a ball in a gaming environment from a single camera. In some examples, an input video including frames of a ball moving in a gaming environment recorded by a camera may be obtained, along with a camera projection matrix associated with at least one frame that maps a two-dimensional pixel space representation to a three-dimensional representation of the gaming environment. Candidate two-dimensional image locations of the ball across the plurality of frames may be identified using a neural network or a computer vision algorithm. An optimization algorithm may be performed that uses a 3D ball physics model, the camera projection matrix and a subset of the candidate two-dimensional image locations of the ball to generate a three-dimensional track of the ball in the gaming environment. The three-dimensional track of the ball may then be provided to a user device.