Method and system for tracking moving objects in a video with non-stationary background is provided. The method comprises estimation of non-stationary background and determining an approximate foreground for each image that learns the background model over time by incorporating various constraints on image pixels. Then, weights based upon spatio-temporal statistical properties of the extracted foreground blobs and blob edge overlap are used to identify and track with bounding boxes displayed for one or more true objects.