Techniques related to video background estimation inclusive of generating a final background picture absent foreground objects based on input video are discussed. Such techniques include generating first and second estimated background pictures using temporal and spatial background picture modeling, respectively, and fusing the first and second estimated background pictures based on first and second confidence maps corresponding to the first and second estimated background pictures to generate the final estimated background picture.