Provided are a system and method of detecting moving objects. The system stores pixel information regarding each of pixels included in frames of the video in a storage, sets a background model comprising at least one background model element, the at least one background model element indicating at least one of a reference visual property and a reference usage time, determines whether the pixels are background pixels or foreground pixels by comparing the pixel information with the at least one background model element, and updating the background model based on a result of the comparing.