An object monitoring apparatus includes: an image receiver to receive at least one frame of captured images; an edge image generator to generate an edge image by detecting edges of objects appearing in the frame; a reference image generator to generate a reference image by detecting a part corresponding to a background in the frame to thereby define the detected part as a background edge; a candidate object extractor to extract one or more candidate object pixels by comparing the edge image with the reference image, and to extract a candidate object by grouping the extracted candidate object pixels into the candidate object; and an object-of-interest determiner to determine whether the candidate object is an object-of-interest based on a size of the candidate object and a duration time of detection of the candidate object.