An image processing device has an image input part to which a frame image of an imaging area taken with an infrared camera is input, a background model storage part in which a background model is stored with respect to each pixel of the frame image input to the image input part, a frequency of a pixel value of the pixel being modeled in the background model, a background difference image generator that determines whether each pixel of the frame image input to the image input part is a foreground pixel or a background pixel using the background model of the pixel, which is stored in the background model storage part, and generates a background difference image, and a threshold setting part.