A difference detection device includes a difference detection unit configured to, based on association among a first image and a second image captured at different times and illustrating a substantially identical space and encoding information of each of the first image and the second image, detect difference between a third image and a fourth image captured at different times and illustrating a substantially identical space, and the encoding information is information acquired from data including the first image encoded and data including the second image encoded, before inverse transform processing is executed in decoding processing executed on each of the first image and the second image.