A resolution enhancing method of a video includes reducing a training video, extracting a high-frequency component from the training video, calculating a first feature vector including a feature amount of a first spatio-temporal box in the reduced video, storing pairs of the first feature vectors and second spatio-temporal boxes in the high-frequency component videos at the same positions as those of the first spatio-temporal boxes, expanding an input video, retrieving a first feature vector similar to a second feature vector including a feature amount of a third spatio-temporal box of an object of the input video to be processed, as an element, and adding a second spatio-temporal box making a pair with the retrieved first feature vector to a fourth spatio-temporal box in the expanded video at the same position as that of the third spatio-temporal box in order to generate an output video.