A cut detection system includes: an extractor configured to extract macroblock type information at least for every B frame from video compressed data including the macroblock type information; a calculator configured to calculate the number of intra-coded macroblocks for every frame from the extracted macroblock type information; and a detector configured to detect the B frame or a group of picture including the B frame to be a replay cut portion, which is a head portion or last portion of a replay shot, by using a frequency characteristic of the intra-coded macroblocks included in the B frame as determination conditions.