A motion vector detection device includes: a separation section that separates a video signal into a brightness component and a color-difference component; a selection section that selects either the brightness component or the color-difference component, or both; a gain controlling section that controls, if the selected brightness or/and color-difference components exceed a predetermined reference level, its/their gains; and a motion vector detection section that detects a motion vector based on the result of gain control by the gain controlling section.