A system and method detects matches between portions of video content. A matching module receives an input video fingerprint representing an input video and a set of reference fingerprints representing reference videos in a reference database. The matching module compares the reference fingerprints and input fingerprints to generate a list of candidate segments from the reference video set. Each candidate segment comprises a time-localized portion of a reference video that potentially matches the input video. A classifier is applied to each of the candidate segments to classify the segment as a matching segment or a non-matching segment. A result is then outputted identifying a matching portion of a reference video from the reference video set based on the segments classified as matches.