Particular embodiments provide a system to determine ad segments in a video asset to enable subsequent ad replacement in video programs. The system is included in a multiple service operator (MSO) system that broadcasts video programs via a broadcast schedule. The MSO may not know the location of the ad segments in the video asset. To determine the ad segments, the MSO uses a classifier to classify video program segments and advertisements in the video asset. The classifier may be integrated with an nDVR system. By integrating with the nDVR system, particular embodiments may determine user behavior information, such as trick play commands, from the nDVR system. The classifier may use the user behavior information to detect ad segments in the video asset. In one embodiment, the classifier may fuse outputs from different detectors to detect and validate ad segments in the video program.