Techniques related to training and implementing a bidirectional pairing architecture for object detection are discussed. Such techniques include generating a first enhanced feature map for each frame of a video sequence by processing the frames in a first direction, generating a second enhanced feature map for frame by processing the frames in a second direction opposite the first, and determining object detection information for each frame using the first and second enhanced feature map for the frame.