A method for searching video includes receiving a video query, the video query including a first video query term identifying a first foreground object, a second video query term identifying a second foreground object, and a third video query term identifying a spatiotemporal relationship; searching the video in response to the video query for the first foreground object and the second foreground object within the spatiotemporal relationship; and generating a search result in response to the searching.