In some embodiments, a method receives a history of videos that were viewed on a video delivery system as a first sentence in a sequential order and a target video as a second sentence as input to a prediction network. The prediction network analyzes representations for the history of videos and a representation of the target video. The prediction network generates a session representation based on bidirectionally analyzing a sequence of the first representations and the second representation. The method uses the session representation to determine whether to recommend the target video.