A method, system, and computer readable storage medium are provided for predicting behavior of a vehicle. More specifically, image data of the vehicle is obtained by generating class prediction data for the vehicle and then identifying one or more related points of interest. The method, system, and computer readable storage medium then predict the behavior of the vehicle based upon the one or more points of interest and the class prediction data.