Driver behavior recognition or driver behavior prediction are described herein. A first image sequence including image frames associated with a forward-facing image capture device of a vehicle and a corresponding vehicle data signal sequence may be received. A second image sequence including image frames associated with a rear or driver facing image capture device of the vehicle may be received. Feature vectors may be generated for respective sequences using neural networks, such as a convolutional neural network (CNN), a depth CNN, a recurrent neural network (RNN), a fully connected layer, a long short term memory (LSTM) layer, etc. A fusion feature may be generated by performing data fusion on any combination of the feature vectors. A predicted driver behavior may be generated based on the LSTM layer and n image frames on an image sequence and include x number of prediction frames.