This application is directed to predicting vehicle turn and brake actions for at least partially autonomous vehicle driving. A first vehicle obtains a plurality of images along a road. For each image of the plurality of images, the first vehicle detects, from the image, a plurality of image regions each corresponding to a respective vehicle light of a second vehicle positioned on the road near the first vehicle. The first vehicle determines, for each image region, a probability that a respective vehicle light of the second vehicle changed its state. The first vehicle predicts a vehicle action of the second vehicle based on the probability for each image region. The first vehicle at least partially autonomously driving the first vehicle based on the predicted vehicle action of the second vehicle.