Techniques are described for estimating intentions of pedestrians and other road users in vicinity of a vehicle. In certain embodiments, the techniques comprise obtaining, by a computer system of a vehicle equipped with one or more sensors, a sequence of video frames corresponding to a scene external to the vehicle, detecting one or more vulnerable road users (VRUs) in the sequence of video frames, wherein the detecting comprises estimating pose of each of the detected one or more VRUs. The techniques further include generating a segmentation map of the scene using one or more of the video frames; estimating one or more intention probabilities using estimated pose of the one or more VRUs and the segmentation map, each intention probability corresponding to one of the detected one or more VRUs, and adjusting one or more automated driving actions based on the estimated one or more intention probabilities.