The described techniques relate to modifying a trajectory of a vehicle, such as an autonomous vehicle, based on a latency associated with one or more systems of the vehicle. In examples, a planning system of the vehicle may predict a future latency (e.g., based on an interval between receipt of sensor data and/or object predictions), and use the future latency to determine a time at which to predict object behavior. Additionally, in some cases, the described techniques may include associating a predetermined acceleration with a predicted future location of the object to create a safety distance around the object, where the predetermined acceleration may be based on a maximum expected acceleration of the object. The safety distance may account for the object potentially accelerating in one or more directions at the future time.