A method of determining a trajectory for an autonomous vehicle is disclosed. An ego-vehicle may detect a moving actor in an environment. To choose between candidate trajectories for the ego-vehicle, the system will consider the cost of each candidate trajectory to the moving actor. The system will use the candidate trajectory costs for the candidate trajectories to select one of the candidate trajectories via which to move the ego-vehicle. An autonomous vehicle system of the ego-vehicle may then move the ego-vehicle in the environment along the selected trajectory.