An example method includes determining objects and actions associated with the objects for completing a task to be executed by a robotic system, where each action is associated with trajectory. The method further includes determining a pose for each person in an environment associated with the robotic system, predicting a trajectory for each person based on the determined pose associated with the respective person and the actions and trajectories associated with the actions, and adjusting trajectories for one or more of the actions to be performed by the robotic system based on the predicted trajectories for each person.