A method may include obtaining input information relating to an environment in which an autonomous vehicle (AV) operates in which the input information describes at least one of: a state of the AV, an operation of the AV within the environment, a property of the environment, or an object included in the environment. The method may include identifying a region of interest that represents a section of the environment based on the input information and identifying a portion of the environment based on the identified region of interest. The portion of the environment may include an object that affects operation of the AV. The method may include determining a first decision that relates to the object and sending an instruction to a control system of the AV describing a given operation of the AV responsive to the object according to the first decision.