Implementations relate to an automated assistant that can determine whether to respond to inputs in an environment according to whether radar data indicates a user is present. When user presence is detected, the automated assistant can virtually segment the environment and apply certain operational parameters to certain segments of the environment. For instance, the automated assistant can enable an input detection feature, such as warm word detection, for a segmented portion of the environment in which a user is detected. In this way, false positives can be mitigated for instances in which environmental and/or user sounds are detected by the automated assistant but do not originate from a particular segment of the environment. Other parameters, such as varying confidence thresholds and/or speech processing biasing, can be temporarily enforced for different segments of an environment in which a user is detected.