A passive authentication method includes, in response to receiving a requested action from a first user, obtaining a set of sensor data and categorizing first sensor data of the set of sensor data into a first modality of a set of modalities. The method includes, for the first modality of the set of modalities, determining a distance value by applying a first modality model to the first sensor data and comparing the distance value to a first verified value of the first user for the first modality. The method includes, based on the comparison, determining a first authentication decision of the distance value. The method includes, in response to the first authentication decision indicating the first sensor data corresponds to the first user, performing the requested action.