Techniques are provided to infer a context associated with a mobile device based on aggregated data from a set of other mobile devices. The set of mobile devices can include mobile devices currently or previously near a location of the mobile device. Each mobile device in the set of other mobile devices can collect sensor data and infer a low-level context (e.g., “sitting” or “standing”). The low-level contexts can be aggregated. Based on the aggregated low-level contexts, a high-level context (e.g., “at a party” or “watching television”) associated with the mobile device can be inferred or a low-level context associated with the mobile device can be refined.