Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for machine learning for home understanding and notification. In one aspect, a method includes collecting activity data from one or more devices in a home network, generating a graph-based knowledge base including two or more nodes based on the activity data, where each node is an object, person, or routine and wherein each link is a relationship between two nodes, generating, based on the graph-based knowledge base, one or more rules, determining, based in part on the activity data, an occurrence of a particular rule of the one or more rules, generating a notification responsive to the particular rule, presenting the notification to a user, receiving, user feedback responsive to the notification, where the user feedback includes a natural language label for the particular rule, and updating the particular rule based on the user feedback.