Disclosed are various embodiments for a behavior-based identity system that recognizes and/or authenticates users based at least in part on determining stored behavioral events. For example, stored behavioral events may have been observed previously at a client or have been predefined by an authenticated user. Multiple behavioral events expressed by the client relative to a network site are recorded. The behavioral events may correspond to data that a user has elected to share, and the user may opt-in or opt-out of the behavior-based identity system. A comparison is performed between the multiple observed behavioral events and the stored behavioral events associated with a user identity. An inverse identity confidence score as to whether the user identity does not belong to a user at the client is generated based at least in part on the comparison.