Disclosed are various embodiments that perform confidence-based authentication of a user. An identification of a user account is obtained from a user, and a minimum confidence threshold is determined. Multiple authentication challenges are presented to the user. Responses are obtained from the user to a subset of the challenges, with each response having a corresponding authentication point value. A confidence score is generated for the user, where the confidence score is increased by the respective authentication point values of the correct responses. The user is authenticated as being associated with the user account in response to determining that the confidence score meets the minimum confidence threshold.