Provided are a computer program product, system, and method for adding conversation context from detected audio to contact records. An audio stream is received through a microphone of a personal computing device. Context analysis is performed of the audio stream to detect conversation context in the audio stream. A request is received to associate the conversation context with a selected contact person. A determination is made of conversation context detected from a segment of the audio stream received during a period of time from when the request was received. The conversation context determined for the period of time is included in a contact record for the selected contact person. The contact record is stored in a contact database including a plurality of contact records for a plurality of contact persons, wherein the contact records include conversation context and location information associated with the contact persons.