Aspects of the technology described herein use acoustic features of a music track to capture information for a recommendation system. The recommendation can work without analyzing label data (e.g., genre, artist) or usage data for a track. For each audio track, a descriptor is generated that can be used to compare the track to other tracks. The comparisons between track descriptors result in a similarity measure that can be used to make a recommendation. In this process, the audio descriptors are used directly to form a track-to-track similarity measure between tracks. By measuring the similarity between a track that a user is known to like and an unknown track, a decision can be made whether to recommend the unknown track to the user.