A social photo curation system is used to automatically identify a subset of photos for an album to provide to a viewing user. The album and its photos are associated with metadata indicating information about the photos, such as individuals tagged in the photos, locations where the photos were taken, keywords or concepts associated with the photos, and the quality and variety of the photos. The social photo curation system uses this metadata to score and select the photos for a particular viewing user. The scoring and selection of photos for the album may be independent of the viewing user, or it may be customized based on the viewing user's interests and connections to other users in a social networking system.