A management of user profiles comprises calculating, for each speaker model of at least one speaker model, a confidence measure representing a probability that the speaker model represents a speaker of a cluster of audio segments. A user profile associated with the speaker model is updated based on a user preference assigned to the cluster of audio segments if the confidence measure calculated for the speaker model represents a probability that is higher than a target probability. The embodiments achieve an efficient user profile management in a voice-controlled context but without the need for any dedicated enrollment sessions to train speaker models.