Presented herein are techniques for training a hearing prosthesis to classify/categorize received sound signals as either including a recipient's own voice (i.e., the voice or speech of the recipient of the hearing prosthesis) or external voice (i.e., the voice or speech of one or more persons other than the recipient). The techniques presented herein use the captured voice (speech) of the recipient to train the hearing prosthesis to perform the classification of the sound signals as including the recipient's own voice or external voice.