Techniques for passive enrollment of a user in a speaker identification (ID) device are provided. One technique includes: parsing, by a processor of the speaker ID device, a speech sample, spoken by the user, into a keyword phrase sample and a command phrase sample; identifying, by a text-dependent speaker ID circuit of the speaker ID device, the user as the speaker of the keyword phrase sample; associating the command phrase sample with the identified user; determining if the command phrase sample in conjunction with one or more earlier command phrase samples associated with the user is sufficient command phrase sampling to enroll the user in a text-independent speaker ID circuit of the speaker ID device; and enrolling the user in the text-independent speaker ID circuit using the command phrase samples associated with the user after determining there is sufficient command phrase sampling to enroll the user.