An element larger than silicon is ion-implanted to a contact liner in an N-channel region to break constituent atoms of the contact liner in the N-channel region. An element larger than silicon is ion-implanted to the contact liner in a P-channel region to break constituent atoms of the contact liner, oxygen or the like is ion-implanted. Thereafter, heat treatment is performed to cause shrinkage of the contact liner in the N-channel region to form an n-channel contact liner, and to cause expansion of the contact liner in the P-channel region to form a p-channel contact liner.