A method of fabricating a semiconductor device includes the following steps: providing a semiconductor substrate having a fin structure thereon; forming a recess in the fin structure so that the semiconductor substrate is partially exposed from the bottom surface of the recess; forming a dopant source layer conformally disposed on side surfaces and a bottom surface of the recess; removing the dopant source layer disposed on the bottom surface of the recess until portions of the semiconductor substrate are exposed from the bottom surface of the recess; and annealing the dopant source layer so as to form a side doped region in the fin structure.