The semiconductor device includes a substrate, an epi-layer, a first etch stop layer, an interlayer dielectric (ILD) layer, a second etch stop layer, a protective layer, a liner, a silicide cap and a contact plug. The substrate has a first portion and a second portion. The epi-layer is disposed in the first portion. The first etch stop layer is disposed on the second portion. The ILD layer is disposed on the first etch stop layer. The second etch stop layer is disposed on the ILD layer, in which the first etch stop layer, the ILD layer and the second etch stop layer form a sidewall surrounding the first portion. The protective layer is disposed on the sidewall. The liner is disposed on the protective layer. The silicide cap is disposed on the epi-layer. The contact plug is disposed on the silicide cap and surrounded by the liner.