A semiconductor device includes a substrate, a semiconductor fin on the substrate, an isolation region on sidewalls of the semiconductor fin and having an upper surface lower than an upper surface of the semiconductor fin, a gate structure on a portion of the semiconductor fin and on a first portion of the isolation region. The portion of the semiconductor fin covered by the gate structure is referred to as a first region, and a second portion of the isolation region disposed on at least one of two opposite sides of the gate structure is referred to as a second region, which has an upper surface lower than an upper surface of the first region. The semiconductor device also includes a first spacer layer on a sidewall of the gate structure and on a sidewall of a portion of the first region disposed above the second region.