A semiconductor device includes a substrate having a fin projecting upwardly through an isolation structure over the substrate; a gate stack over the isolation structure and engaging the fin; and a gate spacer on a sidewall of the gate stack and in physical contact with the gate stack. The semiconductor device further includes a first dielectric layer vertically between the fin and the gate spacer. The semiconductor device further includes a second dielectric layer vertically between the first dielectric layer and the gate spacer, wherein the first and second dielectric layers include different materials, and wherein the second dielectric layer is in physical contact with the gate spacer and the first dielectric layer.