A semiconductor device includes a substrate, a pair of semiconductor fins, a dummy fin structure, a gate structure, a plurality of source/drain structures, a crystalline hard mask layer, and an amorphous hard mask layer. The pair of semiconductor fins extend upwardly from the substrate. The dummy fin structure extends upwardly above the substrate and is laterally between the pair of semiconductor fins. The gate structure extends across the pair of semiconductor fins and the dummy fin structure. The source/drain structures are above the pair of semiconductor fins and on either side of the gate structure. The crystalline hard mask layer extends upwardly from the dummy fin and has an U-shaped cross section. The amorphous hard mask layer is in the first hard mask layer, wherein the amorphous hard mask layer having an U-shaped cross section conformal to the U-shaped cross section of the crystalline hard mask layer.