A semiconductor device and method of forming the same are disclosed. The semiconductor device includes a fin structure, a gate electrode, a source-drain region, a plug and a hard mask structure. The gate electrode crosses over the fin structure. The source-drain region in the fin structure is aside the gate electrode. The plug is disposed over and electrically connected to the gate electrode. The hard mask structure surrounds the plug and is disposed over the gate electrode, wherein the hard mask structure includes a first hard mask layer and a second hard mask layer, the second hard mask layer covers a sidewall and a top surface of the first hard mask layer, and a material of the first hard mask layer is different from a material of the second hard mask layer.