Semiconductor devices and method of forming the same are disclosed. One of the semiconductor devices includes a substrate, a gate structure, a plug and a hard mask structure. The gate structure is disposed over the substrate. The plug is disposed over and electrically connected to the gate structure. The hard mask structure is disposed over the gate structure and includes a first hard mask layer and a second hard mask layer. The first hard mask layer surrounds and is in contact with the plug. The second hard mask layer surrounds the first hard mask layer and has a bottom surface at a height between a top surface and a bottom surface of the first hard mask layer. A material of the first hard mask layer is different from a material of the second hard mask layer.