A method of removing a hard mask on a gate includes forming a first gate structure and a second gate structure. The first gate structure includes a first gate, a first hard mask disposed on the first gate and a first spacer surrounding the first gate and the first hard mask, wherein the second gate structure includes a second gate, a second hard mask disposed on the second gate and a second spacer surrounding the second gate and the second hard mask. Later, the first spacer surrounding the first hard mask and the second spacer surrounding the second hard mask are removed. After that, a dielectric layer is formed to cover the first hard mask and the second hard mask. Finally, the second dielectric layer, the first mask layer and the second mask layer are removed.