A structure and a formation method of a semiconductor device structure are provided. The semiconductor device structure includes a semiconductor substrate and a gate stack over the semiconductor substrate. The gate stack includes a work function layer and a metal filling over the work function layer. The semiconductor device structure also includes a dielectric structure over the semiconductor substrate and adjacent to the gate stack. The dielectric structure is in direct contact with the work function layer and the metal filling.