Structures and formation methods of a semiconductor device are provided. The method includes forming a first fin structure and a second fin structure over a substrate, and forming first, second and third dummy gate stacks over the substrate. The first dummy gate stack and the second dummy gate stack partially cover the first fin structure and the second fin structure respectively. The third dummy gate stack is between the first dummy gate stack and the second dummy gate stack. The method also includes partially removing the third dummy gate stack such that a semiconductor layer of the third dummy gate stack remains over the substrate, forming a protection layer over the semiconductor layer, and replacing the first dummy gate stack and second dummy gate stack with a first gate stack and a second gate stack, respectively.