Structures and formation methods of a semiconductor device are provided. The semiconductor device includes a substrate and a first fin structure and a second fin structure over the substrate. The semiconductor device also includes a first gate stack and a second gate stack partially covering the first fin structure and the second fin structure, respectively, and a stack structure over the substrate. The stack structure is between the first gate stack and the second gate stack. The stack structure includes a semiconductor layer over the substrate and a protection layer over the semiconductor layer.