A structure and formation method of a semiconductor device is provided. The method includes forming a semiconductor stack having first sacrificial layers and first semiconductor layers laid out alternately. The method also includes patterning the semiconductor stack to form a first structure and a second structure. The method further includes replacing the second structure with a third structure having second sacrificial layers and second semiconductor layers laid out alternately. In addition, the method includes removing the first sacrificial layers in the first structure and the second sacrificial layers in the third structure. The method includes forming a first metal gate stack and a second metal gate stack to wrap around each of the first semiconductor layers in the first structure and each of the second semiconductor layers in the third structure, respectively.