A structure and formation method of a semiconductor device is provided. The method includes forming a semiconductor stack having first sacrificial layers and first semiconductor layers laid out alternately. The method also includes patterning the semiconductor stack to form a first fin structure and a second fin structure. The method further includes replacing the second fin structure with a third fin structure having second sacrificial layers and second semiconductor layers laid out alternately. In addition, the method includes removing the first sacrificial layers in the first fin structure and the second sacrificial layers in the third fin structure. The method includes forming a first metal gate stack and a second metal gate stack to wrap around each of the first semiconductor layers in the first fin structure and each of the second semiconductor layers in the third fin structure, respectively.