A disposable gate structure is formed over the alternating stack of first semiconductor material portions and second semiconductor material portions. The second semiconductor material portions are removed selective to the first semiconductor material portions to form suspended semiconductor nanowires. Isolated gate structures are formed in regions underlying the disposable gate structure by deposition and recessing of a first gate dielectric layer and a first gate conductor layer. After formation of a gate spacer, source regions, and drain regions, raised source and drain regions are formed on the source regions and the drain regions by selective deposition of a semiconductor material. The disposable gate structure is replaced with a replacement gate structure by deposition and patterning of a second gate dielectric layer and a second gate conductor layer. Distortion of the suspended semiconductor nanowires is prevented by the disposable gate structure and the isolated gate structures.