A method for forming a semiconductor device is provided. The method includes removing a first portion of a substrate to form a recess in the substrate. The method includes forming an epitaxy layer in the recess. The epitaxy layer and the substrate are made of different semiconductor materials. The method includes forming a stacked structure of a plurality of first semiconductor layers and a plurality of second semiconductor layers alternately stacked over the substrate and the epitaxy layer. The method includes removing a second portion of the stacked structure and a third portion of the epitaxy layer to form trenches passing through the stacked structure and extending into the epitaxy layer. The stacked structure is divided into a first fin element and a second fin element by the trenches, and the first fin element and the second fin element are over the substrate and the epitaxy layer respectively.