A method of forming a semiconductor device includes the following steps. First of all, a substrate is provided, and a dielectric layer is formed on the substrate. Then, at least one trench is formed in the dielectric layer, to partially expose a top surface of the substrate. The trench includes a discontinuous sidewall having a turning portion. Next, a first deposition process is performed, to deposit a first semiconductor layer to fill up the trench and to further cover on the top surface of the dielectric layer. Following these, the first semiconductor layer is laterally etched, to partially remove the first semiconductor layer till exposing the turning portion of the trench. Finally, a second deposition is performed, to deposit a second semiconductor layer to fill up the trench.