A method for manufacturing a semiconductor device includes etching a substrate to form a semiconductor fin. An isolation structure is formed above the substrate and laterally surrounds the semiconductor fin. A fin sidewall structure is formed above the isolation structure and on a sidewall of the semiconductor fin. The semiconductor fin is recessed to expose an inner sidewall of the fin sidewall structure. A source/drain epitaxial structure is grown on the recessed semiconductor fin.