There is provided a method of manufacturing a semiconductor device, which includes: forming a silicon film inside a recess formed in a surface of a workpiece by supplying a film forming gas containing silicon to the workpiece; subsequently, supplying a process gas, which includes a halogen gas for etching the silicon film and a roughness suppressing gas for suppressing roughening of a surface of the silicon film after being etched by the halogen gas, to the workpiece; etching the silicon film formed on a side wall of the recess to enlarge an opening width of the recess by applying thermal energy to the process gas and activating the process gas; and subsequently, filling silicon into the recess by supplying the film forming gas to the workpiece and depositing silicon on the silicon film remaining in the recess.