The present invention relates to a method of forming a coating layer of which a composition can be controlled, the method comprising steps of: preparing a substrate inside a chamber; evaporating a deposition material to generate YF3 or YOF particles in a gas phase by irradiating an electron beam on a YF3 deposition material provided in a solid form in an electron beam source; generating radical particles having activation energy by injecting a process gas containing oxygen into a RF energy beam source; irradiating an RF energy beam including oxygen radical particles toward the substrate; controlling a composition of a thin film by generating YOF deposition particles having a modified atomic ratio by adjusting an amount of fluorine substitution by oxygen as the YF3 or YOF particles and the oxygen radical particles react, and depositing the YOF deposition particles on the substrate with the RF energy beam.