In the charged particle beam lithography system, a pattern area to be drawn is divided into a plurality of frames, a main deflection positions a charged particle beam to a subfield within the frame, and an auxiliary deflection draws a pattern in units of the subfield. The charged particle beam lithography system includes a beam optical system including a deflector deflecting the beam, a driver driving the deflector, and a deflection control portion controlling the driver according to drawing data indicating a pattern to be drawn. The deflection control portion controls the driver according to a settling time that is determined so that an offset of an irradiation position of the charged particle beam has a certain value irrespective of any changes in deflection amount of the auxiliary deflection in the subfield.