Even when the amount of overlay deviation between patterns located in different layers is large, correct measurement of the amount of overlay deviation is stably performed. The charged particle beam device includes a charged particle beam irradiation unit that irradiates a sample with a charged particle beam, a first detection unit that detects secondary electrons from the sample, a second detection unit that detects backscattered electrons from the sample, and an image processing unit that generates a first image including an image of a first pattern located on the surface of the sample based on an output of the first detection unit, and generates a second image including an image of a second pattern located in a lower layer than the surface of the sample based on an output of the second detection unit. A control unit adjusts the position of a measurement area in the first image based on a first template image for the first image, and adjusts the position of a measurement area in the second image based on a second template image for the second image.