A method of classifying the defects on a wafer having some same chips and corresponding system is provided. After receiving images formed by scanning the wafer using a charged particle beam, these images are examined such that both defective images and defect-free images are found. Then, the defect-free images are translated into a simulated layout of the chip, or a database is used to provide the simulated layout of the chip. Finally, the defects on the defective images are classified by comparing the images with the simulated layout of the chip. The system has some modules separately corresponds to the steps of the method.