By causing at least one of Dy and Tb to be adhered to the surface of an iron-boron-rare earth based sintered magnet of a predetermined shape, and is then to be diffused into grain boundary phase, a permanent magnet can be manufactured at high workability and low cost. An iron-boron-rare earth based sintered magnet is disposed in a processing chamber and is heated to a predetermined temperature. Also, an evaporating material made up of a fluoride containing at least one of Dy and Tb disposed in the same or another processing chamber is evaporated, and the evaporated evaporating material is caused to be adhered to the surface of the sintered magnet. The Dy and/or Tb metal atoms of the adhered evaporating material are diffused into the grain particle phase of the sintered magnet before a thin film made of the evaporated material is formed on the surface of the sintered magnet.