The present invention provides fine grain structures for rare earth permanent magnets (REPMs) and their production in a manner to significantly enhance flexural strength and fracture toughness of the magnets with no or little sacrifice in the hard magnetic properties. The tough REPMs can have either homogeneous or heterogeneous refined grain microstructural architectures achieved by introducing a small amount of additive particle materials into the magnet matrix, such as fine-sized, insoluble, chemically stable, and non-reactive with the magnet matrix. These additive materials can act effectively as both heterogeneous nuclei sites and grain growth inhibitors during the heat treatment processes, which in turn resulting in refined grain structures of the REPMs. Alternatively, the fine grain structures were also achieved by using magnet alloy feedstock powders with finer particle sizes. The fine grains acting as the strengthening sites can inhibit the crack nucleation and can also slow down the propagation of micro-cracks, which in turn increasing magnet's fracture toughness.