A tramp metal removing device has a primary housing to define a product flow path for being passed by a stream of raw materials and a moving path. A secondary housing is connected to the primary housing. A plurality of drawer units are sequentially stacked on the primary housing and secondary housing. Each drawer unit has a frame, a plurality of magnetic members and a scraping assembly. The frame is coupled with the primary and secondary housings in a movable way. Each of magnetic members is secured on the frame and has a magnetic section and a non-magnetic section. The scraping assembly is coupled with the frame in a way that it is only moveable in the secondary housing for removing tramp metals of a stream of raw materials in a two-stage manner.