A structure and a formation method of a semiconductor device are provided. The method includes forming a passivation layer over a semiconductor substrate. The method also includes forming a magnetic element over the passivation layer. The method further includes forming an isolation layer over the magnetic element and the passivation layer. The isolation layer includes a polymer material. In addition, the method includes forming a conductive line over the isolation layer, and the conductive line extends across the magnetic element.