A semiconductor device has a conductive layer formed on a substrate. The conductive layer has a first portion constituting contact pads and a second portion constituting an integrated passive device such as an inductor. A spacer is formed on the substrate around the second portion of the conductive layer. The spacer can be insulating material or conductive material for shielding. A semiconductor die is mounted to the spacer. An electrical connection is formed between contact pads on the semiconductor die and the contact pads on the substrate. An encapsulant is formed around the semiconductor die, electrical connections, spacer, and conductive layer. The substrate is removed to expose the conductive layer. An interconnect structure is formed on the backside of the substrate. The interconnect structure is electrically connected to the conductive layer. The semiconductor device can be integrated into a package.