A PoP semiconductor device has a top semiconductor package disposed over a bottom semiconductor package. The top semiconductor package has a substrate and a first semiconductor die disposed over the substrate. First and second encapsulants are deposited over the first semiconductor die and substrate. A first build-up interconnect structure is formed over the substrate after depositing the second encapsulant. The top package is disposed over the bottom package. The bottom package has a second semiconductor die and modular interconnect units disposed around the second semiconductor die. A second build-up interconnect structure is formed over the second semiconductor die and modular interconnect unit. The modular interconnect units include a plurality of conductive vias and a plurality of contact pads electrically connected to the conductive vias. The I/O pattern of the build-up interconnect structure on the top semiconductor package is designed to coincide with the I/O pattern of the modular interconnect units.