A gas turbine engine comprises a main compressor section having a high pressure compressor with a downstream discharge, and more upstream locations. A turbine section has a high pressure turbine. A tap taps air from at least one of the more upstream locations in the compressor section, passes the tapped air through a heat exchanger and then to a cooling compressor. The cooling compressor compresses air downstream of the heat exchanger, and delivers air into the high pressure turbine. A core housing has an outer peripheral surface and a fan housing defining an inner peripheral surface. At least one bifurcation duct extends between the outer peripheral surface to the inner peripheral surface. The heat exchanger is received within the at least one bifurcation duct.