A gas turbine engine has a compressor section with a low pressure compressor and a high pressure compressor. The high pressure compressor has a downstream most location. A cooling air system includes a tap from a location upstream of the downstream most location. The tap passes air to a boost compressor and a heat exchanger, which passes the air back to a location to be cooled. The boost compressor is driven by a shaft which drives the lower pressure compressor.