A gas turbine engine has a compressor section with a low pressure compressor and a high pressure compressor having a downstream end. A cooling air system includes a tap from a location upstream of the downstream most location. The tap passes air to a boost compressor, and the boost compressor passes the air back to a location to be cooled. The boost compressor is driven by a shaft in the engine through an epicyclic gear system. A speed control changes the relative speed between an input and an output to the epicyclic gear system.