An assembly is provided for a gas turbine engine. This gas turbine engine assembly includes a case, a housing, a component and a spring element. The case includes an aperture that extends axially along an axis through the case. The housing is attached to the case with a cavity formed by and axially between the housing and the case. The component includes a base and a projection. The base is disposed within the cavity and axially engages the case. The projection projects out from the base and axially through the aperture. The spring element is disposed within the cavity. The spring element is compressed axially between and engages the base and the housing.