An electromagnetic (EM) position tracking system identifies the pose of objects based on detected strength values of an EM field. To address distortions in the field, the system employs a pose sensor to provide a second pose of the mobile unit. Under conditions where no distortion in the EM field has been detected, the HMD applies a nominal set of corresponding weights to the EM pose data and the IMU pose data, respectively, and combines the weighted pose value to identify a combined pose of the mobile unit. In response to detecting conditions that indicate distortion in the EM field, the HMD can apply different weights to the EM pose data and IMU pose data to, for example increase the influence of the IMU pose data on the combined pose.