To detect movement of a depth camera in a motion capture system, readings from an accelerometer in the depth camera are processed to provide short and long term moving averages for each of the x, y and z axes. For each axis, a difference between the short and long term averages is obtained. A sum of the differences is obtained by summing the absolute value of each difference, across all three axes. This sum of differences is then compared to a threshold to determine if movement is detected. In another approach, the short and long term averages of the accelerometer readings are converted to pitch and roll values, and the pitch and roll values are separately compared to a threshold to determine if movement is detected. A tracking algorithm can be temporarily halted while movement is detected. Readings in the vertical direction can be attenuated by a scaling factor.