A system and method for tracking an object (14) received within a body (16) using ultrasound wherein shifts in position of an ultrasound probe (20) tracking the object (either deliberate or accidental) can be corrected for. The method comprises tracking a position of the object through the body using an ultrasound transducer unit, and sensing movement of the ultrasound transducer unit using a physically coupled motion sensor (26). From the movement of the ultrasound transducer unit, movement of its field of view (24) can be determined. Based on a tracked history of positions (28) of the object through the field of view, and on the tracked movement of the field of view relative to the body, a direction of motion of the object through the body can be derived.