A capsule medical device guidance system includes: a capsule medical device including a permanent magnet and configured to be introduced into a subject; a magnetic field generator configured to generate a magnetic field to be applied to the capsule medical device; an operation input device configured to input operation information; and a processor including hardware. The processor is configured to: control the magnetic field generator, based on the operation information input from the operation input device, to change the magnetic field to change at least one of the position and the posture of the capsule medical device; obtain control information for the magnetic field generator in a state where forces acting on the capsule medical device are balanced before starting or when starting operation on the operation input device; and control the magnetic field generator by using the control information after the operation is finished.