An object linking method including: detecting each first object based on a video image, detecting each second object based on each data measured by each inertial sensor, each first object or each second object having each first state or each second state that is one of states including a moving state and a stopping state, and linking, by a computer, each first object and each second object in one-to-one, based on each first change of each first state and each second change of each second state, wherein when a specified first object of at least one first object is not linked to any of the at least one second object, the specified first object is linked to a virtual second object that is added to the at least one second object.