A virtual image display system and a virtual image display method are provided. The virtual image display method includes: displaying, by a handheld electronic device, an identification pattern, where the identification pattern corresponds to an object image; photographing, by a virtual image display, the identification pattern and displaying the object image according to the identification pattern; and capturing, by the virtual image display, inertial measurement information of the handheld electronic device when the identification pattern disappears and adjusting at least one of a position and a direction of the object image according to the inertial measurement information.