A stress feedback device and a virtual image display system are provided. The stress feedback device includes a position sensor and a pressure generator. The position sensor is configured to generate a position information based on a trigger signal and send the position information to a host. The pressure generator is configured to apply a stress to a target area based on the trigger signal and adjust an intensity of the stress.