The present invention mainly provides a precision calibration method for being applied in a high-precise rotary encoder system, wherein the primary technology feature of the precision calibration method is that: using a laser speckle image capturing module to capture N frames of laser speckle image from an optical position surface of a rotary encoding body, and then using image comparison libraries and particularly-designed mathematical equations to calculate N number of displacement vectors based on the N frames of laser speckle image, so as to eventually calculate N number of angle coordinates corresponding to the N frames of laser speckle image based on the plurality of displacement vectors. Moreover, a rotation matrix is further proposed and used in this precision calibration method, and used for treating the displacement vectors with a displacement vector transforming process in order to effectively enhance the position precision of the high-precise rotary encoder system.