The present invention proposes a method for monitoring the nonlinear effect of an optical fiber link by fractional Fourier transformation, FRFT, by calculating an optimal fractional order of the FRFT of the frequency-domain signal propagating through an optical fiber link, calculating the chromatic dispersion of an optical fiber link based on the optimal fractional order, compensating for chromatic dispersion to the signal, calculating an optimal fractional order of the FRFT for the time-domain signal following the compensation for chromatic dispersion, calculating the time-domain chirp caused by the nonlinear effect of an optical fiber link based on the optimal fractional order, and monitoring the nonlinear effect of an optical fiber link based on the absolute value of the calculated time-domain chirp. The method can be used for quantitatively monitoring the nonlinear effect of an optical fiber link in an optical fiber communication system consisting of different types of optical fibers.