A nonlinear compensation method and apparatus and a system in a multicarrier optical communication system where the method includes: determining a coefficient of a linear filter in nonlinear compensation according to an end-to-end channel linear response of the system; determining taps of a nonlinear compensation filter needing to be opened and coefficients of the taps according to a hardware compensation ability of the system and the coefficient of the linear filter; and compensating for a nonlinear damage of the system by using the selected coefficients of the nonlinear compensation filter. With the method, apparatus or the system provided by this application, very good compensation performance may be achieved in a range of power consumption of the multicarrier optical communication system by only opening and using few taps of the compensation filter.