An optical transport system configured to compensate nonlinear signal distortions using a backward-propagation algorithm in which some effects of polarization mode dispersion on the nonlinear signal distortions are accounted for by employing two or more different approximations of said effects within the bandwidth of the optical communication signal. In an example embodiment, the corresponding digital signal processor (DSP) is configured to switch between different approximations based on a comparison, with a fixed threshold value, of a difference between frequencies of various optical waves contributing to the nonlinear signal distortions, e.g., through four-wave-mixing processes. In different embodiments, the backward-propagation algorithm can be executed by the transmitter's DSP or the receiver's DSP.