In general, optical communication systems and methods may implement a vestigial phase shift keying (PSK) modulation format to enable relatively high transmission rates with closer channel spacing and improved signal detection in a WDM optical communication system. A PSK modulated signal may be correlated with a phase shift, for example, using a delay line interferometer (DLI), to generate a vestigial PSK signal, which may be pre-filtered, combined, and transmitted in the WDM system. The correlation with a phase shift compresses and shifts the signal spectrum such that intersymbol interference (ISI) length may be reduced with minimal increase in the bit error rate (BER) when the vestigial PSK signal is detected.