An optical system may include an optical transmitter or an optical receiver. The optical transmitter may include a laser to provide an input signal, a first digital signal processor (DSP) to receive a data signal, provide non-rotated and rotated symbols corresponding to the data signal, and provide digital signals corresponding to the non-rotated and the rotated symbols to cause one or more digital to analog converts to convert the digital signals to analog signals. The optical transmitter may further provide a modified 5 quadrature amplitude modulation (5QAM) signal based on the analog signals and the input signal. A constellation map of the modified 5QAM signal may include all constellation points of a 16QAM signal. The optical receiver may include a second DSP to receive components associated with the 5QAM signal, filter the components, reduce phase noise of the 5QAM signal, and provide data associated with the 5QAM signal.