A transmitter and method include a LDPC encoder configured to encode source data, and a mapper configured to generate three coordinates in accordance with a 3D signal constellation where the coordinates include an amplitude coordinate and two phase coordinates. A laser source is modulated in accordance with each of the three coordinates to provide a transmission signal. A receiver, includes a demapper receives an input signal from three branches to demap the input signal using a three-dimensional signal constellation having three coordinates. The three branches include a direct detection branch, and two coherent detection branches such that the direct detection branch detects an amplitude coordinate of the input signal and the two coherent detection branches detect in-phase and quadrature coordinates of the input signal. A bit prediction module and at least one LDPC decoder are configured to iteratively decode bits by feeding back extrinsic LLRs to the demapper.