An optical line terminal (OLT) in flexible passive optical network (PON) transmits a downstream signal with two or more modulation formats, wherein at least one of the modulation formats is a modified PAMx modulation format and wherein x is greater than 2. The modified PAMx modulation encodes data bits such that the probability of at least one predetermined transition between amplitude levels is modified. Furthermore, an optical network unit (ONU) in the flexible PON is assigned to one of the plurality of modulation formats based on one or more parameters, such as configuration of the ONU and line conditions.