A disclosed method for optimizing channel selection in a flexible grid of an optical network may be used to select a minimum number of channels to allocate to traffic between pairs of optical transponders at a particular distance based on the modulation format and the FEC mechanism used. The method may include selecting an initial number of channels to allocate to the traffic, tuning the modulation format for the traffic while potentially reducing the number of channels, and tuning the FEC mechanism for the traffic while potentially further reducing the number of channels. The transponders may support multiple modulation formats of different orders and an adaptive FEC mechanism for which the maximum number of FEC overhead bytes is equal to the number of bytes in the transmitted data packets. The method may be implemented by an SDN controller and may be dependent on a feedback mechanism between the optical transponders.