Communication system optimization using a soft receiver masking technique is disclosed. For example, a method for testing a communication device comprises obtaining a software representation of a receiver portion of a given communication device. A data signal received from a transmitter through a given link channel is then processed, wherein the processing step is performed using the software representation of the receiver portion of the communication device. An output signal is caused to be generated by the software representation of the receiver portion. An eye mask test is then applied to the output signal. Based on a result of the eye mask test, one or more parameters of the transmitter may be adjusted.