Some embodiments provide a method for selecting an encoding mode from several encoding modes. For each encoding mode from several encoding modes, the method computes a Lagrangian value based on a distortion value that is identified by using a function that reduces the impact of outliers. The method selects a particular encoding mode based on the computed Lagrangian values. In some embodiments, the function is a Huber function. In some embodiments, the computed Lagrangian value is further based on a bit rate value and a Lagrangian multiplier.