This disclosure describes gradient-based prediction refinement. A video coder (e.g., video encoder or video decoder) determines one or more prediction blocks for inter-predicting a current block (e.g., based on one or more motion vectors for the current block). In gradient-based prediction refinement, the video coder modifies one or more samples of the prediction block based on various factors such as displacement in a horizontal direction, the horizontal gradient, a displacement in the vertical direction, and a vertical gradient. This disclosure provides for gradient-based prediction refinement where a precision level of the displacement (e.g., at least one of the horizontal or vertical displacement) is unified (e.g., the same) for different prediction modes.