A method of video decoding performed in a video decoder includes receiving a coded video bitstream including a current block that is divided into a plurality of sub-blocks. Sub-block based affine motion compensation is performed on the current block to generate a sub-block prediction for each pixel in each sub-block of the current block. One or more spatial gradients are determined for each sub-block prediction. For each sub-block prediction, prediction refinement with optical flow process is performed using the respective determined one or more spatial gradients and at least one constraint included in the coded video bitstream. For each sub-block prediction, an output of the respective prediction refinement is added to the respective sub-block prediction to generate a final prediction for each pixel in each sub-block of the current block.