A method of video decoding performed in a video decoder includes receiving a coded video bitstream including a current block that is divided into a plurality of sub-blocks. The method includes performing sub-block based affine motion compensation on the current block to generate a sub-block prediction for each pixel in each sub-block of the current block. The method further includes determining one or more spatial gradients for each sub-block prediction. The method includes performing, for each sub-block prediction, prediction refinement with an optical flow process using the respective determined one or more spatial gradients and at least one constraint included in the coded video bitstream. The method further includes adding, for each sub-block prediction, an output of the respective prediction refinement to the respective sub-block prediction to generate a final prediction for each pixel in each sub-block of the current block.