Embodiments of the present disclosure relates to improvement for temporal gradient calculating in BIO. A method for video processing is provided, including: determining a reference block for a current block; performing a refinement process for prediction of the current block, based on the reference block; and performing a conversion between the current block and a bitstream representation of a video including the current block, based on a result of the refinement process; wherein, the refinement process comprises: modifying the reference block into a modified reference block, and calculating a temporal gradient based on the modified reference block, or calculating an original temporal gradient based on the reference block, and modifying the original temporal gradient into a modified temporal gradient.