Exemplary video processing methods and apparatuses for encoding or decoding a current block by bi-directional prediction are disclosed. L0 and L1 reference blocks are determined for the current block, and a motion vector refinement is applied to refine the L0 and L1 reference blocks using a motion estimation search performed on each processing unit in the current block. A bi-directional optical flow operation is early terminated according to a cost of each processing unit in the current block. The processing units for cost calculation in the motion vector refinement and the bi-directional optical flow early termination are unified. The bi-directional optical flow operation is applied to the current block to further refine the L0 and L1 reference blocks if the bi-directional optical flow operation is not early terminated. A final inter predictor is derived from the refined L0 and L1 reference blocks for encoding or decoding the current block.