A method for video processing is provided to include performing, for a conversion between a current video block of a video and a coded representation of the video, a motion information refinement process based on samples in a first domain or a second domain; and performing the conversion based on a result of the motion information refinement process, wherein, during the conversion, the samples are obtained for the current video block from a first prediction block in the first domain using an unrefined motion information, at least a second prediction block is generated in the second domain using a refined motion information used for determining a reconstruction block, and reconstructed samples of the current video block are generated based on the at least the second prediction block.