A visual media processing method includes determining, for a conversion between a first video block of a visual media data and a bitstream representation of the visual media data, an availability of a second video block of the visual media data using a rule; and performing the conversion based on the determining, wherein the rule is at least based on a coding mode used for coding the first video block into the bitstream representation, wherein the rule specifies disallowing motion information of the second video block from being used in a merge list construction of the first block by treating the second video block as unavailable.