Devices, systems and methods for constructing low-complexity non-adjacent merge candidates. In a representative aspect, a method for video processing includes receiving a current block of video data, selecting, based on a rule, a first non-adjacent block that is not adjacent to the current block, constructing a first merge candidate comprising motion information based on the first non-adjacent block, identifying a second non-adjacent block that is not adjacent to the current block and different from the first non-adjacent block, based on determining that the second non-adjacent block fails to satisfy the rule, refraining adding a second merge candidate derived from the second non-adjacent block, constructing a merge candidate list based on the first non-adjacent block, and decoding the current block based on the merge candidate list.