A method of video processing is provided to include: maintaining a set of tables, wherein each table includes motion candidates and each motion candidate is associated with corresponding motion information; updating a motion candidate list based on motion candidates in one or more tables using a pruning operation on the motion candidates; and performing a conversion between a first video block and a bitstream representation of a video including the first video block using the constructed motion candidate list.