Devices, systems and methods for encoding and decoding digital video using a table containing coding candidates are described. In a representative aspect, a video processing method is provided to include maintaining a set of tables, wherein each table includes motion candidates and each motion candidate is associated with corresponding motion information; and performing a conversion between a bitstream representation of a video including a current block and the current block, and wherein the performing of the conversion includes checking at most N motion candidates in a table based on a rule.