Devices, systems and methods for encoding and decoding digital video using historical information containing coding candidates are described. In a representative aspect, a method for video processing includes performing a conversion between a first block of video data of a video and a bitstream representation of the video. The method also includes and updating, based on a coding characteristic of the first block of video data, a table of motion candidates using motion information associated with the first block.