Devices, systems and methods for encoding and decoding digital video using historical information containing coding candidates are described. In a representative aspect, a method for video processing includes maintaining one or more tables of motion candidates during a conversion between a current video block and a bitstream representation of a video including the current video block, determining that the current video block is coded using a merge mode or an Advanced Motion Vector Prediction (AMVP) mode, and performing, based on the determining, the conversion between the current video block of and the bitstream representation of the video using one or more tables of motion candidates.