Devices, systems and methods for digital video coding, which includes using default motion candidates for video coding, are described. An exemplary method for video processing includes inserting, for a conversion between a block of a video and a bitstream representation of the video, at least one past motion candidate in a table of past motion candidates to a motion candidate list. The table of past motion candidates is derived based on past conversions. The method includes determining, after the at least one past motion candidate is inserted, that a motion candidate is not to be added to the motion candidate list. The motion candidate is derived based on the table of past motion candidates. The method also includes performing the conversion based on the motion candidate list.