A method of visual media processing method includes performing a conversion between a current video block of a current picture of a visual media data and a bitstream representation of the visual media data. The conversion is based on a reference region from the current picture comprising reference samples used for deriving a prediction block of the current video block. A virtual buffer of a defined size is used for tracking availability of the reference samples for deriving the prediction block.