Devices, systems and methods for digital video coding, which includes matrix-based intra prediction methods for video coding, are described. In a representative aspect, a method for video processing includes performing a conversion between a current video block of a video and a bitstream representation of the current video block using a matrix based intra prediction (MIP) mode in which a prediction block of the current video block is determined by performing, on previously coded samples of the video, a boundary downsampling operation, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation, where the performing the conversion includes deriving, according to a rule, boundary samples by applying a left bit shift operation or a right bit shift operation on a sum of at least one reference boundary sample, and where the rule determines whether to apply the left bit shift or the right bit shift operation.