A video encoder determines scaled transform coefficients, wherein determining the scaled transform coefficients comprises scaling transform coefficients of a block of the video data according to a given quantization step. The video encoder determines scalar quantized coefficients, wherein determining the scalar quantized coefficients comprises applying scalar quantization to the scaled transform coefficients of the block. Additionally, the video encoder applies a neural network that determines a respective set of probabilities for each respective transform coefficient of the block. The respective set of probabilities for the respective transform coefficient includes a respective probability value for each possible adjustment value in a plurality of possible adjustment values. Inputs to the neural network include the scaled transform coefficients and the scalar quantized coefficients. The video encoder determines, based on the set of probabilities for a particular transform coefficient of the block, a quantization level for the particular transform coefficient.