A pipeline video decoder with memory latency management includes memory and at least one processor coupled to the memory. The processor converts a received bitstream into symbol data including pixel data and intra-prediction and inter-prediction control parameters. The processor generates decoder-pipeline region (DPR)-based coarse motion data based on the inter-prediction control parameters. The processor further fetches one or more reference blocks associated with a current prediction unit (PU) of a DPR based on the coarse motion data, and generates decoded motion data based on the inter-prediction control parameters and refined motion data. The refined motion data is generated based on the decoded motion data and the one or more reference blocks.