A hydraulic drive system of a construction machine includes: an over-center pump connected to a turning motor; a regulator that adjusts tilting angle of the over-center pump; a main pump coupled to over-center pump; and a controller that controls the regulator such that when a turning deceleration operation is performed, the over-center pump's tilting angle decreases in accordance with decrease in a turning operation signal outputted from a turning operation device. The controller: when a pressure accumulation condition satisfied, switches a switching valve to pressure accumulation position to bring an accumulator into communication with a delivery line of the main pump; when a pressure release condition is satisfied, switches the switching valve to a pressure release position to bring accumulator into communication with a suction line of main pump; and when neither the pressure accumulation condition nor pressure release condition is satisfied, switches the switching valve to a neutral position.