A high-strength hot rolled steel sheet with excellent bendability and low-temperature toughness includes a chemical composition including, in mass %, C: 0.08 to 0.25%, Si: 0.01 to 1.0%, Mn: 0.8 to 2.1%, P: not more than 0.025%, S: not more than 0.005% and Al: 0.005 to 0.10%, the balance including Fe and inevitable impurities, and a microstructure having a bainite phase and/or a tempered martensite phase as a main phase, the average grain diameter of prior austenite grains being not more than 20 μm as measured with respect to a cross section parallel to a rolling direction and not more than 15 μm as measured with respect to a cross section perpendicular to the rolling direction.