A hot-rolled steel sheet has a composition containing: in mass %, C: 0.01 to 0.2%; Si: 2.5% or less; Mn: 4.0% or less; P: 0.10% or less; S: 0.03% or less; Al: 0.001 to 2.0%; N: 0.01% or less; and O: 0.01% or less, and one kind or a total of two kinds of Ti and Nb for 0.01 to 0.30%. An average effective crystal grain diameter at a sheet thickness ¼ part is 10 μm or less, and an average effective crystal grain diameter at a part of a range of 50 μm from a surface is 6 μm or less. A structure of the steel sheet is tempered martensite or lower bainite, and a volume fraction thereof is 90% or more as a total.