A handle assembly of a lock includes a handle, a sleeve, a stopper and an engaging member, wherein the handle comprises an accommodating slot, a lodge hole and a penetrating hole. The sleeve is disposed at the accommodating slot along the direction of a major axis and comprises a through hole. The stopper is disposed at the sleeve and protrudes to the sleeve via the through hole of the sleeve. Owing to the through hole corresponded to the lodge hole, the stopper is lodged in the lodge hole of the handle. The engaging member is disposed in the accommodating slot and adjustably movable along the direction of a transverse axis so that the handle and the sleeve are steadily engaged from each other.