A handle assembly includes a base seat, a handle, and a button. The handle is pivotally connected with the base seat, and includes a retaining section. There is a fastening slot formed in the retaining section. The button includes a latch which is outwardly extended. When the button is moved to a first position, the latch engages into the fastening slot to restrict a movement of the retaining section, thereby preventing a pivoting movement of the handle relative to the base seat; and when the button is moved to a second position, the latch is separated away from the fastening slot to push the handle, so that the handle is pivoted along a direction away from the base seat, thereby to facilitate a user to grip the handle to pull a hard disk driver from a server.