An armrest assembly includes a mounting plate disposed on a support column and having an elongated moving slot for a first pivot shaft to movably extend therethrough, an armrest plate having a sliding slot elongated transverse to the moving slot such that the first and second pivot shafts extend through and are slidable along the sliding slot to allow the armrest plate to be horizontally movable and turnable relative to the mounting plate, and a retaining unit disposed to produce a magnetically attractive force that generates resistance against a relative movement of the armrest plate to the mounting plate.