A dumbbell assembly includes a number of weight members, a handle device engageable with the weight members for anchoring a selected number of the weight members to the handle device, a base support includes a number of socket openings for accommodating the weight members, and a follower is slidably engaged in the base support, and the follower includes one or more catches for selectively engaging with the weight members that are supported on the base support and for selectively retaining the weight members to the base support. A spring biasing member is engaged with the follower for biasing the catches of the follower to engage with the weight members.