A bicycle handle assembly for a handlebar contains a body including a peripheral fringe and a side fringe connected together to define a receiving space. The receiving space includes a first side and a second side. The peripheral fringe includes a first engaging portion, a first connecting portion and a fixing portion connected with two ends of the first engaging portion. A movable retaining member is installed in the receiving space and includes a second engaging portion, a second connecting portion and a moving portion connected with two ends of the second engaging portion. The second connecting portion is used to connect with the first connecting portion of the body. A locking element is used to force the moving portion toward the fixing portion of the body, so that the external fence of the handlebar is retained tightly between the first engaging portion and the second engaging portion.