The present disclosure relates to a handle operating device for a circuit breaker, and there is provided a handle operating device including a connecting rod configured to transfer power for operating a manual operating handle that manually operates the on/off of a circuit breaker from the outside; a movable frame connected to the connecting rod to be movably coupled to an outer surface of a circuit breaker case in a direction parallel to the operating direction of the manual operating handle; a movable member provided at the movable frame in an interlocking manner to move the manual operating handle; a slot hole is formed on the moving member in an inclined manner from the moving direction of the manual operating handle; and a fastening member fastened to the movable member and movable frame through the slot hole.