A bone anchoring device includes an anchoring element having a shank to be anchored in a bone or a vertebra and a receiving part connected to the shank, and a rod for connecting at least two anchoring elements, the rod having at least a tubular portion. The receiving part comprises a recess for receiving said rod. The bone anchoring element also includes a locking element cooperating with the receiving part to secure the rod in the recess, a filling piece being arranged between the locking element and the rod for exerting pressure onto the rod to lock the rod in the recess. The filling piece includes a rod contacting surface which contacts a portion of said rod. The shape of the rod contacting surface is adapted to the shape of the portion of the rod.