A harness system includes an upper buckle, an upper strap and a restraining assembly including an anti-sliding structure and a beam structure. A through slot is formed on the upper buckle. The upper strap passes through the through slot. The beam structure is slidably disposed on the upper buckle. The anti-sliding structure is disposed on at least one of the beam structure and a wall of the through slot. The upper strap is engaged by the anti-sliding structure for restraining a sliding movement of the upper buckle relative to the upper strap and toward a shoulder portion when the upper strap is forced to drive the beam structure to slide relative to the upper buckle and toward the wall of the through slot.