A pillow block bearing assembly for a drivetrain assembly of a work vehicle may include a pillow block housing defining a bearing cavity, a bearing disposed within the bearing cavity, and first and second sealing devices installed within the bearing cavity. Additionally, the drivetrain assembly may include a shaft received within the pillow block bearing assembly, with the shaft extending through the bearing such that an outer circumferential surface of the shaft directly contacts an inner circumferential surface of an inner race of the bearing at a bearing/shaft interface defined between the shaft and the bearing. Moreover, a sliding fit may be defined between the outer circumferential surface of the shaft and the inner circumferential surface of the inner race at the bearing/shaft interface such that the shaft is allowed to slide axially relative to the bearing when a given axial load is applied through the shaft.