In one aspect, a system for providing haptic feedback to an operator of a work vehicle may include an operator control device and a controller. The controller may be configured to identify one of a first component of the work vehicle or a second component of the work vehicle as a selected component to be controlled via the operator control device. Furthermore, the controller may be configured to control the operation of the operator control device such that the operator control device provides a first haptic response to the operator when first component is identified as the selected component. Additionally, the controller may be configured to control the operation of the operator control device such that the operator control device provides a second haptic response to the operator when second component is identified as the selected component, with the second haptic response differing from the first haptic response.