A roller suspension system for a track-driven work vehicle may generally include a support beam and a resilient bushing configured to be coupled to the support beam. The resilient bushing may define a shaft opening. In addition, the system may include a roller wheel assembly having a roller wheel and a roller shaft. The roller wheel may be rotatable relative to the roller shaft. In addition, the roller shaft may be received within the shaft opening of the resilient bushing such that the roller shaft is coupled to the support beam via the resilient bushing.