A paper conveying device includes: a roller shaft having a first engaging portion and configured to rotate integrally with the first engaging portion; and a roller unit having a second engaging portion and provided on the roller shaft, the roller unit being configured to convey a sheet of paper as the roller unit rotates while being in contact with the sheet of paper. In an engaged state where the first engaging portion and the second engaging portion engage with each other, the roller unit is restricted from moving in an axis direction relative to the roller shaft. In a disengaged state where the first engaging portion and the second engaging portion disengage from each other, the roller unit is movable in the axis direction relative to the roller shaft.