An armrest control device includes a driving mode determiner that determines a switching state of a driving mode of a vehicle capable of switching between an automatic driving mode and a manual driving mode, and an armrest controller that moves a supporter, which supports an arm of a driver of an armrest, so that the supporter approaches a steering wheel, when the driving mode determiner determines that a switching from the automatic driving mode to the manual driving mode has started.