A valve operating device that opens and closes a valve of an internal combustion engine by driving a valve stem in an axial direction includes a rocker cam that rocks in accordance with a rotation of a drive shaft and has a driving cam portion and a lift restricting cam portion; a cam follower that lifts the valve by contacting the driving cam portion slidingly; and a restricting member that prevents the cam follower from separating from the driving cam portion in cooperation with the lift restricting cam portion. A gap is formed between the restricting member and the rocker cam in a state where the cam follower is in contact with the driving cam portion and disappears such that the restricting member and the rocker cam come into contact with each other only when the cam follower separates from the driving cam portion.