A rotary damper device with a one-way clutch includes a rotary damper having a damper housing having a viscous fluid sealed inside, and a rotor having a rotor blade received rotatably inside the damper housing and a rotor shaft; and the one-way clutch including an inner member having a recessed portion on an outer perimeter portion and coaxially joined to the rotor shaft, an outer member having an inner gear, and a gear housed in the recessed portion to engage with the inner gear. When the outer member rotates in a positive direction, the outer member rotates independently from the inner member. When the outer member rotates in a negative direction, the gear couples and the outer member and the inner member rotate together. The inner member has a portion larger than the rotor shaft. The outer member has a locking portion to be locked on the inner member.