A banknote pay-in/pay-out device and a banknote processing machine having a simpler pay-in/pay-out port section mechanism than hitherto. During paying in, out of banknotes inserted into a pay-in/pay-out port section, a pay-in reject banknote that has been determined to be unsuitable for paying in is conveyed to a sorting conveyance path, and temporarily retained in the sorting conveyance path. So doing enables, for example, the pay-in reject banknote to be temporarily retained in the sorting conveyance path until, for example, pay-in counting processing has been completed, with the pay-in reject banknote being replaced in the pay-in/pay-out port section to be returned to a user after the pay-in counting processing has been completed, without providing a partitioning plate to the pay-in/pay-out port section as hitherto.