A recording material processing apparatus includes: a first stacking unit configured to stack a recording material; a second stacking unit provided on a downstream side of the first stacking unit in a conveyance direction of the recording material; an alignment unit configured to align the recording material stacked in the first stacking unit before the recording material is discharged to the second stacking unit; and a control unit configured to perform control of discharging the recording material to the second stacking unit by switching between first control not to align the recording material by the alignment unit and second control to align the recording material by the alignment unit.