A fixing device includes: a fixing member that fixes a toner image onto a recording medium; a pressing member that presses the recording medium against the fixing member; an oiling unit that applies release oil onto a surface of the fixing member to clean the surface; a removing member that is arranged between an downstream side of the oiling unit and an upstream side of the pressing member in a rotating direction of the fixing member and movable between an in-contact state where the removing member is in contact with the surface of the fixing member and an out-of-contact state where the removing member is out of contact with the surface; a drive unit that drives the removing member; and an oiling control unit that, when starting earliest fixing, controls the drive unit to gradually move the removing member from the in-contact state into the out-of-contact state.