A fixing device includes a heat source, a fixing roller, a pressure roller, an elastic layer, a pressing portion, a restricting portion and a driving unit. The fixing roller is heated by the heat source. The pressure roller is held in contact with the fixing roller and forms a fixing nip portion through which a sheet is to be passed. The elastic layer is formed on an inner layer of the fixing roller. The pressing portion presses one roller of the fixing roller and the pressure roller against the other roller while compressing the elastic layer. The restricting portion restricts a compression amount of the elastic layer lest the compression amount should exceed a preset compression amount when the pressing portion presses the roller. The driving unit drives and rotates one of the fixing roller and the pressure roller in a state where the compression amount is restricted.