A semiconductor device for performing a sum-of-product computation and an operating method thereof are provided. The semiconductor device includes an inputting circuit, a scaling circuit, a computing memory and an outputting circuit. The inputting circuit is used for receiving a plurality of inputting signals. The inputting signals are voltages or currents. The scaling circuit is connected to the inputting circuit for transforming the inputting signals to be a plurality of compensated signals respectively. The compensated signals are voltages or currents. The computing memory is connected to the scaling circuit. The computing memory includes a plurality of computing cells and the compensated signals are applied to the computing cells respectively. The outputting circuit is connected to the computing memory for reading an outputting signals of the computing cells. The outputting signal is voltage or current.