A semiconductor device with an arithmetic processing function is provided. In the semiconductor device, an imaging portion and an arithmetic portion are electrically connected to each other through an analog processing circuit. The imaging portion includes a pixel array in which pixels used for imaging and reference pixels used for image processing are arranged in a matrix, and a row decoder. The arithmetic portion includes a memory element array in which memory elements and reference memory elements are arranged in a matrix, an analog processing circuit, a row decoder, and a column decoder.