A neural network device with synapses having memory cells each having a floating gate and a first gate over first and second portions of a channel region disposed between source and drain regions, and a second gate over the floating gate or the source region. First lines each electrically connect the first gates in one of the memory cell rows, second lines each electrically connect the second gates in one of the memory cell rows, third lines each electrically connect the source regions in one of the memory cell rows, and fourth lines each electrically connect the drain regions in one of the memory cell columns. The synapses receive a first plurality of inputs as electrical voltages on the fourth lines, and provide a first plurality of outputs as electrical currents on the third lines.