An optical neural network apparatus that optically implements an artificial neural network includes an input layer, a hidden layer, and an output layer sequentially arranged in a traveling direction of light, wherein the output layer includes an image sensor including a plurality of light sensing pixels arranged in two dimensions, and wherein the input layer or the hidden layer includes at least one passive phase modulator configured to locally modulate a phase of incident light depending on positions on a two dimensional plane.