Memory layout and conversion are disclosed to improve neural network (NN) inference performance. For one example, a NN selects a memory layout for a neural network (NN) among a plurality of different memory layouts based on thresholds derived from performance simulations of the NN. The NN stores multi-dimensional NN kernel computation data using the selected memory layout during NN inference. The memory layouts to be selected can be a channel, height, width, and batches (CHWN) layout, a batches, height, width and channel (NHWC) layout, and a batches, channel, height and width (NCHW) layout. If the multi-dimensional NN kernel computation data is not in the selected memory layout, the NN transforms the multi-dimensional NN kernel computation data for the selected memory layout.