A processor-implemented method of implementing an attention mechanism in a neural network includes obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of an attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.