According to one embodiment, a processor includes an instruction decoder to receive a first instruction to perform first SKEIN256 MIX-PERMUTE operations, the first instruction having a first operand associated with a first storage location to store a plurality of odd words, a second operand associated with a second storage location to store a plurality of even words, and a third operand. The processor further includes a first execution unit coupled to the instruction decoder, in response to the first instruction, to perform multiple rounds of the first SKEIN256 MIX-PERMUTE operations based on the odd words and even words using a first rotate value obtained from a third storage location indicated by the third operand, and to store new odd words in the first storage location indicated by the first operand.