A transpose memory circuit is provided which comprises a number of dual port memory blocks each having a plurality of storage cells each configured for storing one or more data word. The dual port memory blocks form a storage array for storing at least one input matrix and outputting the at least one input matrix in transposed form. A data input is provided to receive a plurality data words on each cycle and a data output is provided to output a plurality of data words on each cycle. A read address logic is provided to generate read addresses such that one cell of each dual port memory block can be read out on each cycle. A write address logic is provided to generate write addresses such that one cell k of each dual port memory block can be written on each cycle. In each cycle, one storage cell of each dual port memory block is addressed by the read address logic. The data words stored in the addressed storage cells are read out from one dual port memory block and outputted through the data output. In each cycle, one storage cell of each dual port memory block is addressed by the write address logic, where the addressed storage cells have been read out in a preceding cycle and into which storage cells data words received through the data input are written. The transpose memory circuit is provided to receive an input matrix in cycles and to output the transposed input matrix in next cycles without any dead cycles interposed between them.