A system and method for storing and retrieving a sparse matrix from memory of a computing device while minimizing the amount of data stored and costly jumps in memory. The computing device may be an FPGA having memory and processing elements. The method comprises storing non-zero data elements of the matrix in a data array and storing their corresponding column address values in a column index array. To read this stored data from memory, each preceding value of the column index array may be compared with each current value of the column index array to determine if the data array value corresponding with the current column index array value belongs on the next row of the matrix. The method may include pre-ordering the matrix with zero-pad placeholders or creating a row increment pointer array which typically stores fewer values than the number of rows in the matrix.