A system and method for efficient sparse matrix processing are provided in one embodiment. A compressed representation of a sparse matrix, the sparse matrix including one or more non-zero entries in one or more of a plurality of portions of the matrix, is obtained by at least one server including one or more streaming multiprocessors, each of the streaming multiprocessors including one or more graphics processing unit (GPU) processor cores. Each of the portions are assigned into one of a plurality of partitions based on a number of the non-zero entries in that portion. For each of the partitions, a predefined number of the GPU processor cores are assigned for processing each of the portions assigned to that partition based on the numbers of the non-zero entries in the portions assigned to that partition. For each of the partitions, each of the portions associated with that partition are processed.