The present disclosure relates to systems and methods for providing block-wise sparsity in neural networks. In one implementation, a system for providing block-wise sparsity in a neural network may include at least one memory storing instructions and at least one processor configured to execute the instructions to: divide a matrix of weights associated with a neural network into a plurality of blocks; extract non-zero elements from one or more of the plurality of blocks; re-encode the extracted non-zero elements as vectors with associated coordinates of the extracted non-zero elements within the one or more blocks; enforce input sparsity in the neural network corresponding to the associated coordinates; and execute the neural network using the vectors and the enforced input sparsity.