Disclosed are an apparatus and a method of multi-phase pruning a neural network with multi-sparsity levels and an SIMD-based neural network pruning method, and the SIMD-based neural network pruning method according to an exemplary embodiment of the present disclosure includes GEMM-transforming an internode weight kernel applied to a layer in a neural network; and pruning the GEMM-transformed weight kernel with a predetermined SIMD width as a unit.