A method for detecting a space-time block code is provided. The method includes randomly selecting one initial candidate layer (xN) with respect to a signal vector {tilde over (y)}, applying a DF algorithm to first J number of layers (xN−1, xN−2, . . . , xN−J) with respect to all the available candidate symbols within the initial candidate layer (xN) to generate candidate symbols of each of the J number of layers, re-arranging the J number of layers, selecting the lowest layer of the re-arranged J number of layers as a new candidate layer, performing a DF process on the other remaining layers, excluding the new candidate layer, to generate N-dimensional candidate symbol vectors xi, and performing a maximum likelihood detection on the xi to detect an N-dimensional input vector {circumflex over (x)}.