The present invention relates to a method of detecting a candidate vector and a method of detecting a transmission symbol using the same. According to an embodiment of the present invention, in a multiple input multiple output (MIMO) system using spatial multiplexing (SM), a receiver selects candidate vectors corresponding to a layer located at a last row among a plurality of rearranged layers, and sequentially ranks constellation dots of a next layer for each of the selected candidate vectors. Then, a plurality of arbitrary constellation dots are selected from the ranked constellation dots, accumulated costs of the arbitrary constellation dots are calculated, and a candidate vector is selected in correspondence with a constellation dot having a minimal accumulated cost. Then, a new arbitrary constellation dot is selected in place of the constellation dot selected as the candidate vector, an accumulated cost of the selected arbitrary constellation dot is compared with accumulated costs of the other arbitrary constellation dots, and another candidate vector is selected.