A receiver of a multiple-input multiple-output (MIMO) system performs QR decomposition of the channel matrix to enable detection of a transmitted vector in a layered manner. In each layer, a sub-vector of the transmitted vector is estimated. A reactive tabu search is performed if an estimated symbol differs from a nearest symbol in the alphabet by a predetermined value. The receiver may order the entries of the channel matrix prior to QR decomposition to enable estimation in an optimum order. In another embodiment, a receiver performs multiple reactive tabu searches to estimate a transmitted vector. The receiver employs a fixed threshold or a variable threshold for a cost function used in the multiple reactive tabu searches depending on whether the MIMO system is under-determined or not. The techniques enable low bit-error rate (BER) performance in MIMO systems with large number of antennas and when higher-order modulation techniques are used.