The invention belongs to the technical field of application of nuclear quadrupole resonance (NQR) and geomagnetic field nuclear magnetic resonance systems, provides a NQR detection system and an antenna thereof. The antenna comprises a first coil which forms a first pattern; a second coil which forms a second pattern; and a gradient antenna composed of the first coil and the second coil and used for receiving radio-frequency interference from target area signals and the outside; wherein the first coil and the second coil are located on the same plane, and the winding directions of the first coil and the second coil are opposite, the second pattern is evenly distributed around the first pattern, and the area of the first pattern is equal to the that of the second pattern. The capacity of the antenna for suppressing radio interference in the environment when the antenna is used for detecting the NQR or applied to the ground magnetic resonance can be effectively improved. The signal detection capability is enhanced, so that the signal noise ratio of the measurement system is improved.