The invention discloses a moving object multi-scale tracking method and device. A neural network is constructed, and weight training is carried out layer by layer to acquire an initialized weight. The truth value of a first video frame is input into the established neural network to update the initial weight. Starting from a second video frame, an offset term is calculated for the input video frame. The output value of the video frame input into the neural network is calculated according to the acquired offset term of the video frame. Whether confidence level output by the neural network in the video frame is less than a preset threshold value is judged. The weight of the neural network is updated if the confidence level is less than the preset threshold value. The moving target of the video frame is estimated according to the neural network with the updated weight. If the confidence level is greater than the preset threshold value, the moving target of the video frame is directly estimated. According to the moving object multi-scale tracking method and device, accurately feature extraction of the moving object is realized; the position and size of the moving object are predicted; and the best moving object tracking result is acquired.