What is disclosed is a system and method for determining respiration rate from a video of a subject. In one embodiment, a video is received comprising plurality of time-sequential image frames of a region of a subject's body. Features of pixels are extracted from that region from each image frame and vectors formed from these features. Each image frame has an associated feature vector. A N×M video matrix of the vectors of length N is constructed such that a total number of columns M in the video matrix correspond to a time duration over which the subject's respiration rate is to be determined. The video matrix is processed to obtain a matrix of eigenvectors where principal axes of variations due to motion associated with respiration are contained in a first few eigenvectors. One eigenvector is selected from the first few eigenvectors. A respiration rate is obtained from the selected eigenvector.