The present disclosure proposes an image-based method for measuring elasticity of biological tissues including following steps: obtaining N successive grayscale images of a testing biological tissue, where N is a positive integer; obtaining regions of interest (ROI) of the grayscale images; dividing the regions of interest into a grid of small sections; calculating relative displacement vector of each interrogation window using texture matching method; and calculating elastic modulus of each interrogation window according to the relative displacement vectors. The present disclosure also proposes an image-based system for measuring elasticity of biological tissues. The present disclosure applies to various resolutions of grayscale images obtaining in ultrasound imaging, optical imaging, photoacoustic imaging, CT imaging, magnetic resonance imaging ect. The system can be integrated into a traditional clinical imaging system as an image post-processing software module to characterize tissue elasticity. Because it's no need to update the traditional clinical imaging system hardware, updating cost is low. It's easy to be accepted by hospital and is convenient to popularize.