An image scaling system includes a window memory, a longitudinal scaler, a buffer and a latitudinal scaler. The window memory temporarily stores partial data of N image lines. The longitudinal scaler performs a longitudinal scaling operation on the partial data of the N image lines to thereby produce partial data of a longitudinal image line. The latitudinal scaler performs a latitudinal scaling operation on the partial data of the longitudinal image line to thereby produce scaling-part data of a latitudinal image line. The longitudinal scaler and the latitudinal scaler use a window average to perform an image shrinking operation and a bi-linear interpolation average to perform an image expanding operation.