An error compensation system for a numerical control (NC) machine tool based on iterative learning control, including a trajectory generating module, a down-sampling module, a position controller, a first holder, a velocity-loop iterative learning controller, a velocity controller, a second holder and a control plant. The trajectory generating module is configured to generate a desired trajectory command including a first sampling command. The first sampling command is transmitted to the down-sampling module and the velocity-loop iterative learning controller. The first sampling command is down-sampled through the down-sampling module to obtain a second sampling command. The velocity-loop iterative learning controller is configured to receive the first sampling command, and obtain a first sampling error compensation sequence according to a first sampling error sequence and a first sampling error compensation sequence of a previous iteration machining process stored therein. An error compensation method is also provided herein.