Provided are a control apparatus, an operation controlling method and a non-transitory computer-readable storage medium storing an operation controlling program. The control apparatus is provided for controlling a display panel which allows an operator to give an instruction at an instruction position by performing a touch operation. The control apparatus includes: an instruction position setting section configured to set an instruction position based on a touch position; a mode determining section is configured to determine a mode for the instruction position setting, by defining a first area and a second area and choosing between a first mode to set the instruction position at the same position as the touch position and a second mode to set the instruction position at a position being away from the touch position, according to which of the first and second areas includes the touch position; and a processing section configured to execute the object processing according to the instruction position.