An operational object controlling device including a motion detection unit, a motion obtaining unit, a motion feature quantities extraction unit, a template storage unit, an operational object motion storage unit, a motion feature quantities transform unit and an operational object motion obtaining unit. The motion obtaining unit obtains the user's motion detected by the motion detecting unit. The motion feature quantities extraction unit extracts the user's motion feature quantities from the obtained motion. The transform unit transforms the motion feature quantities by using a template obtained from the template storage unit. The motion feature quantities of the operational object are obtained from each of the temporal motion sequences of the operational object in the operational object motion storage unit. The operational object motion obtaining unit obtains one of the temporal motion sequences from the storage unit having the feature quantities close to the user's motion feature quantities.