A mounting structure of a handle switch device with which a panel can be replaced by manipulating fasteners for fastening two outer casings of the handle switch device together. A rear casing is temporarily mounted to an inner casing that is mounted to a handle pipe, engagement pieces of a design panel are inserted through through holes of a front casing, and the front and rear casings are fastened together by mounting screws, thereby completing assemblage of the handle switch device. To replace the design panel, the mounting screws are loosened to bring them into a half-engagement state and the front casing is displaced toward a driver side to disengage projection ribs of the inner casing from the through holes. Next, the design panel is dismounted from the front casing, a new design panel is mounted to the front casing, and the mounting screws are tightened.