A holding structure and a portable electronic apparatus therewith are disclosed. The holding structure includes a mount structure and a movable member. The mount structure includes a constraining portion having an accommodating space and an opening. The movable member includes a carrier having an engaging structure. When an end portion of a pencil object is inserted into the constraining portion from the opening to be accommodated together with the carrier in the accommodating space, the constraining portion constrains deformation of the engaging structure such that the engaging structure is engaged with the end portion. The end portion is therefore hardly extracted out. When the carrier departs from the accommodating space, the engaging structure and the end portion are disengaged. The end portion is therefore easily extracted out. The invention uses structure to constrain deformation, which can provide stable holding strength without any metal spring.