A portable accessory device includes a housing having a mounting slot, an accessory unit supported in the housing, and two adjustable retainers for adjusting the size of the mounting slot for a shaft of the outdoor umbrella. Each of the adjustable retainer includes a retention arm, having a pusher surface facing towards the mounting slot, slidably mounted at the housing, wherein the pusher surfaces of the retention arms are facing with each other and are arranged for biasing against an outer surface of the shaft of the outdoor umbrella until the shaft thereof being fitted at the mounting slot so as to substantially mount the housing at the shaft of the outdoor umbrella. Therefore, the portable accessory device is adapted to detachably mount at the outdoor umbrella to provide an additional function via the accessory unit for users to have high quality outdoor activities.