A retractable outside door handle assembly may include: an outside door handle protruded outward in a width direction of a vehicle rather than a door outer panel configuring a door of the vehicle or being receivable to an opening formed at the door outer panel; a first link having one end connected to one end of the outside door handle and the other end extending downward along a height direction of the vehicle; a second link having one end connected to one side and the other side of the outside door handle and the other end extending along the height direction of the vehicle; a main arm having one end rotatably connected to the door outer panel and the other end connected to the second link; and a moving block connected to the other end of the first link and the other end of the second link and movable upwards and downwards along the height direction of the vehicle.