A method of forming a semiconductor device includes forming a plurality of semiconductor fins from an upper semiconductor layer located on a first region of a bulk semiconductor substrate of a structure and then forming at least one gate structure straddling a portion of semiconductor fins. A portion of the lower semiconductor layer from beneath the upper semiconductor layer is then removed to form a vertical semiconductor portion which contacts the bulk semiconductor substrate and at least one of the semiconductor fins. A dielectric layer (e.g., a spacer layer) is then deposited over the structure and laterally surrounds the vertical semiconductor portion such that semiconductor fins and the at least one gate structure are partially isolated from the first region of the bulk semiconductor substrate by the dielectric layer.