A dummy gate structure straddling at least one semiconductor fin is formed on a substrate. Active semiconductor regions and raised active semiconductor regions may be formed. A planarization dielectric layer is formed over the at least one semiconductor fin, and the dummy gate structure is removed to provide a gate cavity. Electrical dopants in the channel region can be removed by outgassing during an anneal, thereby lowering the concentration of the electrical dopants in the channel region. Alternately or additionally, carbon can be implanted into the channel region to deactivate remaining electrical dopants in the channel region. The threshold voltage of the field effect transistor can be effectively controlled by the reduction of active electrical dopants in the channel region. A replacement gate electrode can be subsequently formed in the gate cavity.