A battery backup unit (BBU) for supplying backup power to a load includes at least one battery, an output terminal for coupling to a load, a power converter having an input coupled to the at least one battery and an output coupled to the output terminal, and a bypass circuit having an input coupled to the at least one battery and an output coupled to the output terminal. The power converter is configured to regulate a voltage at the output terminal after startup of the power converter. The bypass circuit is configured to regulate the voltage at the output terminal during startup of the power converter. Other example systems including one or more BBUs are also disclosed.