A distributed load balancer in which a router receives packets from at least one client and routes packet flows to multiple ingress servers. For unknown packet flows, an ingress server cooperates with primary and secondary flow trackers to establish connections to server nodes. For known packet flows, the ingress server sends the packets to target server nodes. The server nodes randomly select egress servers for outgoing packets of the packet flows. The ingress servers, flow trackers, and egress servers are implemented by multiple load balancer nodes in a load balancer node layer. The ingress and egress servers for a given packet flow may be on different load balancer nodes. The load balancer nodes may use a consistent hash function to compute a consistent hash ring for the nodes according to packet flow client/public endpoint pairs so that nodes associated with given packet flows can be located.