Some embodiments provide a method for estimating bandwidth for receiving data at a device during a communication session from a set of servers based on identities of the servers. During the session, the method receives a first set of data packets reflecting a first server identity. The method detects a change of server identity during the session based on a second set of received data packets that reflect a second server identity that is different from the first server identity. The method uses the second server identity to calculate a bandwidth estimate that accounts for the change based on a history of interactions during the session with the second server.