Latency-based timeouts are used for concurrent security processing by multiple in-line network security tools. A network system forwards secure network packets to the tools and uses latency-based timeouts with respect to the return of processed packets from the tools. Initially, the network system measures processing latencies for the tools and sets at least one timeout threshold based upon the processing latencies. The network system then receives an input packet from a network source, generates a timestamp, concurrently sends an output packet to the tools based upon the input packet, tracks return packets from the tools, and determines whether a timeout has occurred with respect to the timeout threshold based upon a difference between the timestamp and a current timestamp. If a timeout does not occur, a secure packet is forwarded to a network destination. If a timeout does occur, return packet tracking for the input packet is ended.