A method and system for authenticating IP source addresses by accessing one or more HTTP requests whose source client identifies itself as a legitimate web crawler. One or more IP addresses are detected from the one or more HTTP requests and each detected IP address is authenticated via a probability estimation regarding its association with a legitimate web crawler. A lookup table is preferably compiled for the authenticated IP addresses for reference, publication and authentication purposes.