Methods and computer-storage media for automated generation of domain sitemap files are provided. A universal resource locator (URL) for a web site having a plurality of web pages associated therewith is received. Log files and permission controls are analyzed to ascertain whether each web page has been previously crawled and which web pages may be crawled and/or indexed. The permitted, not-previously-crawled web pages are subsequently crawled and the relational structure of the web site is ascertained. Other items of metadata, such as web page modification frequency or priority values, also are determined. Once the structure and metadata are available, a current sitemap is generated that provides the hierarchy and related details in the form of metadata. The sitemap file is then written to a disk and may then be sent to search engines as generated or in a compressed format.