Vanessa Fox - Redmond WA, US Amanda Ann Camp - Kirkland WA, US Maximilian Ibel - Pfaeffikon, CH Patrik Rene Celeste Reali - Zurich, CH Jeremy J. Lilley - Mountain View CA, US Katherine Jane Lai - Cambridge MA, US Ted J. Bonkenburg - Mountain View CA, US Neal Douglas Cardwell - San Francisco CA, US
Assignee:
Google Inc. - Mountain View CA
International Classification:
G06F 17/30
US Classification:
707713, 709225
Abstract:
Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.
System And Method For Enabling Website Owner To Manage Crawl Rate In A Website Indexing System
Vanessa Fox - Redmond WA, US Amanda Ann Camp - Kirkland WA, US Maximilian Ibel - Pfaeffikon, CH Patrik Rene Celeste Reali - Zurich, CH Jeremy J. Lilley - Mountain View CA, US Katherine Jane Lai - Cambridge MA, US Ted J. Bonkenburg - Mountain View CA, US Neal Douglas Cardwell - San Francisco CA, US
Assignee:
Google Inc. - Mountain View CA
International Classification:
G06F 17/30
US Classification:
707709, 707E17108, 707802, 709224, 709226
Abstract:
Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.
System And Method For Enabling Website Owners To Manage Crawl Rate In A Website Indexing System
Vanessa Fox - Redmond WA, US Amanda Ann Camp - Kirkland WA, US Maximilian Ibel - Pfaeffikon, CH Patrik Rene Celeste Reali - Zurich, CH Jeremy J. Lilley - Mountain View CA, US Katherine Jane Lai - Cambridge MA, US Ted J. Bonkenburg - Mountain View CA, US Neal Douglas Cardwell - San Francisco CA, US
Assignee:
Google Inc. - Mountain View CA
International Classification:
G06F 17/30
US Classification:
707 3, 709224
Abstract:
Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.