

That said, there may be instances in which crawlers/spiders converge on a page that is erroring out ( 502s in the example below).


Since search-indexing is desirable for most sites, tread carefully in order to avoid wreaking havoc on a site's SEO. Some legitimate bots/crawlers/proxies (such as BingBot or AdsBotGoogle) will identify themselves.
