That said, there may be instances in which crawlers/spiders converge on a page that is erroring out ( 502s in the example below).
Since search-indexing is desirable for most sites, tread carefully in order to avoid wreaking havoc on a site's SEO. Some legitimate bots/crawlers/proxies (such as BingBot or AdsBotGoogle) will identify themselves.
0 kommentar(er)
