Atlanta, GA | November 27, 2023 – Google announced the impending removal of the Googlebot Crawl Rate Limiter Tool from the search console last November 24, 2023, citing its diminishing necessity.
Scheduled for removal on January 8, 2024, Google is sunsetting the search console crawl rate limiter tool, attributing this decision to significant enhancements in crawling methodologies that have rendered the tool redundant.
Search Console’s Crawl Rate Limiter Tool
Debuting in 2008, the crawl rate limiter tool was a vital addition to the search console, enabling publishers to regulate Googlebot crawling and prevent server overload. Initially, certain publishers faced excessive crawling, leading to server strain and potential webpage delivery issues.
Addressing user concerns, Google incorporated this tool into the search console.
The tool’s primary function was to furnish Google with data. Requests to limit crawling took approximately a day to activate and remained effective for 90 days, as per Google’s information.
Reasons Behind Google’s Removal of the Rate Limiter Tool
Google’s announcement highlighted the advancements in crawling algorithms, empowering Googlebot to autonomously detect server capacity thresholds and promptly adjust the crawl rate accordingly.
Moreover, the tool’s infrequent usage was noted, with instances primarily setting the crawl rate to its lowest level whenever employed.
Moving ahead, the default minimum crawl rate will align more closely with the rates commonly requested by publishers.
Simplifying Search Console
The removal of this tool aims to streamline the search console, decluttering it from underutilized features. This streamlining effort intends to enhance the overall user experience within the search console interface.
Publishers experiencing ongoing issues with Googlebot’s crawl rate can still utilize the Googlebot report form to communicate feedback to Google.
The elimination of the Crawl Rate Limiter Tool reflects Google’s commitment to refining and simplifying its services, leveraging advanced algorithms to improve crawling efficiency.
