Apple Confirms Existence of Applebot In-House Web Crawler

[Maura]

Apple has finally confirmed the existence of its web-crawling service, which was first discovered last November by a web developer, and outlined some of its details in a newly updated support document.

The crawler is called AppleBot, and Apple says that it mainly serves Siri and Spotlight. It was previously thought that Apple used existing search engines such as Bing or Google to route queries.

AppleBot works in a similar way to Google’s Googlebot, and according to Apple it will adhere to Googlebot instructions if there are no AppleBot instructions available.

Apple says that website operators can contact Apple’s network operations center if they have any questions about AppleBot.

While AppleBot was first noticed last November, it is not known how long the web crawler has actually been in operation. AppleInsider notes that the support document did not surface until February, and was only updated with content in April.

Source: Apple details new AppleBot web crawler used by Siri and Spotlight