Amazon Investigates Perplexity AI Web Scraping Allegations
Retail Wire
JUNE 29, 2024
Amazon Web Services (AWS) has initiated an investigation into Perplexity AI following allegations that the company’s web crawler is bypassing the Robots Exclusion Protocol. This protocol allows web developers to control how search engines and other bots interact with their websites through a robots.txt file. While adherence to these guidelines is voluntary, reputable crawlers have generally respected them since the protocol's inception in the 1990s.
Let's personalize your content