Meta's crawlers overwhelm small sites

Meta Platforms has faced scrutiny after announcing changes to its terms allowing AI crawlers to run across business accounts. Website managers say Meta has been doing this for years.

Chelsea Paulus's website gets hit

California-based artist Chelsea Paulus says Meta's crawlers crashed her website last week. A bot called Meta-ExternalAgent got stuck in a loop on her remove from cart function.

The bot creates massive traffic

Meta-ExternalAgent created more than a hundred hits per minute, growing to about five hundred fifty thousand hits over the next few days. Paulus's website CPU hit eighty percent.

Her host warns of total failure

Paulus's web host warned that her site could go down at one hundred percent CPU usage. She blocked Meta's bot, but two days later another arrived.

A second bot causes total crash

Meta-WebIndexer showed up and brought Paulus's CPU to one hundred percent, crashing her website entirely. She was unable to serve customers or conduct business.

Why Meta crawls the web

Meta-ExternalAgent crawls the web for training foundation AI models and improving products by indexing content directly, according to Meta's web crawler page.

“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”

Meta-WebIndexer's stated purpose

Meta-WebIndexer navigates the web to improve Meta AI search result quality for users. Meta analyzes online content to enhance the relevance and accuracy of Meta AI.

“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users.”

Small businesses left to deal alone

Website managers say Meta has been running crawlers on websites for years, with small online businesses often left to deal with the repercussions of the traffic they generate.

Entrepreneurs pushed to the brink

Entrepreneurs who rely on their websites to connect with customers and sell products are being inundated with bots, pushing their businesses to the brink of failure.

Terms change announced by Meta

Meta announced upcoming changes to its terms that essentially allow AI crawlers to run wild across business accounts, intensifying scrutiny of the company's practices.