From CBC · · 1m 50s
AI Bots Threaten Library Access
AI data-scraping bots overwhelm university libraries, blocking public access to research and historical resources.
In brief
AI data-scraping bots overwhelm university libraries, blocking public access to research and historical resources. AI companies are using bots to scrape library collections for training data, causing site crashes and limiting public access to academic resources. Originally reported by CBC.
AI Bots Target Libraries
Academic libraries are fighting swarms of data-scraping bots that slow or crash sites for students, researchers and the public.
Bot Traffic Overwhelms Sites
University libraries report entire collections being knocked offline by persistent bot streams trying to access archives.
Finnish Music Searches
One university received over 11,000 searches for Finnish music from thousands of internet addresses in a single day.
Bots Feed AI Models
The bots are used by ai companies to scour websites and retrieve data for large language models like claude, gemini and chatgpt.
Libraries as Targets
Libraries, museums and cultural institutions are prime targets because they have quality, vetted information not behind paywalls or logins.
Impact on Institutions
The bots cause significant work and headaches for institutions digitizing information and making it openly available.
“they're just out there on the internet, just bombarding us with trying to grab up as much material as possible”
Bot Traffic Causes Downtime
Sometimes bot traffic gets so bad that library sites go down completely for a little while.
Public Access at Risk
Regular users now effectively compete with infinite bot traffic for library website access.
Good vs. Bad Bots
Search engines like google and archiving sites follow robots.txt rules, but ai scraping bots often ignore them.
Bots Overwhelm Systems
AI scraping bots sometimes request thousands of documents per second and download them in every available format.
Project Gutenberg Concerns
Project Gutenberg says its materials are too public domain for bots to use respectfully.
“our material is almost all public domain material. it's exactly the sort of material that they should be training on”
AI Firms' Response
Major ai firms support a standardized opt-out signal in robots.txt to block web crawlers from harvesting data.
In short
AI companies are using bots to scrape library collections for training data, causing site crashes and limiting public access to academic resources.
