Artificial intelligence web crawlers are running amok

Artificial intelligence tech companies are refusing to abide by internet protocol when it comes to scraping data. Their ravenous scavenging behavior is upending the basic rules of the internet.


On every website, there’s a message that contains a hidden stop sign. It’s intended for bots, not humans, a way of saying, do not scan this part of the website. The artificial intelligence industry is ignoring these stop signs, and understanding why sheds light on how AI companies are turning the web upside down. NPR’s Bobby Allyn reports.

BOBBY ALLYN, BYLINE: The story starts in the mid-’90s, the days of

→ Continue reading at NPR - Technology

More from author

Related posts


Latest posts

Elephants, Michelin stars and Mughal jewels: Indian weddings are a $130 billion business | CNN Business

New Delhi CNN  —  The wedding of the year is here. In less than a week, Anant Ambani,...

Diaper change: Japan’s aging society is transforming the baby care business. | CNN Business

Hong Kong CNN  —  The world’s population is getting older in what the United Nations calls an “irreversible global...

Listen to ‘The Wonderful Wizard of Oz’ Read By Dorothy Herself Thanks to a New AI App That Cloned Judy Garland’s Voice

Judy Garland passed away 55 years ago — but put any document, e-book, or file into a new AI app, and her voice will...