Searched for
BLOCKING AI CRAWLERS
Cloudflare accuses Perplexity of using stealth crawling techniques to evade network blocksIn a blog post, Cloudflare alleged that Perplexity initially uses a declared user agent for its bots, and later switches to undeclared, gen...05 Aug, 2025, 10.55 AM IST- AI search pushing an already weakened media ecosystem to the brinkGenerative AI tools like ChatGPT are reducing traffic to news websites by providing direct summaries, threatening media revenue models. As ...04 Aug, 2025, 08.54 AM IST
- Cloudflare introduces default blocking of AI data scrapersCloudflare has introduced a new setting allowing websites to block AI bots from scraping content without permission. The move aims to prote...02 Jul, 2025, 09.50 AM IST
- Cloudflare launches tool to help website owners monetise AI bot crawler accessThe tool allows website owners to choose whether artificial intelligence crawlers can access their material and set a price for access thro...01 Jul, 2025, 03.41 PM IST
- Companies alert as along come AI web spidersAI crawlers are computer programs that collect data from websites to train large language models. Enterprises are increasingly blocking AI ...15 Dec, 2024, 06.00 AM IST
- Canadian news media are suing OpenAI for copyright infringement, but will they win?The lawsuits claim that OpenAI "scraped" large amounts of content from media sites without permission. They have also claimed that the AI c...02 Dec, 2024, 01.39 PM IST
- The data that powers AI is disappearing fastWeb data restrictions impact AI models like ChatGPT. MIT's study finds 25% of top-quality data restricted. Smaller firms use synthetic data...22 Jul, 2024, 01.41 PM IST
- Reddit to update web standard to block automated website scrapingAI startups face scrutiny for bypassing Reddit's updated scraping rules. Plagiarism accusations against firms like Perplexity highlight the...26 Jun, 2024, 09.17 AM IST
- Multiple AI companies bypassing web standard to scrape publisher sites, licensing firm saysPerplexity likely bypassed web crawler blocks via the Robots Exclusion Protocol, as reported by Wired, using analytics to track AI traffic.21 Jun, 2024, 08.47 PM IST
- ETtech Explainer: OpenAI’s response to NYT’s copyright lawsuitOpenAI has refuted some of these allegations and claimed that the fair use doctrine of copyright laws will apply in its case. ET explains t...13 Jan, 2024, 09.59 AM IST
- ETtech Explainer | NYT vs OpenAI: Why news publishers are fighting Big Tech over LLMsLast week’s copyright infringement lawsuit by The New York Times ( NYT) against ChatGPT- maker OpenAI has opened another battlefront betwee...01 Jan, 2024, 06.01 AM IST
- As Google pushes deeper into AI, publishers see fresh challengesSince May, Google has begun rolling out a new form of search powered by generative AI, after industry observers questioned the tech giant's...19 Oct, 2023, 10.43 PM IST
- As Google pushes deeper into AI, publishers see fresh challengesSince May, Google has begun rolling out a new form of search powered by generative AI, after industry observers questioned the tech giant's...19 Oct, 2023, 04.01 PM IST
- BBC blocks OpenAI data scraping, to harness Generative AIThe BBC's director of nations, Rhodri Talfan Davies, believes that the use of artificial intelligence (AI), specifically Generative AI (Gen...07 Oct, 2023, 10.35 AM IST
- All the content that's fit to be paid forML deepens the trench between online content and its distribution. Publishers have been seeking a larger share of online advertising revenu...28 Aug, 2023, 10.58 PM IST
Load More
