Tech Desk
Jul 29, 2024
Reddit, one of the most popular social news and discussion websites, has blocked the Bing search, a Microsoft search engine, from scraping its content. This means that Google is the only search engine that can show new Reddit posts in its search results.
This significant move is part of Reddit's crackdown on web crawlers and its efforts to control how its content can be used by other websites.
What’s happening?
Reddit has recently updated its robots.txt file, which directs AI crawlers to random web pages on which they can go or can't glean information.
This change has prevented Bing search along with other search engines including the DuckDuckGo from scraping Reddit's pages with its AI bots, effectively walling off its content from search results.
Read more: Spotify's 'hi-fi' promise yet to become reality
However, Reddit spokesperson Tim Rathschmidt stated that these steps are not related to its recent partnership with Google, but rather a result of discussions with multiple search engines that were unable or unwilling to make enforceable promises regarding their use of Reddit content, including its use for AI.
You won't see any recent posts from Reddit using Bing or DuckDuckGo. Instead, you'll see older posts, or a message saying the site refuses to allow the content display.
What's next?
Unless an agreement is reached between Reddit and its competitors, Google is going to be the only place you can find Reddit content in its search results for a long time.
The move from Reddit accentuates the raging and never-ending war for control of Web content. As more sites try to protect their content, it may just get to a point where search engines struggle to provide results on an extensive basis.
In case you're affected, you can try searching for Reddit content on Google, or go directly to the site.
