Reddit has recently started blocking search engines that do not have a commercial agreement with the company. This strategic move is part of Reddit's effort to control how its content is used and to generate new revenue streams.
Google's $60 Million Deal with Reddit
In February, Google entered a $60 million expanded partnership with Reddit. This deal allows Google to use Reddit data to improve its AI models. Additionally, Reddit content will now be more prominently displayed on Google Search. This means that users will find it easier to engage with Reddit’s discussions and communities when they search for related topics on Google.
Real-Time Reddit Data for Google
One significant aspect of this partnership is that Google now has access to the Reddit Data API. This API provides real-time content from Reddit, which is a valuable asset for Google’s AI advancements.
Blocking Other Search Engines
According to 404 Media, Reddit is blocking its content from appearing on other search engines such as Bing, DuckDuckGo, and Mojeek. As a result, users on these platforms may either not see recent Reddit content or may find incomplete search results.
Microsoft’s Reaction
Microsoft, which owns the Bing search engine, has responded to this move. Caitlin Roulston, a Microsoft spokesperson, said that Microsoft respects the robots.txt standard—a file that tells web crawlers which pages they can or cannot access on a site. She mentioned that Bing stopped crawling Reddit when Reddit updated its robots.txt file on July 1.
Reddit's Post-IPO Strategy
Reddit’s decision to block major search engines could indicate its new strategic direction since going public earlier this year. By restricting access to its content, Reddit is exploring new ways to increase its revenue. This move could push other search engines to enter commercial agreements with Reddit if they want to include Reddit content in their search results.
Reddit’s Explanation
Despite the speculations, a Reddit spokesperson, Tim Rathschmidt, clarified the situation to The Verge. He stated, “This is not at all related to our recent partnership with Google.” Rathschmidt explained that Reddit has been in discussions with several search engines but couldn’t reach agreements with all of them. The primary issue is that some search engines are unable or unwilling to make enforceable promises about how they will use Reddit content, particularly for AI training.
Changes to Reddit’s Robots.txt File
The changes to Reddit’s robots.txt file are aimed at all crawlers that do not agree to specific terms regarding the use of Reddit data for AI purposes. Reddit mentioned that some web crawlers like Internet Archie and reddit4research still work because they adhere to Reddit's guidelines.
Conclusion
Reddit’s decision to block search engines that don’t have a commercial agreement marks a significant shift in how the company manages its content. By prioritizing partnerships that offer clear and enforceable terms, Reddit aims to protect its data and explore new revenue opportunities. This move also underscores the evolving relationship between tech giants and content platforms as they navigate the complex landscape of data usage and monetization.