Your preferred AI chatbot may not be completely honest.

      AI search tools are gaining popularity, with one in four Americans reporting they use AI instead of conventional search engines. However, it is crucial to note that these AI chatbots do not always deliver precise information.

      A recent study by the Tow Center for Digital Journalism, as reported by the Columbia Journalism Review, reveals that chatbots often have difficulty retrieving and accurately citing news content. Even more concerning is their tendency to fabricate information when the correct answer is not available.

      The AI chatbots evaluated in this survey included some of the most recognized names, such as ChatGPT, Perplexity, Perplexity Pro, DeepSeek, Microsoft’s Copilot, Grok-2, Grok-3, and Google Gemini.

      In the assessments, the AI chatbots were given direct excerpts from 10 online articles published by various sources. Each chatbot received 200 queries, covering 10 articles across 20 different publishers, totaling 1,600 queries. The chatbots were tasked with identifying the article headline, the original publisher, the publication date, and the URL.

      In similar tests with traditional search engines, correct information was successfully provided. However, the performance of AI chatbots was notably less effective.

      The findings showed that chatbots often have trouble declining questions when they cannot provide accurate answers, frequently offering incorrect or speculative replies instead. Premium chatbots often delivered confidently incorrect answers more frequently than their free versions. Furthermore, many chatbots seemed to ignore the Robot Exclusion Protocol (REP) preferences that websites use to communicate with web robots like search engine crawlers.

      The survey also indicated that generative search tools were prone to fabricating links and citing syndicated or copied versions of articles. Additionally, content licensing agreements with news sources did not ensure accurate citations in chatbot responses.

      What can you do?

      The most significant takeaway from this survey is not just that AI chatbots commonly provide incorrect information, but that they do so with a concerning level of confidence. Rather than admitting uncertainty, they often respond with phrases such as “it appears,” “it’s possible,” or “might.”

      For example, ChatGPT misidentified 134 articles but signaled uncertainty only 15 times out of 200 responses and never refrained from providing an answer.

      Given the survey findings, it is advisable not to depend solely on AI chatbots for answers. Instead, using a combination of traditional search methods and AI tools is recommended. At the very least, consulting multiple AI chatbots could prove helpful. Otherwise, you risk receiving inaccurate information.

      Looking ahead, it would not be surprising to see a consolidation of AI chatbots, with the better-performing ones distinguishing themselves from those of lower quality. Eventually, their results are likely to match the accuracy of traditional search engines. When that will occur is uncertain.