Scientists discovered that Microsoft’s chatbot on Copilot gave incorrect and confusing details about European elections. AlgorithmWatch, a human rights group, reported that when it asked Bing Chat (now called Copilot) about recent elections in Switzerland and the German states of Bavaria and Hesse, approximately one-third of the responses contained factual mistakes. Moreover, they found that safety measures were not consistently implemented.
The organization gathered Bing responses from August to October of this year. They selected three elections because these were the first ones in Germany and Switzerland since Bing’s introduction. This choice also enabled researchers to examine local contexts and compare responses in different languages: German, English, and French.
Bing’s Copilot Chatbot, Issues with Election
The researchers posed inquiries about fundamental details such as voting procedures, the list of candidates, polling statistics, and even included prompts related to news reports. Following these, they asked questions about candidate stances, political matters, and, in the instance of Bavaria, scandals affecting the campaign.
AlgorithmWatch categorized responses into three groups: answers with factual errors, varying from misleading to nonsensical, evasions, where the model declined to answer or redirected by claiming incomplete information and completely accurate responses. The organization also observed that some answers exhibited political bias, with Bing presenting information in a way that favored a particular party’s framing or language.
Bing’s replies featured false controversies, inaccurate election dates, incorrect polling figures, and, in some instances, candidates who weren’t actually participating in the elections. These responses with errors comprised 31 percent of the total answers.
The report stated that even when the chatbot extracted polling numbers from a single source, the reported numbers in the answer frequently differed from the linked source. In some instances, it ranked political parties in a different order than the sources did.
Microsoft, the company behind Bing/Copilot, has implemented safety measures on the chatbot. These guardrails are designed to prevent Bing from providing harmful, false, or offensive responses. Typically, AI guardrails often result in the chatbot refusing to answer a question to ensure compliance with the company’s rules. In the test, Bing chose to avoid answering questions 39 percent of the time. This meant that only 30 percent of the answers were considered factually correct.
Bing’s Safety Guidelines and Microsoft’s Response
During its investigation, AlgorithmWatch noted that Bing adhered to safety guidelines when questioned for opinions but not when asked for facts. In instances where factual information was sought, Bing went to the extent of making significant false allegations of corruption, presenting them as if they were factual.
Additionally, the group pointed out that Bing’s performance was less satisfactory in languages other than English.
In a statement, Microsoft mentioned that it has implemented measures to enhance its conversational AI platforms, with a specific emphasis on improvements before the 2024 elections in the United States. This involves prioritizing authoritative sources of information for Copilot.
“We are taking a number of concrete steps in advance of next year’s elections, and we are committed to helping safeguard voters, candidates, campaigns, and election authorities,” said Microsoft spokesperson Frank Shaw.
He said that Microsoft urges individuals “to exercise their best judgment when reviewing results on Copilot.”
The apprehension regarding the potential of AI to mislead voters during elections is acknowledged. Microsoft (which is founded by Bill Gates), in November, expressed its intention to collaborate with political parties and candidates to restrict deepfakes and counter election misinformation.
In the United States, legislative bills have been proposed, mandating campaigns to disclose AI-generated content, and there is a possibility that the Federal Election Commission may impose restrictions on AI advertisements.
Conclusion
The investigation exposed serious issues with Microsoft’s Copilot Chatbot, revealing inaccuracies and biases in its responses to European election questions. The findings highlight the need for better safety measures and addressing language-related issues. Microsoft’s commitment to improvement, especially before the 2024 U.S. elections, shows a proactive stance in ensuring accurate information and building voter trust.
