Battle of the AI chatbots: Claude outperforms his rivals, ChatGPT disappoints

An evaluation conducted by the University of Applied Sciences Northwestern Switzerland reveals uneven performance among the leading artificial intelligence chatbots. While Anthropic's Claude comes out on top, ChatGPT, despite being the market leader, performs disappointingly. SRF's “Kassensturz” programme aired these surprising findings.

Suisse-blog has already published a IA chatbot comparison

Surprising mistakes on local issues

Discount code: ZNVC5

Ordering a “Stange” in a Swiss-German restaurant and being told that it's an elongated pastry that you can personalise? This was the aberrant answer given by two chatbots during the test. In reality, a “Stange” refers to a mug of beer. These “hallucinations” - the term used to describe invented answers - occurred frequently during the experiment.

The researchers subjected ten chatbots for the general public to more than 300 questions covering everyday life, general culture, law and health. A fifth was specifically about Switzerland. A committee of experts then analysed the results.

Switzerland's chatbot at the back of the queue

Simon Felix, an AI specialist at the FHNW, notes that a number of chatbots struggle particularly with Swiss questions, producing either inventions or German-centric answers.

Lumo, developed by Swiss company Proton, comes last. Despite its impeccable data protection, the quality of its answers remains problematic. Simon Felix observes that Lumo oscillates between ultra-brief three-word responses and interminable developments, sometimes completely invented. China's Deepseek and Meta's chatbot also score poorly.

ChatGPT: mediocre results despite its popularity

The use of language models is exploding. ChatGPT dominates, with around 800 million weekly users according to OpenAI. Yet its performance is no better than average. “It avoids catastrophic responses, but never achieves excellence”, comments Simon Felix. The model also shows its limitations when it comes to Switzerland's specific characteristics.

Discount code: ZNVC5

Anthropic's Claude tops the charts

The Claude chatbot, created by Anthropic - a company founded by former OpenAI researchers - takes the cake: “It gets straight to the point with concise answers that leave out nothing crucial”, stresses the expert.

Microsoft's Copilot and Google's Gemini also scored well. Intriguing fact: Copilot uses the same technology as ChatGPT but delivers superior results. Simon Felix explains that this is due to Microsoft's more elaborate guidelines and greater computing power, illustrating the importance of suppliers customising their models.

Caution recommended for delicate subjects

The expert warns against using chatbots for certain topics. These tools are only suitable for questions where mistakes will not have serious consequences. “For medical, psychological or legal questions, the wrong answer can have dramatic repercussions”, warns Simon Felix.

The protection of personal data also remains paramount. Private information, sensitive names or confidential documents must never be entered into these systems.