Are AI Therapy Chatbots Safe? New Study Raises Ethical Concerns

At a glance

AI chatbots fall short in mental health care: A study from Brown University found that large language models often fail to meet the ethical standards expected in professional psychotherapy.
Ethical risks in simulated therapy sessions: When tested in counselling scenarios, AI systems sometimes mishandled crises, reinforced harmful beliefs, and produced responses that appeared empathetic without true understanding.
Need for stronger oversight and standards: Researchers say clearer ethical guidelines, accountability, and regulation are needed before AI chatbots can be safely relied upon for mental health support.

As more people turn to tools like ChatGPT and other large language models (LLMs) for mental health advice, new research suggests these systems may not yet be ready to safely fill that role. A study by researchers at Brown University found that AI chatbots often fail to meet the ethical standards expected in professional psychotherapy, even when they are prompted to follow established therapeutic approaches.

Working with mental health professionals, the researchers tested AI systems in simulated counselling sessions. They found repeated patterns of problematic behaviour, including mishandling crisis situations, reinforcing harmful beliefs, and using language that creates the appearance of empathy without genuine understanding.

“In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counsellors violate ethical standards in mental health practice by mapping the model’s behaviour to specific ethical violations,” the researchers wrote in their study.

“We call on future work to create ethical, educational and legal standards for LLM counsellors — standards that are reflective of the quality and rigour of care required for human-facilitated psychotherapy.”

The findings were presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. To test the systems, trained peer counsellors conducted simulated sessions with AI models prompted to act as therapists. These included versions of OpenAI GPT Series, Claude and Llama. Three licensed clinical psychologists then reviewed the conversations and identified fifteen ethical risks, including failure to adapt advice to a person’s circumstances, biased responses, and inadequate handling of sensitive issues such as suicidal thoughts.

Lead author Zainab Iftikhar said the lack of accountability is a key concern compared with human therapy. “For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” she said. “But when LLM counsellors make these violations, there are no established regulatory frameworks.”

The researchers stress that AI tools could still help improve access to mental health support, particularly where cost or availability limits traditional care. However, they argue that stronger safeguards, clearer ethical standards and better oversight are needed before AI chatbots are relied upon in high-stakes mental health situations.