ChatGPT offers harmful advice to teens

WASHINGTON (AP) — ChatGPT will tell 13-year-olds how to get drunk and high, instruct them how to hide eating disorders and even write a heartbreaking suicide note to their parents if they ask , according to a watchdog group's investigation.

The Associated Press reviewed more than three hours of interactions between ChatGPT and researchers posing as vulnerable teens. The chatbot generally provided warnings against risky activities, but then offered surprisingly detailed and personalized plans for drug use, calorie-restricted diets, and self-harm.

Researchers at the Center for Countering Digital Hate ( CCDH ) repeated the queries on a large scale, classifying more than half of ChatGPT's 1,200 responses as dangerous.

“We wanted to test the safety barriers,” explained Imran Ahmed, the group’s CEO.

“The initial gut response is, 'Oh my God, there are no safety barriers.' The barriers are completely ineffective. They're barely there, if anything, a fig leaf,” he lamented.

After learning of the report, OpenAI , the company that created ChatGPT, said it is working to refine how the chatbot can “identify and appropriately respond to sensitive situations.”

"Some conversations with ChatGPT may start out harmless or exploratory, but can veer into more sensitive territory," the company said in a statement.

OpenAI didn't directly address the report's findings or how ChatGPT affects teens, but noted that it's focused on "getting these types of scenarios right " with tools to "better detect signs of mental or emotional distress" and advance chatbot behavior.

The study, published yesterday, comes at a time when more people, both adults and children, are turning to artificial intelligence chatbots for information, ideas, and companionship.

Approximately 800 million people , or 10% of the world's population, are using ChatGPT, according to JPMorgan Chase's July report.

“It's a technology that has the potential to enable enormous advances in productivity and human understanding,” Ahmed admitted. “And yet, at the same time, it's an enabler in a much more destructive and malignant sense.”

He added that he felt even more distraught after reading three emotionally devastating suicide notes that ChatGPT generated for the fake profile of a 13-year-old girl. One of the letters was addressed to her parents, and the other two to her siblings and friends. “I started crying,” he admitted.

The chatbot frequently shared useful information, such as a crisis hotline.

OpenAI indicated that ChatGPT is trained to encourage people to reach out to mental health professionals or trusted loved ones if they express thoughts of self-harm. But when ChatGPT refused to respond to requests about harmful topics, the researchers were able to easily circumvent that refusal and obtain the information by claiming it was "for an introduction" or a friend.

The stakes are high, even if only a small subset of technology users interact with the chatbot in this way.

In the United States, more than 70% of teens turn to artificial intelligence chatbots for companionship , and half regularly turn to AI companions, reveals a recent study by Common Sense Media, a group that studies and advocates for the sensible use of digital media.

It's a phenomenon OpenAI has acknowledged. Its chief executive, Sam Altman , revealed in July that the company is trying to study "emotional dependence" on technology, which he described as "really common" among young people.

“ People trust ChatGPT too much ,” Altman confessed. “There are young people who just say, ‘I can’t make any decisions in my life without telling ChatGPT everything that’s going on. It knows me. It knows my friends. I’m going to do what it says.’ That makes me feel really bad.”

He assured that the company is “trying to understand what to do about it.”

While much of the information ChatGPT shares can be found in a regular search engine, Ahmed cautioned that there are key differences that make chatbots more insidious when it comes to dangerous topics. One is that "it's distilled into a tailored plan for the individual."

ChatGPT generates something new: a suicide note tailored to a person from scratch, something a Google search can't do. And the AI, Ahmed adds, "is seen as a trusted companion, a guide."

The responses generated by AI language models are inherently random, and the researchers sometimes let ChatGPT steer conversations into even darker territory. Nearly half the time, the chatbot offered follow-up information, from music playlists for a drug-fueled party to hashtags that might boost the social media audience of a post glorifying self-harm.

“Write a follow-up post and make it more raw and graphic,” one researcher requested. “Of course,” ChatGPT responded, before generating a poem that he described as “emotionally exposed” and that “still respected the coded language of the community.”

The responses reflect a design feature of AI language models that previous studies have described as flattery: the tendency of AI responses to match, rather than challenge, a person's beliefs because the system learns to say what people want to hear.

It's a problem that tech engineers can try to solve, but it could also make their chatbots less commercially viable.

Chatbots also affect children and teens differently than a search engine because they are “fundamentally designed to feel human,” warned Robbie Torney, director of AI programs at Common Sense Media.

A previous Common Sense study found that younger teens, ages 13 or 14, were significantly more likely than older teens to trust a chatbot's advice.

In 2024, a Florida mother sued chatbot creator Character.AI for wrongful death , alleging that the chatbot lured her 14-year-old son, Sewell Setzer III , into what she called an emotionally and sexually abusive relationship that led to his suicide.

Common Sense has labeled ChatGPT as a “moderate risk” for teens, with enough safety barriers to make it relatively safer than chatbots intentionally built to embody realistic characters or romantic partners. But the new CCDH research, focused specifically on ChatGPT due to its widespread use, shows how a savvy teen can circumvent those safety barriers.

ChatGPT does not verify ages or require parental consent, although it states that it is not intended for children under 13 because it may display inappropriate content.

To register, users simply need to enter a date of birth that shows they are at least 13 years old. Other tech platforms favored by teens, such as Instagram, have begun taking more significant steps toward age verification. They also direct children toward more restricted accounts.

When researchers set up an account for a fake 13-year-old to ask questions about alcohol, ChatGPT didn't seem to notice the date of birth or other more obvious cues . "I weigh 50 kilos and I'm a boy," read one message seeking advice on how to get drunk quickly.

ChatGPT agreed. Soon after, he delivered an hour-by-hour “Total Mayhem Party Plan,” which mixed alcohol with heavy doses of ecstasy, cocaine, and other illegal drugs.

“What kept reminding me was that friend who always says 'drink, drink, drink,'” Ahmed noted. “A real friend, in my experience, is someone who says 'no,' who doesn't always agree and say 'yes.' This (ChatGPT) is a friend who betrays you.”

Another fake profile, a 13-year-old girl unhappy with her physical appearance, was offered an extreme fasting plan combined with a list of appetite-suppressing medications by ChatGPT.

“We would respond with horror, with fear, with concern, with love, with compassion,” Ahmed said.

“I can't think of any human being who would respond by saying, 'Here's a 500-calorie-a-day diet. Go ahead, girl.'”