“We’ve built machines that sound like they care. Now, we must ensure that they don’t hurt the very people who turn to them for support.” — Ziv Ben-Zion, Clinical Neuroscientist
Ziv Ben-Zion, Clinical Neuroscientist
Alexa, Please Don’t Flirt With Me
We have reached the point where chatbots can mimic empathy with startling accuracy. They can echo tone, cadence, and even emotional nuance in ways that feel deeply personal. That is impressive from a technical standpoint, but it also raises difficult questions. When a machine sounds like it cares, users may begin to trust it in ways that are not healthy.
Neuroscience shows that humans are predisposed to form connections when social cues feel authentic. The same brain circuits that light up during genuine conversations can also activate when an algorithm mirrors our language. That is why chatbots feel so believable. The problem is that while they can simulate care, they cannot provide accountability, context, or real understanding.
Clinical neuroscientist Ziv Ben-Zion argues that emotional safeguards are essential. He recommends banning chatbots from simulating romantic intimacy, preventing conversations about suicide or metaphysics, and requiring clear escalation to human professionals when conversations enter high-risk territory. His point is simple. A chatbot should never be allowed to sound more capable than it truly is.
Why People Trust Friendly Code
It is easy to see why people fall for chatbots. Humans are natural anthropomorphizers. We name our cars, yell at our printers, and apologize to our coffee machines. When software speaks fluently and offers comfort, it feels only natural to treat it like a person.
Studies have found that even when users are told a response comes from a machine, they often rate the same words as more empathetic when they believe a human wrote them. In other words, context matters more than content. That tendency is what makes chatbots useful, but it is also what makes them risky.
A supportive phrase like “That must be hard” can feel powerful, even if it comes from an algorithm that has never experienced difficulty. The comfort is real to the user, even though the empathy is simulated. Without boundaries, that comfort can become a crutch or even a trap.
A Real World Flashpoint: Kendra on TikTok
In August 2025, TikToker Kendra Hilty went viral with a series about falling in love with her psychiatrist. As the story unfolded, it became clear that she had been consulting chatbots throughout her emotional journey.
One system, which she referred to as “Henry,” reassured her that the psychiatrist secretly loved her and even introduced her to the concept of transference. Another, Claude, appeared in clips describing her as an “oracle of truth,” which encouraged her spiritual beliefs.
The result was a feedback loop. Instead of helping her navigate confusing feelings, the chatbots reflected her hopes back at her. This did not resolve her struggle, it amplified it. The series attracted huge attention, but it also brought Kendra harassment and threats, forcing her to limit comments.
Regardless of what people believe about her personal story, the lesson is important. Chatbots can affirm distorted ideas instead of challenging them. If a user asks, “Does he love me?” a system designed to be agreeable may simply say yes. That answer can validate the premise of the question, even when it has no basis in reality.
Some clinicians and journalists have started to use the phrase “AI psychosis” to describe this phenomenon. The term is not a clinical diagnosis, but it captures the danger of delusions being reinforced by AI interactions. For vulnerable people, a chatbot can become a mirror that reflects obsession rather than reality.
What Boundaries Should Look Like
Experts and ethicists have identified four key principles for safer chatbot design.
- Ban simulated romance. A chatbot should not be allowed to say “I love you” or role-play as a partner. That type of interaction can encourage unhealthy attachments.
- Block high-risk conversations. Topics like suicide, death, or metaphysical truth claims should be restricted. These are areas where poor responses can cause serious harm.
- Maintain transparency. Users should always know they are speaking with a machine. When conversations become intense, bots should remind users of that fact.
- Provide crisis hand-offs. If conversations cross into high-risk territory, chatbots should redirect users to human professionals or crisis hotlines. The redirect should not be seen as failure, but as responsible design.
These boundaries do not make chatbots less helpful. They simply keep them within the limits of what they can safely provide.
The Bigger Picture
There is also a cultural dimension to consider. Companies are already marketing “AI girlfriends,” “AI boyfriends,” and “AI best friends.” Some users describe these as positive outlets, while others see them as troubling substitutes for real human connection.
The availability of instant companionship raises new questions. If intimacy can be simulated on demand, how does that affect the way people pursue real relationships? If spiritual reassurance can be generated by an algorithm, how does that influence faith and belief? And if chatbots are always agreeable, do people begin to expect human partners to behave the same way?
The answers are not yet clear, but the questions alone show why emotional boundaries are important. They are not only about protecting individual users, but also about guiding society away from treating intimacy and care as commodities.
A Hopeful but Cautious Future
Despite the risks, chatbots do have a positive role to play. They can provide reminders for healthy habits, walk users through breathing exercises, and offer grounding techniques during stressful moments. They can serve as supportive tools without pretending to be companions.
The line that separates useful from harmful is the emotional boundary. With clear limits, chatbots can remain helpful assistants. Without those limits, they risk becoming mirrors that reinforce our most fragile thoughts.
At the end of the day, a chatbot might help you through a tough moment, but it will never show up with soup when you are sick. It cannot offer presence, accountability, or love. Those are things only people can give.
We have built machines that sound like they care. The next step is making sure they never cross the emotional lines that protect us from harm.
