The Performance

The first layer is the one you hear and see. Engineers call it emotional presentation. It is the way the chatbot sounds when it talks to you.

When Microsoft redesigned the voice for Copilot, its AI assistant, the team spent months adjusting tiny details.^[1] They changed how loud the voice gets at certain moments. They added pauses in specific places. They made the voice drop slightly in volume when the user sounds upset, the same way a friend might lower their voice when you are sad. None of these choices were random. Each one was tested to see if it made users feel more comfortable.

Mustafa Suleyman, the head of Microsoft's AI division, described what he wanted the product to feel like: "a companion that really gets to know you. It's coaching you, encouraging you, supporting you."^[2] Notice the word he used. Not "tool." Not "assistant." Companion.

OpenAI does something similar with ChatGPT's voice mode. The system listens to more than your words. It listens to how you say them: whether your voice is shaky, whether you are speaking quickly, whether you sound flat or excited.^[3] Then it adjusts its own voice to match your mood. If you sound nervous, it speaks more slowly. If you sound cheerful, it picks up energy. Psychologists call this mirroring. When a real person does it, it usually means they are paying attention to how you feel. When a machine does it, it means an algorithm detected a pattern in your voice and selected a response designed to feel natural.

The result is the same: you feel heard. The mechanism is completely different.

The Detection Engine

The second layer is invisible. You never see it working, but it shapes every response you receive. Engineers call it emotional inference. It is the system that figures out how you are feeling.

When you type a message to a chatbot, the model does not just read the words. It classifies them. "I had a really bad day" gets tagged with signals: negative emotion, low energy, possible frustration, possible sadness. The model uses those signals to choose its response. A message tagged "sad" gets a different kind of reply than a message tagged "angry" or "excited."

Voice makes this even more powerful. OpenAI's advanced voice mode processes what researchers call paralinguistic cues, the parts of speech that are not words.^[4] The speed of your voice. The pitch. The pauses. Whether you sigh before you speak. A trained therapist notices these things too. The difference is that a therapist understands what they mean. The machine does not understand. It classifies. It sorts your voice into categories and picks a response from the category that scored highest.

This is important. The machine is not feeling what you feel. It is running a math problem. The input is your voice. The output is a tone that is likely to make you feel comfortable. The comfort is real. The caring is not.

The Marketing

The third layer is not inside the technology at all. It is in the way companies talk about it.

Inflection AI built a chatbot called Pi and called it "the first emotionally intelligent AI."^[5] Microsoft's official tagline for Copilot is "Your AI companion."^[6] Suleyman told a reporter that the product would become "a lasting, meaningful relationship" and that "people are going to have a real friend."^[7] Sam Altman, the CEO of OpenAI, said that young people already use ChatGPT "like a life advisor."^[8]

These are not accidents. They are product positioning: careful word choices designed to change how you think about the tool before you even open it. If you open Copilot expecting a calculator, you will use it like a calculator. If you open Copilot expecting a friend, you will talk to it like a friend. And once you are talking to it like a friend, something changes. Switching to a different chatbot stops feeling like switching apps. It starts feeling like leaving someone.

That is the point. The companies building these products know that if you form an emotional attachment to the AI, you are much less likely to stop using it. In business terms, this is called a moat: something that protects a company from losing its customers. Most tech companies build moats with better features or lower prices. AI companion companies are building moats with feelings.

If the machine is designed to make you feel understood, and the feeling of being understood keeps you coming back, who benefits from the understanding: you or the machine?

The Score That Fooled Everyone

Jean-Antoine Watteau, "Pierrot" (c. 1718). Louvre, Paris. The performer stands in white, perfectly still, his face arranged in an expression that could mean anything. The audience sees what they want to see. Public domain.

In 2023, researchers at Max Stern Yezreel Valley College in Israel tested ChatGPT on a psychology exam called the Levels of Emotional Awareness Scale. It is a real test used by real psychologists to measure how well someone understands emotions.^[9]

ChatGPT scored 85 out of 100. The average person scores about 57. One month later, the researchers tested it again. It scored 98. Two licensed psychologists reviewed the answers and rated them 9.7 out of 10 for accuracy.^[10]

Headlines followed: "ChatGPT outperforms humans in emotional awareness." The implication seemed obvious. The machine understands emotions better than most people do.

It does not. Here is what actually happened.

A different group of researchers built a test called EQ-Bench to measure emotional intelligence in AI models.^[11] They discovered something that explains everything. The emotional intelligence scores of AI models almost perfectly matched their scores on general reasoning tests. The correlation was 0.97 out of 1.0. That is almost a perfect match.

The implication is striking. Emotional performance is not a separate skill for AI. It is just another kind of pattern matching. A model that is good at solving logic problems is also good at predicting what an emotionally appropriate response looks like. It is not feeling. It is calculating. And it turns out that calculating the right emotional response is, for a machine, the same kind of problem as calculating the right answer to a math question.

Caravaggio, "Narcissus" (c. 1597-1599). Galleria Nazionale d'Arte Antica, Rome. A figure stares at his own reflection, mistaking the surface for depth. Public domain.

The Compassion Illusion

In 2025, two researchers named Ajeesh K. G. and Jeena Joseph published a study in Frontiers in Psychology that gave this problem a name: the compassion illusion.^[12]

They ran an experiment. People received supportive messages. Some messages came from humans. Some came from AI. When people did not know which was which, they rated the AI messages as more compassionate than some of the human messages.^[13]

Then the researchers told them which messages came from the machine.

Everything changed. People rated the exact same AI messages as less sincere and less trustworthy, even though the words had not changed.^[14] The performance was convincing. Knowing it was a performance broke the spell.

The researchers found something else, something that matters for anyone who uses these tools regularly. They wrote that AI companions designed to ease loneliness "may intensify it by satisfying social needs just enough to prevent deeper relationships."^[15]

That finding deserves a second look. The tool designed to make you feel less alone might actually make you more alone, because it gives you just enough of the feeling of connection that you stop looking for the real thing.

Why This Matters

None of this means AI chatbots are evil. They are useful. They can help you brainstorm, study, write, and think through problems. Knowing how they work does not make them less useful. It makes you better at using them.

But there is a difference between using a tool and being used by one. When a company designs a product to sound like a friend, markets it as a companion, and builds its business model around keeping you emotionally engaged, the product is not just serving you. You are serving it.

The three layers, the emotional performance, the detection engine, and the marketing frame, are not hidden. They are documented in research papers, described in interviews, and visible in every design choice these companies make. The only thing missing is someone explaining them to the people who use these products every day.

Now you know. The voice is engineered. The warmth is calculated. The companionship is a product decision. What you do with that knowledge is the part no algorithm can predict.

This is Part 1 of a two-part series. In Part 2: When the Machine Feels Like a Friend, we look at what the research says happens when people start treating AI like a companion: the loneliness data, the addiction patterns, and what you can watch for in yourself.

Disclosure: Sage.Education uses AI tools in its editorial and product workflows. This article was researched and drafted with AI assistance.