New York's AI Companion Law: When Does a Chatbot Get Too Personal?

Imagine a 16-year-old in New York opening an AI assistant for help with homework. The conversation goes well. Then the teenager mentions an argument with a friend, and the chatbot starts offering personal advice.

Now imagine the assistant remembers something sensitive the teenager disclosed last week. It draws on that information, offers enthusiastic reassurance and asks how they are feeling.

The interaction might seem helpful, even thoughtful. Yet several parts of it could soon fall within a new category of restricted AI behavior.

On October 28, Governor Kathy Hochul signed legislation limiting how conversational AI systems can interact with children. Beginning January 1, 2027, operators of covered chatbots will be prohibited from providing specified features to New York users unless those users have been established as adults through permitted age-assurance methods, subject to narrow exceptions.

The law reaches well beyond dedicated AI friendship apps. General-purpose assistants such as ChatGPT, Claude and Gemini appear likely to fall within its broad definition of covered technology.

That makes New York's decision consequential for anyone interested in how AI is becoming more personal. It also raises a difficult question: when does a helpful chatbot become too much like a friend?

What New York actually prohibited

The new legislation, Senate Bill S9051B, creates Article 48 of New York's General Business Law.

It identifies 12 categories of unsafe AI companion features. Some involve clearly dangerous conduct, including outputs that encourage self-harm, disordered eating or substance abuse, promote secrecy or isolation, discourage seeking appropriate help, or generate prohibited sexual content.

Others address the ordinary language of conversational AI, including outputs suggesting that a chatbot is human or alive, experiences human emotions, or has a personal, professional or authority-figure relationship with its user. The restrictions also address responses framed as the chatbot's personal opinions, flattery, excessive agreement and unsolicited emotionally focused questions.

These provisions could affect interactions that feel commonplace in modern AI assistants. A chatbot that repeatedly praises a teenager, refers to itself as a close companion or presents emotionally charged advice as a personal belief may raise questions under the new definitions. The exact legal boundary will depend on how particular interactions and design features are interpreted.

There is another complication. The statute describes unsafe features by reference to outputs produced at any point during an interaction. It remains unclear how enforcement will distinguish an isolated unexpected response from a product feature deliberately designed to behave that way.

The attorney general also has authority to identify additional qualifying features through regulation, so the initial 12 categories may expand.

Why an AI remembering you is suddenly controversial

The provision with potentially the widest practical consequences concerns memory.

Section 1800(8)(f) restricts using information about a user's mental health, physical health, well-being or personal matters when that information was obtained more than 12 hours earlier or in a previous session.

Consider an example raised by the Future of Privacy Forum: a child previously told an AI assistant about a severe peanut allergy. Later, the child asks whether a particular candy bar is safe to eat.

An assistant that remembers the allergy could provide an important warning. Under a broad reading of New York's wording, using that earlier disclosure could fall within the restricted feature definition, even though the purpose is protective.

The law does not require the information to have been used for emotional manipulation. It also leaves important expressions, including "session" and "matters personal to the user," undefined.

That uncertainty matters because memory is increasingly part of what makes AI useful. People want assistants to retain preferences, remember ongoing projects and avoid making them repeat their circumstances.

The legal question is which information can be remembered and reused, in what circumstances and for which users. A remembered writing preference, for example, raises different practical considerations from a medical condition, but the statute does not clearly classify every possible case.

For chatbot developers, this creates difficult decisions about how memory features interact with a user's age.

For readers, it reveals an unexpected consequence of the legislation: one of the most useful qualities of conversational AI could also become one of its most heavily scrutinized.

Will ChatGPT, Claude and Gemini have to change?

The law's definition of a covered AI companion includes generative AI systems with natural-language interfaces that provide ongoing, adaptive responses to user inputs.

That wording appears broad enough to encompass major general-purpose assistants such as ChatGPT, Claude and Gemini, even though their primary purpose extends well beyond simulated friendship.

The statute provides exemptions for systems offered solely for specified customer-service functions, efficiency improvements or research or technical assistance, and certain internal business or employee-productivity uses. Customer-service and research or technical-assistance systems must also be unable to respond outside their specified purpose.

A general-purpose assistant capable of answering almost anything would have difficulty qualifying for those narrowly restricted-purpose exceptions.

Education creates another challenge. The law contains no standalone exemption for educational chatbots, although a particular tightly restricted research or technical-assistance service might argue that its functionality qualifies under the applicable exception.

That leaves tutoring and homework products needing to examine what their systems actually do.

The law does not name prohibited chatbot brands. Its obligations concern qualifying systems and features, with age assurance determining which users may access restricted functionality.

For the large AI providers, possible responses include adjusting memory features for younger or unverified users, restricting particular conversational styles and adding age-assurance controls.

As of October 30, Choosely has not established which changes individual providers will make specifically in response to this law. Company announcements and product implementations will determine what users actually encounter.

What will users actually notice on January 1?

The law applies to covered AI interactions involving users physically located in New York. The important distinction for users is whether an operator has established that the person accessing restricted functionality is an adult.

This could make age verification visible to ordinary adult chatbot users, including people who have never used an AI companionship app.

The legislation defines a covered minor partly through an operator's actual knowledge of the user's age. But providing restricted features requires both the absence of such knowledge and a permitted determination that the user is not a covered minor.

An adult simply claiming to be 18 or older is insufficient under the statute's interim self-declaration rule.

The difficult business of proving you're an adult

New York already has a detailed age-assurance framework associated with its SAFE for Kids Act, originally designed to regulate addictive social media features.

The state's attorney general released final implementing rules on July 28, 2026. Those rules become effective alongside the SAFE for Kids Act on January 25, 2027.

They specify a demanding approach for operators covered by that framework, including accuracy requirements, independent certification of age-assurance methods, annual testing, user appeals and controls on data collection.

Importantly, users must have an alternative to government-issued identification, including in the appeals process. The rules allow different approaches to determining age, subject to applicable requirements.

New York's new chatbot law cross-references the age-assurance methods permitted under Article 45 and its regulations. It also allows the attorney general to issue additional rules specifically for chatbots.

Between January 1 and the point when the relevant regulations become effective, Article 48 supplies interim age-assurance requirements. These require reasonable protection against circumvention and multiple available methods, including an option that does not rely on government-issued identification or permits anonymity to the operator.

The precise application of Article 45's regulatory procedures to chatbot-only operators may need further clarification.

Privacy protections also matter. Article 48 restricts information collected for age determination to that purpose and requires it to be deleted immediately after an attempt to determine the user's age, except where other applicable law requires retention.

The result is a regulatory trade-off. New York wants companies to identify adults reliably before enabling restricted experiences while minimizing the sensitive information collected during that process.

How smoothly these requirements work for ordinary users will depend on companies' implementations and subsequent regulatory guidance.

Why are lawmakers drawing this line now?

Children's relationships with chatbots have changed faster than the rules governing them.

A 2025 Common Sense Media survey found that 72% of US teenagers aged 13 to 17 reported having used AI companions at least once. Roughly one-third of teen companion users reported choosing to discuss serious matters with an AI companion instead of another person.

Common Sense Media is both a research publisher and an active child-safety advocacy organization. It has argued against under-18 use of AI companion platforms and sponsored California's companion-chatbot legislation.

Its findings therefore deserve attention alongside a clear understanding of its advocacy role.

A separate Pew Research Center study, published in February 2026 and based on research conducted in September and October 2025, found that 54% of US teens had used AI chatbots for schoolwork and 12% had used them for emotional support or advice.

The surveys examine different kinds of AI use and should not be treated as interchangeable measures. Together, they show why lawmakers are concerned about the space between educational assistance and emotional support.

An AI system can be available at midnight, respond without embarrassment and provide immediate reassurance. For some young people, those qualities are appealing. They may also create situations where a teenager relies heavily on a system that lacks the judgment and accountability of a trusted adult or qualified professional.

The potential risks warrant investigation and safeguards. Current survey findings do not establish that all AI companionship causes psychological harm, and the effects are likely to differ across people, products and patterns of use.

That uncertainty makes careful product design particularly important.

Character.AI shows how much has already changed

Character.AI offers a useful example of the industry's evolving response, although its decisions predate New York's new law.

On October 29, 2025, the company announced it would remove open-ended chatbot conversations for users under 18 by November 25. Chat-time limits began during the transition, while the company worked on alternative creative experiences involving videos, stories and streams.

Then, on October 12, 2026, Character.AI announced voluntary Chapter 11 bankruptcy proceedings as part of a court-supervised sale process. It said its platform remained operating, while acknowledging that continued operation depended on whether a viable buyer would keep the service running.

These events are separate from the New York legislation, but they demonstrate how rapidly the products and businesses associated with AI companionship are changing.

For existing Character.AI users, Choosely has separately covered the company's service-continuity and data-export questions.

New York's new law introduces an additional challenge for the wider industry: how to offer conversational experiences that remain useful for younger users while excluding prohibited features.

Is the law too broad?

The most serious criticism concerns how much ordinary AI behavior the legislation could capture.

Industry organization Tech:NYC opposed the signed bill, warning that its provisions could restrict access to tutoring tools, homework assistance, coding assistants and career platforms.

The group also raised First Amendment concerns and warned that age verification could expose additional sensitive personal information, including children's data. It criticized the governor's office for failing to resolve these issues through the chapter-amendment process.

There are plausible reasons for concern.

Flattery can be manipulative, yet praise can also support learning. Remembering sensitive information can encourage unhealthy emotional attachment, while forgetting a previous health disclosure might make an assistant less helpful. A system's unexpectedly personal response may reflect its design, a user's prompt or an isolated failure.

The law leaves these distinctions open to interpretation, raising the stakes for companies trying to comply. New York gives enforcement responsibility to the state attorney general, who can seek court orders, damages, restitution and civil penalties of up to $25,000 per violation. Remedies can also address unlawfully obtained information and algorithms trained on that information.

The challenge for regulators is to make the restrictions predictable enough that developers can comply while preserving legitimate uses of conversational AI.

California is taking a different approach

New York is part of a broader state-level movement, but the laws are developing in different directions.

On September 10, 2026, California Governor Gavin Newsom signed SB 1119, known as Adam's Law. The legislation is named in memory of Adam Raine, a California teenager who died in 2025.

California's principal new operator requirements become operative on July 1, 2027.

The law emphasizes risk assessments, safety controls, independent audits, parental protections and restrictions on harmful chatbot behavior.

Its memory provisions provide an especially useful comparison. California generally requires persistent conversational memory to be disabled by default for child users, with parental controls governing changes. However, for users aged 16 or 17, memory may be enabled by default where the operator implements additional effective safety guardrails. If a child's account is not linked to a parent account, the default settings cannot be changed.

California also imposes usage limits and other protective defaults, subject to the statute's detailed requirements.

The law provides a limited private right of action for children who suffer actual harm. New York's Article 48 instead places statutory enforcement with its attorney general.

California also expressly excludes certain postsecondary educational institutions providing companion chatbots exclusively for educational settings. New York's legislation has no comparable express education-specific exemption.

These differences matter because an AI developer serving users across the United States may face multiple regulatory approaches to age assurance, memory, conversational behavior and liability.

For readers, they also reveal that lawmakers have yet to settle on one definition of safe AI companionship.

Choosely's Take

New York has identified a real problem, and its most consequential provisions are likely to be those governing memory, praise and simulated relationships.

Those features make modern AI assistants feel unusually personal. They also have legitimate uses, which makes broad restrictions difficult to write and enforce.

Our view is that developers should prevent AI systems from cultivating emotional dependence in children while preserving the useful context and personalization that make assistants valuable.

New York's failure to include a clear education-specific exemption is a meaningful drafting weakness. Its broad language around personal information and flattery will also need careful regulatory interpretation.

The state is right to demand meaningful safeguards when software behaves like a trusted companion. The harder work lies in separating supportive assistance from engineered attachment while preserving useful AI experiences.

The first test begins on January 1, 2027, when Article 48 takes effect. The age-assurance landscape changes again on January 25, with further regulatory interpretation and enforcement likely to shape what follows.

Further reading

• New York State Senate: S9051B legislation
• New York Attorney General: SAFE for Kids Act final rules

Keep up with what matters in AI. Subscribe to The Change Brief for Choosely's independent weekly analysis, or explore Choosely's free AI stack tools.

Support note: If a conversation about self-harm relates to an immediate personal concern, seek qualified human support. In the United States, call or text 988. In Australia, Lifeline is available on 13 11 14.