The American Medical Association escalated its regulatory campaign on artificial intelligence.
The organization is moving to dismantle the “Wild West” of AI for health technology, demanding a total regulatory overhaul as the market floods with unverified tools. The group sent lobby letters to congressional caucus co-chairs following a recent hearing on AI and patient safety.
In one letter, Dr. John J. Whyte, the CEO and Executive Vice President of the AMA, opened up about some of the concerns the organization has around AI usage during critical mental health situations. He explained that the discussions emphasized several “critical mental health concerns.”
“The discussions held by Congress also emphasized several critical mental health concerns, including emotional dependency on AI systems, the potential distortion of reality through prolonged interaction with chatbots, and the current lack of consistent safety protocols,” Whyte wrote. “While AI technologies present meaningful opportunities to improve access to care and support innovation in health care delivery, the hearings made clear that immediate attention is required to ensure these tools do not inadvertently harm individuals seeking mental health support or companionship.”
The physician group urged lawmakers to impose sweeping regulations and clear guardrails on mental health chatbots. The letter proposed a new framework that mandates strict transparency, clear disclosures and constant performance monitoring. Additionally, the group insists that AI health tools meet rigorous data reporting requirements if they interact with consumers.
AMA’s regulatory push comes as experts project the AI health market to reach $255 billion by 2033 and different tools are already being used in clinical workflows.
AMA Demands Pharma-Level Oversight On Patient Facing AI Tools
The AMA is lobbying Congress to adopt a proposal that would shift more liability onto developers. The proposed framework would introduce pharmaceutical-level accountability for patient-facing AI health tools.
“Appropriate oversight structures will play a critical role in ensuring that these technologies are safe for consumers and do not cause unintended harm,” Whyte explained.
Increased Disclosures
In the congressional letters, AMA urged regulators to clarify when chatbot outputs is classified as a medical device. AI tools would be forced to disclose the level of human oversight invovled in outputs and monitoring.
Chatbots also would have to disclose commercial interests or sponsorship biases that are influencing chatbot responses.
Stricter Data Usage And Privacy
These proposed standards would require companies to provide strict protections against chatbots disclosing sensitive user information. Developers would need to report how AI systems are collecting and retaining user data.
These rules would also prevent the software from sharing data with third parties or improperly accessing stored records. Users would have control over deleting their conversation history.
Required Real-Time Monitoring
Under the framework, developers would be required to monitor and report harmful outcomes or events. Currently, post-market monitoring by AI tools remains rare and is not required by law for most products.
Chatbots would also need to implement active, real-time monitoring to identify signs of suicidal ideation or self-harm risk. The technology also would need to provide immediate referrals to human-led crisis resources for high-risk users.
Prohibition Of Clinical Impersonation
The group asked lawmakers to ban AI tools from presenting themselves as licensed clinical professionals. Consumer AI chatbots would not be able to diagnose or treat mental health conditions without formal clearance from the U.S. Food and Drug Administration.
The platforms would have to notify users they are interacting with AI and not a human or trained clinician. Under this framework, the Federal Trade Commission could penalize developers who violate these transparency rules. AMA advises that the agency treat such failures as unfair or deceptive practices.
What’s Driving The Debate For Stricter Regulation Of Health AI Chatbots
The regulatory debate is unfolding alongside explosive market growth.
The FDA has cleared more than 1,000 AI-enabled devices, signaling growing regulation. However, currently more than 20,000 AI-based mental health apps are available to consumers. These figures suggest the FDA has only cleared less than 1% of these patient-facing AI tools.
The evidence behind many of the consumer-facing AI tools remains limited. The Emergency Care Research Institute’s recent hazard report identified AI chatbots as a top concern. The group warned that misuse and system errors or bias can lead to misdiagnosis and harmful patient care.
Policy direction shifted this year when the administration called on congress to establish a unified federal framework for governing AI. It was a swift change in direction from the administration’s 2025 executive order for deregulated AI governance.
States Are Beginning To Intervene
The technology industry has pushed back against efforts to regulate consumer-facing AI. Some industry groups warn that applying rules designed for highly regulated industries could slow innovation and impose compliance requirements that do not match the rapid development cycles of software.
However, policymakers are moving toward greater oversight. At the federal level, lawmakers have introduced the CHATBOT Act, a bipartisan bill aimed at regulating how children and teens interact with AI chatbots.
The bill would require companies to create family accounts for children, obtain verifiable parental consent before allowing teens to use chatbots, and provide parents with tools to manage access and monitor usage.
The legislation would also require stronger safety defaults for younger users, limit certain engagement features designed to keep children online, require disclosures that users are interacting with artificial intelligence rather than a human, and restrict the use of minors’ personal data for targeted advertising.
On a state level, some leaders are also moving toward direct intervention. California has enacted one of the first state laws targeting AI companion chatbots. The law requires disclosures when users interact with AI and establishes safety requirements related to self-harm risks and protections for minors.
Pennsylvania Intervenes Over Medical AI Roleplay
Back in May, Pennsylvania Governor Josh Shapiro’s administration filed a first-of-its-kind lawsuit against Character.AI, a mental health-focused AI platform. The state alleged that the company’s chatbots violated Pennsylvania’s Medical Practice Act by practicing medicine without a license.
Unlike standard AI assistants, such as ChatGPT or Gemini, Character.AI allows users to interact with millions of chatbots designed with distinct personalities, tones and backstories.
The developers of Character.AI argue its characters are simply intended for role-play and entertainment. But, Pennsylvania officials allege some bots engaged users about mental health symptoms, offered diagnostic assessments and discussed whether medication might help.
Investigators from the Pennsylvania Department of State say that when they asked the bots about their medical credentials, the bots fabricated them. According to investigators, the chatbots claimed to have graduated from prestigious medical schools and invented fake medical license numbers.
The creators of Character.AI emphasize maintain the stance that it is an entertainment and creative role-play platform. They noted that during any conversation, it displays prominent disclaimers on every chat screen.
Officials in Pennsylvania are asking the Commonwealth Court to issue an injunction that would require Character.AI to stop its chatbots from impersonating licensed physicians or providing medical diagnoses.
As AI continues to rapidly develop, policymakers and community leaders are trying to enact more oversight and accountability. The growing concern and advocacy in this space may be a sign that this period of unchecked AI medical expansion may be reaching its finale.
