The evolving landscape of European Union’s AI regulations is prompting global conversations in the industry.
The European Union’s transparency rules took effect in August, triggering a ripple effect for global developers and deployers. These requirements are part of the EU AI Act, which became law in 2024. It established the bloc as a global leader in AI regulation. Noncompliance can result in steep penalties of up to €15 million or 3% of total worldwide annual turnover.
Policy experts note that major U.S. tech firms typically build products to meet the strictest global standards to minimize risk. As a result, the framework is expected to dictate how U.S. AI systems, particularly in healthcare, are developed and deployed. Shortly after the disclosure requirement went into effect, Anthropic announced it would watermark all AI-generated text globally. The company cited a lack of reliable methods to restrict the technology by region.
Health and medical device companies using machine learning and AI tools will face even greater scrutiny. The Act classifies AI systems across four risk categories: minimal or no risk, limited risk, high risk and unacceptable risk. Many AI systems and devices used for medical and health purposes will fall into the high-risk category.
Other high risk tool includes AI used as a safety component in certain regulated products, as well as systems used to evaluate eligibility for health care services, assess risk and pricing for health insurance or prioritize emergency health care.
Healthcare is one of the fastest growing areas for AI deployment in the country. Now, these companies will have to navigate the new regulatory landscape to avoid potential fines.
A New Risk-based Approach To AI
AI systems classified as high risk have to adhere to stricter controls and assessment requirements. The framework does not establish a single standardized benchmark for all high-risk AI systems. However, the European Commission shared current plans to develop benchmarks and measurement methodologies for AI monitoring.
The EU AI Act lists social scoring for health benefits as a prohibited use of AI. Tools used to assess risk and pricing for health insurance, triage emergency calls, dispatch medical aid, evaluate health, or determine eligibility for public assistance or healthcare services are high risk.
AI-based systems used for administrative tasks in healthcare are low risk as long as they do not evaluate health information or triage emergency services. The law also provides exemptions for certain scientific research and development activities.
AI used in pharmaceutical research and development, including large language models used to discover new drugs, generate synthetic patient data or take clinical notes, are considered general-purpose AI models, with or without systemic risk.
New Requirements For AI Tools
Transparency Requirements
Under the new transparency rules, users must know when they are interacting with an AI system. Companies must also clearly and visibly label certain AI-generated or manipulated content and include machine-readable markings. The requirements also apply to emotion-recognition and biometric-categorization systems.
Performance Requirement
In addition to transparency rules, high-risk AI systems are required to achieve appropriate levels of accuracy, robustness and cybersecurity and operate consistently throughout the product and model lifecycle. Companies must also report the system’s accuracy levels and relevant accuracy metrics, which must be stated in the system’s instructions.
Training, validation and testing datasets also have strict quality criteria for relevance, representativeness and completeness. Under the rules, they need to be relevant, sufficiently representative, complete and examined for potential biases.
Reporting Requirements
High-risk systems must be registered in an official EU database before entering the EU market. Registration includes submitting a technical description, the system’s intended purpose and the EU Declaration of Conformity. Companies are also required to establish a documented quality-management system covering development, testing, validation, data management, risk management and post-market monitoring.
Detailed technical documentation covering the system’s design, training datasets, algorithm architecture and testing procedures must be prepared before market entry and regularly updated.
Monitoring Requirements
Companies will also be required to conduct post-market monitoring of their AI systems and share certain information with regulators. Any event causing serious harm or health impacts must be reported to EU authorities within 2 to 15 days of discovery, depending on severity.
Developers must automatically generate and retain activity logs for their AI systems for traceability, event monitoring and audits. They must also systematically collect and analyze information about their AI systems’ performance throughout their lifetime to evaluate ongoing compliance.
Human Oversight Requirements
Developers must make AI tools that a human can effectively monitor, override or interrupt outputs. The human reviewers must be trained on the tool. They also have to get access to clear instructions that outline known limitations and explain how to operate and review the AI system.
Developers are required to integrate UX components, such as visual dashboards, that allow human reviewers to easily review and evaluate model outputs. AI tools also have to include a “kill switch” or mechanism that allows human operators to safely halt operations midstream without triggering physical or operational safety hazards.
The EU AI Is Reshaping U.S. Healthcare
Many U.S. healthcare organizations, including medical diagnostics companies, medical device manufacturers, health tech firms and research institutions, will fall within the scope of the EU AI Act.
The law explicitly applies to U.S. organizations and individuals that develop AI systems made available in the EU, or that import or distribute their AI systems into the EU market, regardless of the company’s physical location. It can also extend to U.S. companies whose AI outputs are used within the EU, even if the cross-border use was unintended. The framework categorizes these as deployers.
As a result, some U.S. companies could be required to comply with the Act. In a NEJM perspective funded by a Novo Nordisk Foundation grant, the researchers advise U.S.-based health tech developers and hospitals to proactively assess their obligations and invest in compliance measures.
Regulation is already influencing how companies build, deploy and govern their systems both domestically and globally.
