Apply new AI governance regimes to health first to test their benefits

China is the most advanced in rolling out a set of laws and regulations governing AI. Starting  with its regulation on recommendation  algorithms which came into force in March  2022, the country has since unveiled rules for  the management of deep synthesis and  generative AI. The public consultation phase  for trial measures on the ethical review of  advances in science and technology ended in  May 2023. 

Unlike legislation in Western countries, China’s  legislative framework is built around type of AI technology rather than on the risk that  crosscuts all AI no matter the specific  technology stream that is utilized. Public  security concerns and the role of the state are  fundamental and underlie all the regulations,  and each has specific language to safeguard against discrimination and the spread of  negative information to “respect social morality and ethics, abide by business and professional  ethics, and follow the principles of impartiality,  fairness, openness and transparency, scientific rationality, and honesty”, in the words of Article  4 of the algorithm recommendation regulation. 

The European Union’s AI Act finalized in  December 2023 has been termed “visionary”  and a “world’s first” not because it is literally  either, but because it aims to cover all aspects  of AI in a single piece of legislation and also  serve as a template for a global model of  governance for the technology. Rather than be  organized along the specific technologies that  comprise AI, the legislation classifies risks  (always used in the plural) into four categories:  minimal or no risks; limited risks; high risks; and  unacceptable risks. The first two categories of  risks require either no or light regulation with an  emphasis on user education and empowerment.  Activities deemed high risk carry more  obligations and stringent regulations to operate  in the EU. The final category, unacceptable risks,  contains activities that are banned outright  sometimes with limited exceptions. It includes  cognitive manipulation, predictive policing,  emotion recognition in workplaces and schools,  social scoring, and some remote biometric  identification systems. 

The system is overseen by a European Artificial  Intelligence Board, a scientific panel and an  advisory forum which together will bring a  measure of independence embedded in an  official legislative process with appropriate  representation of key stakeholders from within  the industry and outside it. While much will only  be known when the Act is interpreted and  applied in individual member countries—and there are already critiques that the Act is  outdated, or at least not resilient enough to  keep up with the pace of advance in the AI  field—it, along with China’s suite of regulations,  is the state-of-the-art in public policy responses  to a new and rapidly evolving technological  frontier. Other countries are either playing  catch up or at this point choosing to not  explicitly regulate AI. 

Like the EU’s approach, the United States’ approach is also risk-based but varies by sector  and is distributed across several federal  agencies, with no overarching ethos around  managing risks or the industry. Documents like Executive Order 13859 and the AI Bill of Rights issued by the White House Office of Science  and Technology Policy provide a framework,  reiterate risk-based and sector-based  governance, but implementation is fragmented  across the federal government and is  sometimes no more than aspirational. 

Canada’s proposed Artificial Intelligence and  Data Act (AIDA) is also patterned on the EU’s  risk-based approach with escalating penalties  for non-compliance but has run into a flurry of  criticism centred on three issues. First, most  major concepts (such as “high impact systems”  and “material harm”) are undefined and will be  defined later via administrative decree thus  circumventing Parliamentary scrutiny during  debate of the proposed legislation. Second,  government departments and entities are  exempt from its provisions. Third, all power is  vested in the Minister of Industry and  departmental officials from a Ministry  historically concerned with overseeing  industrial policy rather than broader questions  of the public good, privacy and human rights. A  series of amendments to the original proposal  has just been tabled but the core issues in play illustrate how complicated the matter is, and  what dilemmas and trade-offs all countries face  when it comes to managing AI. 

The US approach provides for more flexibility  (across sectors). A presidential executive order signed in October 2023 requires that safety  test results for AI-based applications in  healthcare be shared with the government, and that methods be developed to protect against  the risks of using AI to engineer dangerous  biological materials. The new directive also calls  for voluntary commitments from companies  working in the field of AI to develop methods  that ensure the safety and personal security of  US citizens. 

The EU’s and China’s comprehensive approach  are likely to encourage certainty, consistency  and stability in governance, thus encouraging  trust and investment in the AI field. On balance  the EU puts a premium on precaution, favouring  regulation over promoting innovation while the  Chinese and US approaches tilt towards the other side. 

In most other parts of the world, AI governance  tends to be guidance and aspirational and is at  early stages of anything beyond this. In August  2023, the UK government presented to  Parliament a Policy Paper that proposes a risk based, “pro-innovation approach to AI  regulation” that is likely to be a light touch on  most of the spectrum of AI use, and is months  away from implementation. In April 2023, India’s Ministry of Electronics and IT said that it does not intend to introduce legislation to regulate  the growth of AI but will implement necessary  policies and infrastructure to cultivate a robust  AI sector in the country. In addition to a set of Principles for Responsible AI enunciated in 2021,  a Digital India Act (forthcoming) will replace the  Information Technology Act of 2000. Its core constituents will be online safety, trust and  accountability, open internet, and regulations of new age technologies like AI and blockchain  technologies. In relation to AI, the legislation may delineate specific “no-go areas” for  companies and internet intermediaries  employing AI and machine learning in  consumer-facing applications with penalties for  non-compliance. 

Such approaches may be seen as a plus,  particularly for countries intending to nurture a  high-tech innovation sector or those with  limited capacity to develop policies and enforce  them—if regulations are not baked in, there is  room to leave spaces for innovation and  flexibility.

The overall picture of AI governance is one of  experimentation, significant gaps in  coverage, and with it a risk of either over- or  under-regulation of a fast-moving game changing general purpose technology. But  AI’s impacts, good and bad, will vary across  sectors, geographies and time periods. Existing attempts to regulate AI are  thus a platform on which to build out, dealing  in specific cases and situations where the  technology is applied. 

The health sector is a good example of one  where the potential gains from the proper  use of Big Data and AI are high, as are the  potential risks to personal and public safety,  privacy and human rights. The demand for  health services outstrips available supply by  far, leading to great expectations from AI by  the public, providers and policymakers.  However, the complexity and contextuality of  patient needs makes current AI systems only  suitable for use under human supervision,  despite large recent strides in generative and  foundational AI. The few examples where  regulators have permitted autonomy to AI  systems have been in very specific contexts  such as normal chest X-rays where it is clear  that AI algorithms are better than humans at  picking abnormalities. Abnormalities are still  referred to human overseers for final  determination. Despite hype about self learning AI, which would improve with use,  current regulations explicitly prohibit any  change to AI algorithms after approval, which  is typically bounded to specific use cases.  Portability of health-AI solutions to  populations other than those on whom they  were trained has also been suboptimal and  thus the onus of their use typically vests with  the physician, which is an unsatisfactory  governance system equivalent of passing the  buck. Regulatory mechanisms paving the  way for approving autonomous AI systems  that can truly overcome the shortage of  healthcare personnel and improve on human-based diagnostic outcomes—even in  cases where there isn’t a shortage of qualified  personnel—require two distinct spheres of  improvement. First, in AI itself, with strong  foundational models that have generative  capacity but are anchored in relevant  knowledge and are free from hallucinations.  Second, in the data that goes into the  foundational knowledge. 

Presence of diverse datasets of suitable quality,  with equitable representation of different  patient groups, is a necessary step in the  development and testing of future AI solutions.  However, this is far from reality and the  problem is even more severe when looking at  available data for non-white populations. Existing science is often based on  clinical trials using datasets that are not  representative of the diversity in populations,  so AI systems based on such studies bake in  current biases in our knowledge. There are  three distinct aspects to this: digital data  generation, standardized data formats, and  availability of usable data. Rapidly expanding  digital transformation and a concerted move  towards common data models and dictionaries  is likely to increase usable digital data, but  there are still severe governance challenges in  making such data available. Unresolved  ownership questions are a major contributor to  this data lethargy, as is data-protectionism for  economic or strategic interests. 

The ownership of health data varies across the  globe. While US federal law gives patients legal  privacy, security and accuracy rights related to  their health data, they are not treated as owners. Personal data laws vary across states and consumers typically do not have rights to  demand that their data residing with service  providers not be used without their permission.  Current interpretations typically allow service  providers the right to use such data to improve  services, which can have a wide meaning in  health. In contrast, the EU is more explicit  about ownership vesting with the patient, who  has the right to restrict its usage by others,  including after deidentification. Other  countries, such as India, place importance on health data privacy and acknowledge  ownership, but have permissive laws for use  of deidentified data. Given that true  deidentification is nearly impossible, especially for data like genomes, this may not  be sufficient. There have been attempts to bypass this problem altogether by using  purpose-limitation and data solidarity, rather  than ownership and consent, as the ethical  underpinning, alongside technical solutions  like federated learning that allow AI algorithm  development without any direct access to  discrete identifiable data.