iCARE Calls for Responsible AI Adoption in the NHS

At the Health Foundation’s 2025 AI in the NHS conference, Erik Mayer urged collaboration to turn AI’s promise into meaningful healthcare impact.

At the Health Foundation’s AI in the NHS 2025 conference on 14 October, leading experts gathered to discuss how artificial intelligence can move “from promise to practice.” Among them was Erik Mayer, Clinical Associate Professor in Digital Health at Imperial College London and Director of iCARE, who spoke about the practical realities of embedding AI into everyday clinical workflows and ensuring its safety, effectiveness, and long-term sustainability.

iCARE sits across Imperial College London and Imperial College NHS Healthcare Trust and receives funding from the National Institute for Health and Care Research (NIHR) Imperial Biomedical Research Centre. It provides enables access to de-identified routinely collected healthcare data for North West London’s 2.7 million population and beyond via a Secure Data Environment, that allows researchers and clinicians to translate analytical insights into better care for patients. The centre focuses on using AI and advanced analytics to improve quality, safety, and patient experience across the NHS. As iCARE director, Mayer is also co-lead of the Digital Health Theme under the NIHR Imperial BRC.

Drawing on iCARE’s work, Mayer highlighted the value of linking structured and unstructured health data to derive real-time insights. “By combining them, we can identify risks and trends earlier and feed that learning directly back into the clinical front end” he explained.

Mayer’s remarks also underscored a recurring theme across the conference: that AI must be evaluated continuously, not only before deployment but throughout its use in the NHS. He noted that automating parts of this evaluation will be essential if the NHS is to manage the thousands of algorithms expected to be in use in the coming years. “We cannot rely on armies of data engineers,” he said. “Evaluation itself will need to become data-driven and automated.”

Other panellists echoed his emphasis on responsible scaling. Consultant in Older Adult Medicine and Innovation Fellow at NHS Lancashire, Dr Ana Talbot, warned of “pilotitis,” where local successes fail to translate into national benefit, while Principal AI Scientist at Newton’s Tree, Dr Aysha Luis, drew a distinction between “evaluation and monitoring,” urging continuous oversight once AI systems are live.

Mayer also emphasised the human dimension of digital transformation. “The key is taking the workforce on the journey,” he said. “We must design AI that fits clinical workflows, not the other way around. Otherwise, we risk creating alert fatigue or mistrust.” He pointed out that the integration of AI into NHS systems must not only improve outcomes but also reinforce staff confidence and patient trust.

Throughout the discussion, Mayer’s contributions reflected iCARE’s broader mission: to translate innovation into sustainable impact. By embedding evaluation into everyday data flows and ensuring that insights return to frontline teams, iCARE’s model offers a glimpse of how the NHS could operationalise the government’s forthcoming AI strategic roadmap.

The panel “From Promise to Practice” captured the dual challenge now facing the NHS — to harness AI’s potential responsibly and to align it with the systems, governance, and human factors that underpin safe care.

The full discussion is available to watch online via the Health Foundation’s website.