Use of AI by NHS Healthcare Professionals

Executive Summary

Evidence-based medicine remains fundamental to NHS practice across all care settings. The NHS acknowledges the transformative potential of Artificial Intelligence (AI) in healthcare. The NHS England 10-Year Health Plan identifies AI as one of five strategic priority technologies alongside data, genomics, wearables, and robotics.¹ AI offers significant opportunities for improving medicines optimisation, enhancing clinical decision-making, and supporting patient safety initiatives.² However, integration of AI must be approached with caution due to inherent risks including potential for inaccuracies such as ‘hallucinations’, algorithmic biases, data privacy concerns, and lack of transparency in decision-making processes.

Until specialised, clinically validated AI systems become available through established regulatory pathways such as the NHS TEST framework or the MHRA AI Airlock programme, AI should not be used as a primary source for answering medicine-related enquiries.^3,4 This position statement affirms that current AI models must not replace reputable sources of information such as the BNF, other reputable reference sources or professional clinical judgement. All NHS professionals, remain responsible for providing evidence-based, patient centred care. 

AI can be utilised for technical tasks including text summarisation or administrative functions, provided that appropriate safeguards are implemented. However, it is recommended that medicines-related AI-generated content must be critically evaluated by an appropriately trained healthcare professional. The accuracy and validity of the medicines related information or advice provided, remain the responsibility of the healthcare professional providing it.

Given current limitations and patient safety considerations, publicly available Large Language Models (LLMs) such as, but not limited to, ChatGPT, Copilot and Gemini should not be used routinely or independently for clinical decision-making regarding medicines. Patient identifiable information must never be entered into AI systems due to data protection requirements.⁵ NHS professionals who seek information or advice regarding medicines related questions should use appropriate resources or seek advice from the most appropriate healthcare professional. Where available, this may be their designated pharmacist, their local Medicines Information Centre, or other specialists where necessary.

By adhering to emerging regulatory frameworks and maintaining patient safety as the paramount concern, the NHS can harness AI's potential whilst minimising associated risks. This position statement will be reviewed regularly as evidence and regulatory guidance evolve.

Background

In November 2022, OpenAI released ChatGPT, an AI model based on Generative Pre-trained Transformer (GPT) architecture, which gave birth to the AI revolution.⁶ Since then, there has been widespread interest in Large Language Models (LLMs) across healthcare and the NHS has begun to implement numerous AI initiatives, including those used in medical imaging, patient safety warning systems and screening programmes.^{7, 8}

The MHRA has established the AI Airlock regulatory sandbox to support safe development and deployment of AI-enabled medical devices.^9,10 This initiative represents the UK's leadership in AI healthcare regulation focusing on safe AI implementation.⁴ Current NHS England guidance emphasises responsible AI adoption, with specific requirements for data protection and compliance with UK GDPR regulations.¹¹ 

Despite significant potential, evidence demonstrates that current AI systems lack the specialised medical knowledge required for complex medicines-related decision-making and that additional training is required to understand AI systems.^12,13 AI models may allow us to augment rational intelligence but currently, it lacks the ability to demonstrate social, cultural and emotional intelligence – the application of which is required for any health system.

One of the major concerns is that AI models can produce incorrect outputs, known as ‘hallucinations’, where inaccurate information is presented convincingly and therefore risks a high degree of confirmation bias. In addition, AI models may not have the ability to access all the available and/or reputable sources of information such as those requiring a subscription and may only search freely available information. This can lead to incomplete and/or entirely or partially incorrect answers to medicines-related questions. Additionally, AI raises concerns about data privacy, the high implementation costs and the environmental impact.

Understanding Generative AI and LLMs

Generative AI is named for its capability to generate a diverse set of outputs such as text, images, audio, computer code, and more. Large Language Models (LLMs) represent just one category of generative AI, focusing specifically on text generation which is currently the main output from AI models. LLMs can power software that aids in language-related tasks and synthetically generates written text, such as drafting emails, enhancement of essays, or summarising long documents.¹⁴

Despite these strengths, they present notable risks. These systems function by predicting text sequences based on extensive training datasets, enabling capabilities such as document summarisation, patient communication assistance, and clinical note generation.^15,16 However, Generative AI and LLMs operate through statistical prediction rather than clinical reasoning, creating inherent limitations in medical applications. They are designed to predict text based on extensive training data, often sourced from the internet. These models can summarise information, adapt responses for different audiences, and streamline communication. 

Throughout this document the term ‘AI’ is used to cover all sub-groups of AI including Generative AI and LLMs

Limitations and risks of using AI in answering medicine-related questions

1. Training data bias and limitations. AI models learn from their training data, which may contain embedded biases or may not be current. This means that AI may not be aware of recent up to date evidence such as new NICE guidance, or the latest MHRA Drug Safety Alert. Furthermore, publicly available AI models may not be able to access subscription-only medical databases, clinical guidelines, or proprietary healthcare resources that form the foundation of evidence-based practice. This limitation results in incomplete knowledge biases that may omit critical safety information such as drug interactions, or recent regulatory updates.
2. ‘Hallucinations’ or fabricated information. AI models frequently generate plausible-sounding but factually incorrect responses, particularly in specialised medical domains.¹⁷ AI may create fake journal citations or incorrect dosing information, that appears highly credible.¹⁸ These 'hallucinations' present significant patient safety risks when healthcare professionals accept AI-generated information to medicines-related questions without verification.
3. Lack of clinical context. Publicly available AI models may not be aware of individual patient factors including blood results, patient comorbidities or specific clinical circumstances that influence prescribing decisions. The absence of contextual understanding limits the clinical applicability of AI-generated recommendations to medicines-related questions.
4. Lack of verifiable sources. Given the variety and complexity of AI models, there may be a lack of information and transparency in the mechanistic reasoning of how AI functions. Furthermore, AI can customise data based on user experience lacking reproducibility. This means that the same medicines-related question may be answered differently when asked on another occasion.
5. Data Privacy and Security Concerns. A Data Protection Impact Assessment (DPIA) must legally be completed prior to implementing AI-based technologies. Processing patient information through publicly available AI models raises significant data protection issues, with potential violations of UK GDPR and NHS information governance standards.¹⁷

Risk mitigation strategies of using AI in answering medicine-related questions

To ensure safe integration of AI technologies in medicine-related decision-making, NHS staff must implement the following measures:

1. Follow local Trust or Health Board policies. All NHS staff must adhere to their local NHS Trust's policies on the use of AI and information governance.
2. Do not solely rely on current AI models. The reputable sources of information should be used when answering medicines-related questions. Publicly available AI models such as, but not limited to, ChatGPT, Copilot and Gemini should not be used as a sole resource for answering medicines-related questions. These systems can supplement, but not replace, established clinical resources or professional expertise. NHS professionals must maintain competency in the use of standard clinical resources and decision-making processes.¹⁹ It may support in the finding of reputable guidelines or evidence e.g. current NICE guidelines, but all sources should be checked for verification.
3. Mandatory clinical verification. Any AI-generated medicines information must be verified by an appropriately qualified healthcare professional, such as a pharmacist, before clinical application. This verification process must include cross-referencing with authoritative clinical sources such as the BNF, BNFC, NICE guidance or a pharmacist.
4. Data protection and security. Sensitive or patient-specific data must never be entered into AI models. All AI implementations must comply with UK GDPR requirements and undergo Data Protection Impact Assessments as mandated by NHS information governance standards.¹⁷
5. Maintain professional accountability. Outputs derived from AI always require a human-in-the-loop (HITL) process.²⁰ All NHS professionals are fully responsible and accountable for any clinical advice they provide, regardless of the tools they use to formulate it.¹⁹
6. Professional development. NHS professionals utilising AI technologies must maintain appropriate training in both AI capabilities and limitations, ensuring competent oversight of AI-generated outputs. This includes understanding when to abstain from AI use in complex clinical medicines-related scenarios.

References

1. Leiu A. NHS 10-Year Health Plan: AI, Data & Digital Transformation. 18 Jul 2025. Available at: https://www.burges-salmon.com/ articles/102ku3j/nhs-10-year-health-plan-ai-data-digital-transformation/ [Accessed 08 Sep 2025].
2. NHS England. Medicines optimisation. Available at: https://www.england.nhs.uk/medicines-2/medicines-optimisation/ [Accessed 08 Sep 2025].
3. Health Innovation Network. NHS Test: An intelligent framework to choose new technologies. Available at: https://healthinnovationnetwork.com/resources/nhs-test-an-intelligent-framework-to-choose-new-technologies/ [Accessed 08 Sep 2025].
4. GOV.UK. UK MHRA leads safe use of AI in healthcare as first country in new global network. 23 Jun 2025. Available at: https://www.gov.uk/government/news/uk-mhra-leads-safe-use-of-ai-in-healthcare-as-first-country-in-new-global-network [Accessed 08 Sep 2025].
5. NHS England Transformation Directorate. Artificial Intelligence - Information Governance Guidance. 30 April 2025. Available at: https://transform.england.nhs.uk/information-governance/guidance/artificial-intelligence/ [Accessed 08 Sep 2025].
6. OpenAI. ChatGPT: Optimising language models for dialogue. 2022. Available at: https://openai.com/blog/chatgpt [Accessed 08 Sep 2025].
7. GOV.UK. World-first AI system to warn of NHS patient safety concerns. 29 Jun 2025. Available at: https://www.gov.uk/government/ news/world-first-ai-system-to-warn-of-nhs-patient-safety-concerns [Accessed 08 Sep 2025].
8. GOV.UK. AI to be trialled at unprecedented scale across NHS screening. 22 Sep 2025. Available at: https://www.gov.uk/government/ news/ai-to-be-trialled-at-unprecedented-scale-across-nhs-screening [Accessed 17 Nov 2025].
   © UKMi, December 2025. All rights reserved Page 4 of 4
9. Digital Health. MHRA AI Airlock regulatory sandbox receives £1m investment. 24 Jun 2025. Available at: https://www.digitalhealth.net/2025/06/mhra-ai-airlock-regulatory-sandbox-receives-1m-investment/ [Accessed 08 Sep 2025].
10. MedRegs. Exploring AI in Healthcare: Insights from the AI Airlock Pilot. 26 Mar 2025. Available at: https://medregs.blog.gov.uk/2025/03/26/exploring-ai-in-healthcare-insights-from-the-ai-airlock-pilot/ [Accessed 08 Sep 2025].
11. AI and Digital Regulations Service. Learn what regulations to follow. Available at: https://www.digitalregulations.innovation.nhs.uk/ [Accessed 08 Sep 2025].
12. Morath B, Chiriac U, Jaszkowski E, et al. Performance and risks of ChatGPT used in drug information: an exploratory real-world analysis. Eur J Hosp Pharm. 2024;31(6):491-7. Available at: https://ejhp.bmj.com/content/31/6/491 [Accessed 08 Sep 2025].
13. Zhang X, Tsang CCS, Ford DD, et al. Student Pharmacists' Perceptions of Artificial Intelligence and Machine Learning in Pharmacy Practice and Pharmacy Education. Am J Pharm Educ. 2024;88(12):101309. Available at: https://www.ajpe.org/article/S0002-9459(24)11028-5/abstract [Accessed 08 Sep 2025].
14. Sandhu, J.A. What are LLMs and generative AI? A beginner’s guide to the technology turning heads. University of Toronto Schwartz Reisman Institute for Technology and Society. 25 Jan 2024. Available at: https://srinstitute.utoronto.ca/news/gen-ai-llms-explainer [Accessed 08 Sep 2025].
15. Huang W, Wang X, Chen Y, et al. Applications of artificial intelligence in pharmaceutical practice and education. Front Pharmacol. 2025;16:1618701. Available at: https://www.frontiersin.org/journals/pharmacology/articles/10.3389/fphar.2025.1618701/full [Accessed 08 Sep 2025].
16. NHS England. Guidance on the use of AI-enabled ambient scribing products in health and care settings. 29 Apr 2025. Available at:
    https://www.england.nhs.uk/long-read/guidance-on-the-use-of-ai-enabled-ambient-scribing-products-in-health-and-care-settings/ [Accessed 08 Sep 25]
17. Royal Pharmaceutical Society. Artificial Intelligence (AI) in Pharmacy. 2025. Available at: https://www.rpharms.com/recognition/all-our-campaigns/policy-a-z/ai [Accessed 08 Sep 2025].
18. iatroX Clinical Team. Artificial intelligence in healthcare (2025): real-world uses, risks, NHS/EU/US rules, and how to deploy safely. 12 Aug 2025. Available at: https://www.iatrox.com/blog/artificial-intelligence-in-healthcare-uses-risks-governance-2025 [Accessed 08 Sep 2025].
19. Sunder A.K. et al. Position Statement on the use of Artificial Intelligence (AI) in Medicines Information (MI) or Medicines Advice Services. May 2025. Available at: Use of AI in MI Services [Accessed 08 Oct 2026]
20. Stryker C. IBM Think. What is human-in-the-loop? Available at: https://www.ibm.com/think/topics/human-in-the-loop [Accessed 17Nov 2025]

Author

Anuj K. Sunder, Lead Pharmacist Medicines Information and Medication Safety, Royal Wolverhampton Hospitals NHS Trust, and members of the UKMi Technology and Innovation Group