Use of AI in MI services

Executive Summary

Evidence-based medicine remains at the forefront of UKMi’s principles. UKMi acknowledges the increasing use of AI in the medical arena. It offers transformative opportunities for improving MI services by enhancing efficiency and therefore supporting clinical decision making and final answers to enquiries. It may also offer support in teaching and training activities from voice overs in presentations, to summarising complex subjects or the creation of slides, bulletins and newsletters. However, its integration must be approached cautiously due to inherent risks such as inaccuracies, biases, data privacy concerns, and lack of transparency.

UKMi recognises that AI may be used to augment human intelligence but not replace human intelligence. For this reason, MI practitioners/pharmacy professionals will still be required to provide evidence-based answers tailored to the specific needs of the patient or cohort of patients. Until specialised medical-rade AI systems become available, AI should only serve as a supplementary tool under strict oversight rather than a primary source for answering enquiries in MI. The accuracy and validity of the final answer remain the responsibility of the MIpractitioner/pharmacy professional.

AI may be used in technical tasks such as spelling, grammar checking, text extraction and summarisation of non-confidential information. Such written prose should be checked to ensure it meets the required standard of answer for the enquirer. AI may also be used to complement but not replace Standard Search Patterns (SSPs) such as those listed in the UKMi Enquiry Answering Guidelines. Any resources obtained from an AI model must be checked to ensure they are reputable sources of information. Where AI has been used, this needs to be documented on MiDatabank/MiDBOnline.

Given the limitations of use, the potential health and patient safety risks, in addition to reputational risks, the currently available AI models readily available online or via mobile apps, including but not limited to ChatGPT, Gemini, Perplexity and Copilot, should not be used routinely or alone in the derivation f  final answer to MI enquiries. In addition, copyrighted material, sensitive data or information such as patient-specific or enquirer specific information should not be provided to current AI servers.

By adhering to future developing governance standards in healthcare technology and continuing to prioritise patient safety, the potential benefits of AI can be realised responsibly. UKMi will continue to monitor the use and developments around AI and will review this position as we gain more insights and experience. 

Background

In November 2022, OpenAI released ChatGPT, an AI model based on Generative Pre-trained Transformer (GPT) architecture, which gave birth to the AI revolution.¹ Fast forwarding to today, AI has gained prominence in various healthcare sectors such as medical imaging.² The Royal Pharmaceutical Society recognises that the introduction of AI tools within NHS England, NHS Scotland and NHS Wales may enhance patient access to care, improve patient experience, support clinical decision-making and improve the safety and efficiency of the medicines supply chain.³

It is evident that although AI has significant potential in healthcare, it has demonstrated notable limitations. AI offers significant potential for improving efficiency, streamlining processes, and enhancing access to information however, their application in Medicines Information services (MI) requires careful consideration to ensure safe, effective, and ethical use. While AI technologies have demonstrated promise in areas such as radiology, evidence has shown thatthey currently lack the specialised knowledge and contextual understanding required for complex medical applications in Pharmacy Practice and Medicines Information.^4-7 This position statement outlines the opportunities, challenges, and recommendations for incorporating AI into Medicines Information ethically and responsibly, ensuring any potential risks are mitigated.

AI has made significant strides since the release of ChatGPT in November 2022, impacting a variety of sectors including healthcare. Whilst AI enhances user interactions and data analysis, it also poses numerous challenges. In healthcare, AI systems are starting to develop – for example, in the field of medical imaging; however, it currently lacks the specialised knowledge and understanding needed for complex medical applications. AI may allow us to augment rational intelligence but currently lacks the ability to demonstrate social, spiritual, cultural and emotional intelligence – the application of which is required in the current and future NHS.

One of the major concerns is that AI models can produce incorrect outputs, known as ‘hallucinations’, where inaccurate information is presented convincingly and therefore risk a high degree of confirmation bias. AI may also not access all the available and/or reputable sources of information as the current widely used AI models only search freely available information; they cannot access subscription resources. This can lead to incomplete and/or entirely or partially incorrect answers to MI enquiries. Additionally, AI raises concerns about data privacy, high implementation costs and the environmental impact.

Understanding Generative AI and Large Language Models (LLMs)

Generative AI is named for its capability to generate a diverse set of outputs such as text, images, audio, computer code, and more. Large Language Models (LLMs) represent just one category of generative AI, focusing specifically on text generation which is of interest in the MI setting. LLMs can power software that aids in language-related tasks and synthetically generates written text, such as drafting e-mails, enhancement of essays,
or summarising long documents.⁸

Generative AI and LLMs are systems designed to predict text based on extensive training data, often sourced from the internet. These models can summarise information, adapt responses for different audiences, and streamline communication.⁸ Despite these strengths, they present notable risks.

Throughout this document the term ‘AI’ is used to cover all sub-groups of AI including LLMs.

Limitations and risks of using AI in Medicines Information

1. AI systems may rely solely on their training data, which exclude certain resources critical for MI (including subscription-only resources). This limitation can result in incomplete or inaccurate and/or entirely or partially incorrect answers to MI enquiries.
2. AI can generate plausible-sounding responses rather than ensuring factual correctness. This can lead to ‘hallucinations’ where fabricated or inaccurate information is presented convincingly, increasing the risk of confirmation bias.
3. Given the variety and complexity of AI models there may be a lack of information and transparency in the mechanistic reasoning of how AI functions. Furthermore, AI can customise data based on user experience. Together these issues raise the problemof the reproducibility of an MI answer.
4. Processing sensitive or confidential data through AI causes a concern with storage of data on external servers and therefore compliance with data regulations such as GDPR and copyright.
5. AI systems require a large amount of hardware and software to function. The training of AI and running large AI models also requires a significant amount of energy which contributes to carbon emissions.

Risk mitigation strategies

To safely integrate AI into Medicines Information workflows, the following measures must be adopted:

1. The currently available AI models readily available online or via mobile apps, such as but not limited to ChatGPT, Gemini, Perplexity and Copilot, should not be used routinely or alone in the derivation of a final answer to MI enquiries.
2. All AI-generated information must be critically appraised for accuracy and relevance by an appropriately trained person. 
3. References suggested by AI must be in addition to Standard Search Patterns (SSPs) and not used in place of SSPs. 
4. All references suggested by AI must be checked for existence and appropriateness before inclusion in developing an MI answer. 
5. Sensitive patient-specific or enquirer-specific data must never be inputted into an AI server. Compliance with data protection  standards is mandatory. 
6. Where AI has been used for its technical outputs, such as spelling or grammar checking, image text extraction and summarisation of non-sensitive data, the resulting written prose should be checked to ensure it meets the required standard of answer for the enquirer. 
7. Where AI has been used, this needs to be documented. For example, on MiDatabank/MIDBOnline under an appropriate research section.

References

1. OpenAI. ChatGPT: Optimising language models for dialogue. San Francisco, 2022. Available at: https://openai.com/ blog/chatgpt [Accessed 04 Apr 2025].
2. The Royal College of Radiologists. AI Registry. AI Registry Listing. 2025. Available at: https://www.rcr.ac.uk/our-services/artificial-intelligence-ai/about-the-ai-registry/ [Accessed 04 Apr 2025].
3. Royal Pharmaceutical Society. Artificial Intelligence (AI) in Pharmacy. 2025. Available at: https://www.rpharms.com/ recognition/all-our-campaigns/policy-a-z/ai [Accessed 04 Apr 2025].
4. Goundrey-Smith, S. Artificial Intelligence in Pharmacy and the Pharmaceutical Industry –Opportunities and Ethics. PM Healthcare Journal. Spring 2024;08:11-8. Available at: https://www.pmhealthcare.co.uk/uploads/Journals/PM%20Healthcare%20Journal%20-%20Spring%202024_Final.pdf [Accessed 04 Apr 2025].
5. Morath B, Chiriac U, Jaszkowski E, et al. Performance and risks of ChatGPT used in drug information: an exploratory real-world analysis. Eur J Hosp Pharm. 2024;31(6):491-7. Available at: https://ejhp.bmj.com/content/31/6/491 [Accessed 08 Sep 2025].
6. Zhang X, Tsang CCS, Ford DD, et al. Student Pharmacists' Perceptions of Artificial Intelligence and Machine Learning in Pharmacy Practice and Pharmacy Education. Am J Pharm Educ. 2024;88(12):101309. Available at: https://www.ajpe.org/article/S0002-9459(24)11028-5/abstract [Accessed 08 Sep 2025].
7. Mahuli, S., Rai, A., Mahuli, A., Kumar, A. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J. 2023;235:90-2. Available at: https://www.nature.com/articles/s41415-023-6132-y [Accessed 04 Apr 2025].
8. Sandhu, J.A. What are LLMs and generative AI? A beginner’s guide to the technology turning heads. University of Toronto Schwartz Reisman Institute for Technology and Society. 25 Jan 2024. Available at: https://srinstitute.utoronto.ca/news/gen-ai-llms-explainer [Accessed 04 Apr 2025].

Author

Anuj K. Sunder, Lead Pharmacist Medicines Information and Medication Safety, Royal Wolverhampton Hospitals NHS Trust, and members of the UKMi Technology and Innovation Group