Publication No. 1
Publication Data: 2023/4/8
Title: Talk with ChatGPT About the Outbreak of Mpox in 2022: Reflections and Suggestions from AI Dimensions
Link: https://doi.org/10.1007/s10439-023-03196-z
Full Author: Kunming Cheng, Yongbin He, Cheng Li, Ruijie Xie, Yanqiu Lu, Shuqin Gu, Haiyang Wu
Summary: In the era of big data, generative artificial intelligence (AI) models are currently in a boom. The Chatbot Generative Pre-trained Transformer (ChatGPT), a large language model (LLM) developed by OpenAI (San Francisco, CA), is a type of AI software that could generate text based on the input it receives. In this study, in order to explore how ChatGPT could give reflections and suggestions about the sudden outbreak of Mpox in 2022 from the AI dimensions, our group talked with ChatGPT with several questions about Mpox. We hope this talk could enrich our knowledge on Mpox from the new AI dimensions and also explore the possibility of human and AI fight shoulder to shoulder for prevention and containment of the potential epidemics or pandemics in future.
Citation: Cheng, Kunming, et al. “Talk with ChatGPT about the outbreak of Mpox in 2022: reflections and suggestions from AI dimensions.” Annals of Biomedical Engineering (2023): 1-5.
Publication No. 2
Publication Data: 2023/4/10
Title: The potential impact of ChatGPT/GPT-4 on surgery: will it topple the profession of surgeons?
Full Author: Kunming Cheng, Zaijie Sun, Yongbin He, Shuqin Gu, Haiyang Wu Highlights
Link: https://doi.org/10.1097/JS9.0000000000000388
Summary: This is the first study to summarize the potential applications of ChatGPT (Generative Pre-trained Transformer)/GPT-4 in the surgical field. ChatGPT/GPT-4 is capable of participating in multiple aspects of surgical work, including scientific writing, doctor–patient communication, diagnostic imaging, and patients’ perioperative management. ChatGPT/GPT-4 could be a good assistant for surgeons, but it was not possible to topple the profession of surgeons.
Citation: Cheng, Kunming, et al. “The potential impact of ChatGPT/GPT-4 on surgery: will it topple the profession of surgeons?.” International Journal of Surgery (2023): 10-1097.
Publication No. 3
Publication Data: 2023/4/18
Title: Will ChatGPT/GPT-4 be a Lighthouse to Guide Spinal Surgeons?
Link: https://doi.org/10.1007/s10439-023-03206-0
Full Author: Yongbin He, Haifeng Tang, Dongxue Wang, Shuqin Gu, Guoxin Ni, Haiyang Wu
Summary: The advent of artificial intelligence (AI), particularly ChatGPT/GPT-4, has led to advancements in various fields, including healthcare. This study explores the prospective role of ChatGPT/GPT-4 in various facets of spinal surgical practice, especially in supporting spinal surgeons during the perioperative management of endoscopic spinal surgery for patients with lumbar disc herniation. The AI-driven chatbot can facilitate communication between spinal surgeons, patients, and their relatives, streamline the collection and analysis of patient data, and contribute to the surgical planning process. Furthermore, ChatGPT/GPT-4 may enhance intraoperative support by providing real-time surgical navigation information and physiological parameter monitoring, as well as aiding in postoperative rehabilitation guidance. However, the appropriate and supervised use of ChatGPT/GPT-4 is essential, considering the potential risks associated with data security and privacy. The study concludes that ChatGPT/GPT-4 can serve as a valuable lighthouse for spinal surgeons if used correctly and responsibly.
Citation: He, Yongbin, et al. “Will ChatGPT/GPT-4 be a lighthouse to guide spinal surgeons?.” Annals of Biomedical Engineering (2023): 1-4.
Publication No. 4
Publication Data: 2023/4/19
Title: Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?
Full Author: Lingxuan Zhu, Weiming Mou, Rui Chen
Link: https://doi.org/10.1186/s12967-023-04123-5
Summary: This article evaluates whether large language models (LLMs), like ChatGPT and others with internet access, can address prostate cancer patients’ questions and democratize medical knowledge. It discusses the accuracy, comprehensiveness, and humanistic care in responses from various LLMs to 22 questions based on clinical guidelines and experience. The study finds that most LLMs provide high-quality responses, particularly ChatGPT, which had the highest accuracy rate. However, it also notes limitations, including occasional inaccuracies and lack of depth in some answers, highlighting that while LLMs have potential in patient education and support, they cannot yet replace professional medical advice.
Citation: Zhu, Lingxuan, Weiming Mou, and Rui Chen. “Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?.” Journal of Translational Medicine 21.1 (2023): 1-4.
Publication No. 5
Publication Data: 2023/4/19
Title: Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example
Link: https://doi.org/10.1007/s10439-023-03203-3
Full Author: Kunming Cheng, Zhiyong Li, Yongbin He, Qiang Guo, Yanqiu Lu, Shuqin Gu, Haiyang Wu
Summary: Over the past month, a new AI model called Chatbot Generative Pre-trained Transformer (ChatGPT), has received enormous attention in the media and scientific communities due to its ability to process and respond to commands in a humanistic fashion. As reported, five days after its launch, the number of registered users of ChatGPT exceeded one million, and its monthly active users had exceeded 100 million two months later, making it the most rapidly growing consumer application in history. The advent of ChatGPT has further brought about new ideas and challenges in the realm of infectious disease. In view of this, in order to evaluate the potential use of ChatGPT in clinical practice and scientific research of infectious disease, we conducted a brief online survey by using the publicly available ChatGPT webpage. Also, the present study also talks about the relevant social and ethical issues related to this program.
Citation: Cheng, Kunming, et al. “Potential use of artificial intelligence in infectious disease: take ChatGPT as an example.” Annals of Biomedical Engineering (2023): 1-6.
Publication No. 6
Publication Data: 2023/4/28
Title: Exploring the Potential of GPT-4 in Biomedical Engineering: The Dawn of a New Era
Link: https://doi.org/10.1007/s10439-023-03221-1
Full Author: Kunming Cheng, Qiang Guo, Yongbin He, Yanqiu Lu, Shuqin Gu, Haiyang Wu.
Summary: Biomedical engineering is a relatively young interdisciplinary field based on engineering, biology, and medicine. Of note, the rapid progress of artificial intelligence (AI)-based technologies has made a significant impact on the biomedical engineering field, and continuously bring innovations and breakthroughs. Recently, ChatGPT, an AI chatbot developed by OpenAI company, has gained tremendous attention due to its powerful natural language generation and understanding ability. In this study, we explored potential of GPT-4 in the eight branches of biomedical engineering including medical imaging, medical devices, bioinformatics, biomaterials, biomechanics, gene and cell engineering, tissue engineering, and neural engineering. Our results show that the application of GPT-4 will bring new opportunities for the development of this field.
Citation: Cheng, Kunming, et al. “Exploring the potential of GPT-4 in biomedical engineering: the dawn of a new era.” Annals of Biomedical Engineering (2023): 1-9.
Publication No. 7
Publication Data: 2023/5/10
Title: Can ChatGPT/GPT-4 assist surgeons in confronting patients with Mpox and handling future epidemics?
Link: https://doi.org/10.1097/JS9.0000000000000453
Full Author: Yongbin He, Haiyang Wu, Yan Chen, Dewei Wang, Weiming Tang, M. Anthony Moody, Guoxin Ni, Shuqin Gu
Summary: The article discusses the potential of ChatGPT/GPT-4 in assisting surgeons with managing patients with Mpox during the outbreak and in future epidemics. It highlights the importance of differentiating Mpox from common diseases, outlines the necessary precautions and guidelines for surgeons, and evaluates ChatGPT/GPT-4’s capability to provide comprehensive and organized information relevant to surgical care. The study suggests that with ChatGPT’s assistance, surgeons can more confidently and effectively manage Mpox cases and be better prepared for future health crises.
Citation: He, Yongbin, et al. “Can ChatGPT/GPT-4 assist surgeons in confronting patients with Mpox and handling future epidemics?.” International Journal of Surgery 109.8 (2023): 2544-2548.
Publication No. 8
Publication Data: 2023/5/29
Title: WHO declares the end of the COVID-19 global health emergency: lessons and recommendations from the perspective of ChatGPT/GPT-4
Link: https://doi.org/10.1097/JS9.0000000000000521
Full Author: Kunming Cheng, Chunchun Wu, Shuqin Gu, Yanqiu Lu, Haiyang Wu, Cheng Li
Summary: The article discusses the WHO’s declaration ending the COVID-19 global health emergency, emphasizing the shift towards long-term management of the virus. It highlights the role of AI and ChatGPT/GPT-4 in pandemic response and future vigilance, suggesting that despite the downgrade, COVID-19 remains a health threat requiring ongoing surveillance and vaccine development. The piece also addresses the pandemic’s impact on healthcare practices, particularly in surgery, and underscores the importance of leveraging lessons learned for future preparedness.
Citation: Cheng, Kunming, et al. “WHO declares end of COVID-19 global health emergency: lessons and recommendations from the perspective of ChatGPT/GPT-4.” International Journal of Surgery (2023): 10-1097.
Publication No. 9
Publication Data: 2023/6/22
Title: ChatGPT can pass the AHA exams: Open-ended questions outperform multiple-choice format
Link: https://doi.org/10.1016/j.resuscitation.2023.109783
Full Author: Lingxuan Zhu, Weiming Mou, Tao Yang, Rui Chen
Summary: The study by Fijačko et al. tested ChatGPT’s ability to pass the BLS and ACLS exams of AHA, but found that ChatGPT failed both exams. A limitation of their study was using ChatGPT to generate only one response, which may have introduced bias. When generating three responses per question, ChatGPT can pass BLS exam with an overall accuracy of 84%. When incorrectly answered questions were rewritten as open-ended questions, ChatGPT’s accuracy rate increased to 96% and 92.1% for the BLS and ACLS exams, respectively, allowing ChatGPT to pass both exams with outstanding results.
Citation: Zhu, Lingxuan, et al. “ChatGPT can pass the AHA exams: Open-ended questions outperform multiple-choice format.” Resuscitation 188 (2023): 109783.
Publication No. 10
Publication Data: 2024/1/22
Title: Can DALL-E 3 Reliably Generate 12-Lead ECGs and Teaching Illustrations?
Link: https://doi.org/10.7759/cureus.52748
Full Author: Lingxuan Zhu, Weiming Mou, Keren Wu, Jian Zhang, Peng Luo
Summary: The recent integration of the latest image generation model DALL-E 3 into ChatGPT allows text prompts to easily generate the corresponding images, enabling multimodal output from ChatGPT. We explored the feasibility of DALL-E 3 for drawing a 12-lead ECG and found that it can draw rudimentary 12-lead electrocardiograms (ECG) displaying some of the parameters, although the details are not completely accurate. We also explored DALL-E 3’s capacity to create vivid illustrations for teaching resuscitation-related medical knowledge. DALL-E 3 produced accurate CPR illustrations emphasizing proper hand placement and technique. For ECG principles, it produced creative heart-shaped waveforms tying ECGs to the heart. With further training, DALL-E 3 shows promise to expand easy-to-understand visual medical teaching materials and ECG simulations for different disease states. In conclusion, DALL-E 3 has the potential to generate realistic 12-lead ECGs and teaching schematics, but expert validation is still needed.
Citation: Zhu, Lingxuan, et al. “Can DALL-E 3 Reliably Generate 12-Lead ECGs and Teaching Illustrations?.” Cureus 16.1 (2024).
Publication No. 11
Publication Data: 2024/2/26
Title: Potential of Large Language Models as Tools Against Medical Disinformation
Link: [https://doi.org/10.1001/jamainternmed.2024.0020]{.underline}
Full Author: Lingxuan Zhu, Weiming Mou, Peng Luo
Summary: This letter discusses the potential of large language models (LLMs) like ChatGPT to combat medical disinformation. While acknowledging the risks of generating false medical information, the authors argue that LLMs can also serve as valuable tools in identifying and countering health misinformation. They tested several LLMs with misleading posts and found a significant portion of the responses correctly identified the misinformation, often providing scientifically grounded explanations and urging caution. This underscores the dual potential of LLMs in both spreading and fighting disinformation, suggesting the importance of harnessing these technologies responsibly.
Citation: Zhu, Lingxuan, et al. “Potential of Large Language Models as Tools Against Medical Disinformation” JAMA Internal Medicine (2024).
Publication No. 12
Publication Data: 2024/2/1
Title: STAGER checklist: Standardized testing and assessment guidelines for evaluating generative artificial intelligence reliability
Link: https://doi.org/10.1002/imo2.1
Full Author: Jinghong Chen, Lingxuan Zhu, Weiming Mou, Anqi Lin, Dongqiang Zeng, Chang Qi, Zaoqu Liu, Aimin Jiang, Bufu Tang, Wenjie Shi, Ulf D Kahlert, Jianguo Zhou, Shipeng Guo, Xiaofan Lu, Xu Sun, Trunghieu Ngo Zhongji Pu, Baolei Jia, Che Ok Jeon, Yongbin He, Haiyang Wu, Shuqin Gu, Wisit Cheungpasitporn, Haojie Huang, Weipu Mao, Shixiang Wang, Xin Chen, Loïc Cabannes, Gerald Sng Gui Ren, Iain S Whitaker, Stephen Ali, Quan Cheng, Kai Miao, Shuofeng Yuan, Peng Luo
Summary: Generative artificial intelligence (AI) holds immense potential in medical applications. Numerous studies have explored the efficacy of various generative AI models within healthcare contexts, but there is a lack of a comprehensive and systematic evaluation framework. Given that some studies evaluating the ability of generative AI for medical applications have deficiencies in their methodological design, standardized guidelines for their evaluation are also currently lacking. In response, our objective is to devise standardized assessment guidelines tailored for evaluating the performance of generative AI systems in medical contexts. To this end, we conducted a thorough literature review using the Web of Sciences, Cochrane Library, PubMed, and Google Scholar databases, focusing on research that tests generative AI capabilities in medicine. Our multidisciplinary team, comprising experts in life sciences, clinical medicine, medical engineering, and generative AI users, conducted several discussion sessions and developed a checklist of 32 items. The checklist is designed to encompass the critical evaluation aspects of generative AI in medical applications comprehensively. This checklist, and the broader assessment framework it anchors, address several key dimensions, including question collection, querying methodologies, and assessment techniques. We aim to provide a holistic evaluation of AI systems. The checklist delineates a clear pathway from question gathering to result assessment, offering researchers guidance through potential challenges and pitfalls. Our framework furnishes a standardized and systematic approach for research involving the testing of generative AI’s applicability in medicine. It enhances the quality of research reporting and aids in the evolution of generative AI in medicine and life sciences.
Citation: Chen, Jinghong, et al. “STAGER checklist: Standardized testing and assessment guidelines for evaluating generative artificial intelligence reliability” iMetaomics (2024).
Publication No. 13
Publication Data: 2024/3/4
Title: What is the best approach to assessing generative AI in medicine?
Link: https://doi.org/10.1016/j.resuscitation.2024.110164
Full Author: Lingxuan Zhu, Weiming Mou, Jiarui Xie, Peng Luo, Rui Chen
Summary: The article discusses the assessment of generative AI technologies like ChatGPT in the field of clinical medicine. Specifically, it addresses their capabilities in passing the American Heart Association’s Basic Life Support (BLS) and Advanced Cardiovascular Life Support (ACLS) exams. Initial studies showed that ChatGPT-3.5 could not pass these exams; however, modifications in test formats, such as converting multiple-choice questions to open-ended questions, enabled the AI to succeed in later assessments. The launch of ChatGPT-4V, which includes image recognition capabilities, further enhanced its performance by allowing it to handle image-based questions, simulating a real exam environment.
The article highlights the importance of including methodological details like version numbers and test dates in research to enhance the validity and reproducibility of studies. It suggests that evaluating AI should extend beyond structured exams to more dynamic, simulated clinical scenarios to better gauge its potential in real-world medical settings. The authors advocate for a comprehensive evaluation process that mirrors the progression of medical education from simple tests to complex clinical practice, thereby assessing the practical impact of AI technologies in healthcare.
Citation: Zhu, Lingxuan, et al. “What is the Best Approach to Assessing Generative AI in Medicine?.” Resuscitation (2024).
Publication No. 14
Publication Data: 2024/3/18
Title: Step into the era of large multimodal models: A pilot study on ChatGPT-4V(ision)’s ability to interpret radiological images
Link: https://doi.org/10.1097/JS9.0000000000001359
Full Author: Lingxuan Zhu, Weiming Mou, Yancheng Lai, Jinghong Chen, Shujia Lin, Liling Xu, Junda Lin, Zeji Guo, Tao Yang, Anqi Lin, Chang Qi, Ling Gan, Jian Zhang, Peng Luo
Summary: This study explores ChatGPT-4V’s ability to interpret radiological images, assessing its diagnostic accuracy and capacity to formulate treatment plans. The model displayed a 77.01% diagnostic accuracy on USMLE-style questions, demonstrating enhanced performance when provided with comprehensive patient histories. Although effective in identifying abnormalities in chest X-rays, it struggled with precise diagnoses due to limited patient data. The findings suggest that while ChatGPT-4V can integrate imaging with patient histories effectively, further enhancements and extensive patient data are necessary for accurate medical diagnostics.
Citation: Zhu, Lingxuan, et al. “Step into the era of large multimodal models: A pilot study on ChatGPT-4V (ision)’s ability to interpret radiological images.” International Journal of Surgery (2024): 10-1097.
Publication No. 15
Publication Data: 2024/3/29
Title: Language and cultural bias in AI: comparing the performance of large language models developed in different countries on Traditional Chinese Medicine highlights the need for localized models
Link: https://doi.org/10.1186/s12967-024-05128-4
Full Author: Lingxuan Zhu, Weiming Mou, Yancheng Lai, Junda Lin, Peng Luo
Summary: The study presented in the article evaluates the performance of large language models (LLMs) developed in China and the West when answering questions related to Traditional Chinese Medicine (TCM). The investigation reveals a significant disparity in performance, attributed to the linguistic and cultural nuances inherent in TCM, which are better captured by models trained on Chinese-language data. Chinese LLMs like Baidu’s Ernie Bot series and Alibaba’s Qwen-max, which are specifically tailored with extensive local data, demonstrate superior accuracy compared to Western models such as OpenAI’s ChatGPT and Google’s Gemini-pro. The research used a sample of 140 questions from the National Medical Licensing Examination for TCM in China, showing that Chinese LLMs had an average accuracy of 78.4%, significantly higher than their Western counterparts at 35.9%. This study underscores the critical role of localized training in enhancing LLM performance, suggesting a need for developing models that can understand and interact within specific linguistic and cultural contexts, thus providing more reliable and culturally coherent AI applications in fields like medicine.
Citation: Zhu, Lingxuan, et al. “Language and cultural bias in AI: comparing the performance of large language models developed in different countries on Traditional Chinese Medicine highlights the need for localized models.” Journal of Translational Medicine 22.1 (2024): 319.
Publication No. 16
Publication Data: 2024/4/9
Title: Multimodal Approach in the Diagnosis of Urologic Malignancies: Critical Assessment of ChatGPT-4V’s Image-Reading Capabilities
Link: https://doi.org/10.1200/CCI.23.00275
Full Author: Lingxuan Zhu, Weiming Mou, Yancheng Lai, Junda Lin, Peng Luo
Summary: The study evaluates the potential application of the ChatGPT-4V model, a large multimodal language model, in the pathological diagnosis of urologic malignancies. It specifically assesses the model’s performance in distinguishing between malignant and benign tissues in prostate cancer (PCa) and renal cell carcinoma (RCC). The model showed promising results in differentiating RCC from normal kidney tissues, achieving an AUC of 0.871. However, it struggled to effectively differentiate between benign and malignant prostate tissues, achieving an AUC of only 0.51. The study highlights the potential of next-generation multimodal AI models in enhancing clinical diagnosis and improving doctor-patient communication by providing explanations in plain language and introducing treatment principles based on image analysis.
Citation: Zhu, Lingxuan, et al. “Multimodal Approach in the Diagnosis of Urologic Malignancies: Critical Assessment of ChatGPT-4V’s Image-Reading Capabilities.” JCO Clinical Cancer Informatics 8 (2024): e2300275.
Publication No. 17
Publication Data: ACCEPTED
Title: International Journal of Surgery：Advancing Generative AI in Medicine: Recommendations for Standardized Evaluation
Link:
Full Author: Anqi Lin, Lingxuan Zhu, Weiming Mou, Zizhi Yuan, Quan Cheng, Aimin Jiang, Peng Luo
Summary: In this paper, we examine current challenges in evaluating generative AI systems for medical applications. We find generative AI shows promise but instability issues make manual evaluation critical yet lacking in standardization, risking biased assessments. To enable more objective evaluations of generative AI performance in medicine, we first recommend establishing standardized multi-criteria scoring systems to reduce subjectivity. Second, implementing training, multi-reviewer scoring, and audits can minimize individual biases. Finally, statistical analysis of scoring differences and iterative refinement of protocols can continually improve consistency. Overall, our proposed recommendations for rigorous generative AI evaluation aim to advance safe, effective integration of this powerful technology in clinical medicine. Standardized frameworks will let us empirically validate capabilities as generative AI matures. We look forward to leveraging more objective assessments to unlock the vast potential of generative AI in transforming healthcare.
Citation: Anqi Lin, et al. “International Journal of Surgery：Advancing Generative AI in Medicine: Recommendations for Standardized Evaluation.” International Journal of Surgery (2024).
Publication No. 18
Publication Data: ACCEPTED
Title: ChatGPT’s Ability to Generate Realistic Experimental Images Poses a New Challenge to Academic Integrity
Link:
Full Author: Lingxuan Zhu, Yancheng Lai, Weiming Mou, Haoran Zhang, Anqi Lin, Chang Qi, Tao Yang, Liling Xu, Jian Zhang, Peng Luo
Summary: The rapid advancements in large language models (LLMs) such as ChatGPT have raised concerns about their potential impact on academic integrity. While initial concerns focused on ChatGPT’s writing capabilities, recent updates have integrated DALL-E 3’s image generation features, extending the risks to visual evidence in biomedical research. Our tests revealed ChatGPT’s nearly barrier-free image generation feature can be used to generate experimental result images, such as blood smears, Western Blot, immunofluorescence and so on. Although the current ability of ChatGPT to generate experimental images is limited, the risk of misuse is evident. This development underscores the need for immediate action, suggesting AI providers restrict the generation of experimental image generation, develop tools to detect AI-generated images, and consider adding “invisible watermarks” to the generated images. By implementing these measures, we can better ensure the responsible use of AI technology in academic research.
Citation: Lingxuan Zhu, et al. “ChatGPT’s Ability to Generate Realistic Experimental Images Poses a New Challenge to Academic Integrity.” Journal of Hematology & Oncology (2024).
Publication No. 19
Publication Data: ACCEPTED
Title: Multimodal ChatGPT-4V for ECG Interpretation: Promise and Limitations
Link:
Full Author: Lingxuan Zhu, Weiming Mou, Keren Wu, Yancheng Lai, Anqi Lin, Tao Yang, Jian Zhang, Peng Luo
Summary: Electrocardiogram (ECG) interpretation is an essential skill in cardiovascular medicine. This study evaluated the capabilities of newly released ChatGPT-4V, a large language model with visual recognition abilities, in interpreting ECG waveforms and answering related multiple-choice questions. A total of 62 ECG-related multiple-choice questions were collected from reputable medical exams. ChatGPT was prompted to answer the questions by analyzing the accompanying ECG images. Requiring at least 1 of 3 responses to be correct, ChatGPT achieved an overall accuracy of 83.87% across all question types. ChatGPT demonstrated significantly lower performance on counting-based questions like calculating QT intervals compared to diagnostic and treatment recommendation questions. The findings indicate that while ChatGPT shows promising potential in ECG interpretation and decision-making, its diagnostic reliability and quantitative analysis abilities need improvement before real clinical use. Further large-scale studies are warranted to fully evaluate ChatGPT’s capabilities and track its progress as the model accumulates more medical knowledge through ongoing training. With technological advancements, multimodal AI like ChatGPT may one day play an important role in assisting clinicians with ECG interpretation and cardiovascular care.
Citation: Lingxuan Zhu, et al. “Multimodal ChatGPT-4V for ECG Interpretation: Promise and Limitations.” Multimodal ChatGPT-4V for ECG Interpretation: Promise and Limitations (2024).