JDSE
Journal of Dental Sciences and Education

The Journal of Dental Sciences and Education deals with General Dentistry, Pediatric Dentistry, Restorative Dentistry, Orthodontics, Oral diagnosis and DentomaxilloFacial Radiology, Endodontics, Prosthetic Dentistry, Periodontology, Oral and Maxillofacial Surgery, Oral Implantology, Dental Education and other dentistry fields and accepts articles on these topics. Journal of Dental Science and Education publishes original research articles, review articles, case reports, editorial commentaries, letters to the editor, educational articles, and conference/meeting announcements.

eISSN: 2980-1389
Creative Commons CC BY-NC-ND 4.0 International License.
EndNote Style

Online Manuscript System

Index
Original Article
Evaluation of AI-based chatbot responses to orthodontic retention-related patient questions: a comparative analysis
Abstract
Aims: This study aimed to evaluate and compare the quality, accuracy, and readability of orthodontic retention-related information provided by five AI-based chatbot platforms: ChatGPT-4, Copilot, Gemini, MediSearch, and OpenEvidence.
Methods: A set of 25 commonly asked patient-oriented questions on orthodontic retention was submitted to each chatbot. Responses were evaluated using five key metrics: EQIP (Ensuring Quality Information for Patients), Reliability Score, Global Quality Score (GQS), SMOG Readability Index, and Similarity Index. Mean±standard deviation values were calculated. Kruskal-Wallis and Dunn’s post-hoc tests were used to assess statistical differences.
Results: MediSearch and OpenEvidence outperformed others in EQIP and Reliability scores. ChatGPT-4 generated the most original content with the lowest Similarity Index. SMOG readability was significantly better for ChatGPT-4, while MediSearch and OpenEvidence produced more technically complex language. Statistically significant differences (p<0.05) were found between platforms in EQIP, SMOG, and Similarity Index metrics.
Conclusion: Performance among AI chatbots varies significantly in delivering orthodontic retention guidance. While medical-specific tools offer superior accuracy and reliability, general-purpose models like ChatGPT-4 excel in readability and originality. The results highlight the importance of matching AI platform selection with patient communication goals.

Keywords : Artificial intelligence, chatbots, orthodontic retention, health information quality, readability, patient education

Download Article


1. Alqerban A, Hedesiu M, Schroeder TM. Patient compliance during orthodontic retention: a systematic review. Angle Orthod. 2021;91(5):617-626.
2. Littlewood SJ, Millett DT, Doubleday B, Bearn DR, Worthington HV. Retention procedures for stabilising tooth position after treatment with orthodontic braces. Cochrane Database Syst Rev. 2016;(1):CD002283. doi:10.1002/14651858.CD002283.pub2
3. Bickmore TW, Giorgino T. Health dialog systems for patients and consumers. J Biomed Inform. 2006;39(5):556-571. doi:10.1016/j.jbi.2005. 12.004
4. Lewis D, Gillies D. Retrieval-augmented generation: a survey of methods and applications. J Artif Intell Res. 2023;76:1221-1256.
5. Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. JAMA Netw Open. 2022;5(9):e2237120. doi:10.1101/2022.12.23.521610
6. Chen E, Lin C. Improving medical question-answering systems with retrieval-augmented generation and structured evidence. Nat Digit Med. 2023;6:78-84.
7. Walker M, Hancock JT. Evaluating factual accuracy in large language models for clinical use. NPJ Digit Med. 2022;5:142.
8. Pandey SK, Sharma V. Domain-specific fine-tuning of large language models for medical applications. Comput Biol Med. 2023;158:106789.
9. OpenAI. GPT-4 technical report. arXiv. 2023;2(5):1. doi:10.48550/arXiv. 2303.08774
10. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2(2):e0000198. doi:10.1371/journal.pdig.0000198
11. Ji Z, Lee N, Frieske R, et al. Survey of hallucination in natural language generation. ACM Comput Surv. 2023;55(12):1-38. doi:10.1145/3571730
12. McInnes N, Haglund BJ. Readability of online health information: implications for health literacy. Inform Health Soc Care. 2011;36(4):173-189. doi:10.3109/17538157.2010.542529
13. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America&rsquo;s Adults: Results from the 2003 National Assessment of Adult Literacy. Washington, DC: National Center for Education Statistics; 2006.
14. Pithon MM, Oliveira VC, Ruellas ACO. Quality of information on the World Wide Web about pain after orthognathic surgery. Int J Oral Maxillofac Surg. 2013;42(5):577-582.
15. Leong P, Moy B. A critical review of internet-based patient information on orthodontic retention. Br Dent J. 2017;222(8):609-612.
16. Denecke K, Bamidis P, Bond C, et al. Ethical issues of social media usage in healthcare. Yearb Med Inform. 2015;10(1):137-147. doi:10.15265/IY-2015-001
17. Choudhury MD, Morris MR, White RW. Seeking and sharing health information online: a study of social media use. ACM Trans Comput Hum Interact. 2014;21(1):1-27.
18. Singhal S, Fernandez-Luque L, Patil S. Pharmacists as social media influencers in healthcare. J Med Internet Res. 2013;15(7):e140.
19. Bavbek AB, Tuncer BB. Quality assessment of websites providing information on orthognathic surgery using the DISCERN tool. J Stomatol Oral Maxillofac Surg. 2023;124(4):101376.
20. Alshahrani I, Alqahtani F. Evaluation of online health information about periodontal disease. BMC Oral Health. 2022;22:647.
21. Yurdakurban E, Demirbas F. Evaluation of AI-based chatbots for temporomandibular disorders. J Oral Rehabil. 2024;51(3):234-242.
22. Wong RWK, Freer TJ. A comprehensive review of retainer design and effectiveness. Am J Orthod Dentofacial Orthop. 2005;127(6):651-658.
23. Costa C, Santos C. Evaluating multilingual AI language models in healthcare: opportunities and risks. Front Artif Intell. 2023;6:1180120.
24. Mavragani A. Tracking and predicting COVID-19 outbreaks using Google Trends data. J Med Internet Res. 2020;22(5):e18992.
25. Denecke K. Using natural language processing to improve patient-provider communication: tools and strategies. Stud Health Technol Inform. 2015;216:349-353.
Article available in :
Volume 3, Issue 3, 2025
Page : 74-79
https://doi.org/10.51271/JDSE-0059
Article Information
Received : 07 Ağu 2025
Accepted : 30 Ağu 2025
Published : 31 Ağu 2025
Corresponding Author :

Melis Büşra Aşkın: Department of Orthodontics, Faculty of Dentistry, Bozok University, Yozgat, Turkiye

[email protected]
Citation : Aşkın MB, Manav ÖC. Evaluation of AI-based chatbot responses to orthodontic retention-related patient questions: a comparative analysis. J Dent Sci Educ. 2025;3(3):74-79.

Read: 33
Download : 424
Cited: 0
_Footer