E-POSTER DETAIL | 11th Asia Pacific Tele Ophthalmology Society Symposium

Title
Large Language Models in Initial Symptom Recognition and Diagnostic for Thyroid Eye Disease

Authors
Shiqi Hui, Dongmei Li

Presenting
Shiqi Hui

PURPOSE:
To preliminarily evaluate agreement between a large language model (LLM) and ophthalmic clinicians in recognizing initial symptoms and making diagnostic decisions for thyroid eye disease (TED).

METHODS:
A structured pilot online survey was conducted among 17 experienced oculoplastic and orbital clinicians from eight countries. The questionnaire assessed physicians’ approaches to demographics, initial symptom recognition, differential diagnostic reasoning, and early diagnostic decisions in TED. The same questionnaire items were posed to two leading large language models (LLMs) (GPT-5 and Gemini 2.5 Pro). All answers were binarized (1 = selected, 0 = unselected). The primary outcome was the mean Jaccard similarity between LLM and human responses, and cosine similarity served as a secondary metric.

METHODS:

A structured pilot online survey was conducted among 17 experienced oculoplastic and orbital clinicians from eight countries. The questionnaire assessed physicians’ approaches to demographics, initial symptom recognition, differential diagnostic reasoning, and early diagnostic decisions in TED. The same questionnaire items were posed to two leading large language models (LLMs) (GPT-5 and Gemini 2.5 Pro). All answers were binarized (1 = selected, 0 = unselected). The primary outcome was the mean Jaccard similarity between LLM and human responses, and cosine similarity served as a secondary metric.

RESULTS:
The LLMs demonstrated moderate overall agreement with physicians. Sensitivity analyses showed broadly consistent categorical patterns, although confidence intervals were wide. No significant similarity difference was observed among physician’s experience levels. GPT showed statistically significant slightly higher alignment with physician responses Gemini across demographic and diagnostic domains (p < 0.05). A strong positive correlation between Jaccard and Cosine similarities for both LLMs (p <0.001).

RESULTS:

The LLMs demonstrated moderate overall agreement with physicians. Sensitivity analyses showed broadly consistent categorical patterns, although confidence intervals were wide. No significant similarity difference was observed among physician’s experience levels. GPT showed statistically significant slightly higher alignment with physician responses Gemini across demographic and diagnostic domains (p < 0.05). A strong positive correlation between Jaccard and Cosine similarities for both LLMs (p <0.001).

CONCLUSIONS:
In this pilot sample, LLMs exhibit promising alignment with expert opinions in TED management for demographic and diagnostic decisions. Nonetheless, domain-specific fine-tuning and expert oversight before clinical integration are needed. These findings provide preliminary, hypothesis-generating evidence for the potential role of LLMs as decision-support tools in early TED diagnosis. Larger, representative studies are warranted to confirm these results.

CONCLUSIONS:

In this pilot sample, LLMs exhibit promising alignment with expert opinions in TED management for demographic and diagnostic decisions. Nonetheless, domain-specific fine-tuning and expert oversight before clinical integration are needed. These findings provide preliminary, hypothesis-generating evidence for the potential role of LLMs as decision-support tools in early TED diagnosis. Larger, representative studies are warranted to confirm these results.