Skip to main content

Table 2 Cohen’s kappa estimates with lower and upper confidence intervals for external consistency (compared to reference A) and internal consistency (1st vs. 2nd analysis) for all 79 students previously affected by CL (35 females and 44 males) analysed in phase 1B

From: Advancing AI-driven thematic analysis in qualitative research: a comparative study of nine generative models on Cutaneous Leishmaniasis data

Model

Kappa_All_1st Vs Ref_A

Kappa_All_2nd Vs Ref_A

Internal_Consistency All 1st Vs 2nd

Kappa_Female_1st Vs Ref_A

Kappa_Female_2nd Vs Ref_A

Internal_Consistency Female 1st Vs 2nd

Kappa_Male_1st Vs Ref_A

Kappa_Male_2nd Vs Ref_A

Internal_Consistency Male 1st Vs 2nd

Man

0.59 (0.42–0.77)

0.77 (0.63–0.92)

0.82 (0.72–0.93)

0.47 (0.15–0.79)

0.76 (0.44-1.00)

0.57 (0.25–0.90)

0.63 (0.44–0.83)

0.78 (0.61–0.94)

0.88 (0.80–0.96)

Claude 3.5 Sonnet

0.66 (0.51–0.81)

0.71 (0.54–0.87)

0.98 (0.94-1.00)

0.80 (0.52-1.00)

0.80 (0.52-1.00)

1.00 (1.00–1.00)

0.64 (0.47–0.81)

0.70 (0.51–0.89)

0.97 (0.92-1.00)

NoteboookLM

0.76 (0.64–0.88)

0.82 (0.71–0.93)

0.91 (0.81-1.00)

0.64 (0.38–0.90)

0.78 (0.56-1.00)

0.73 (0.41-1.00)

0.80 (0.67–0.93)

0.83 (0.71–0.95)

0.97 (0.91-1.00)

Gemini1.5 Advanced Ultra

0.77 (0.63–0.90)

0.82 (0.71–0.93)

0.97 (0.92-1.00)

0.78 (0.58–0.99)

0.88 (0.70-1.00)

0.90 (0.73-1.00)

0.76 (0.59–0.93)

0.80 (0.67–0.93)

0.99 (0.98-1.00)

LlaMA 405B

0.82 (0.68–0.97)

0.83 (0.68–0.97)

0.97 (0.92-1.00)

0.82 (0.51-1.00)

0.82 (0.51-1.00)

1.00 (1.00–1.00)

0.82 (0.66–0.98)

0.83 (0.67–0.99)

0.95 (0.88-1.00)

ChatGPT o1

0.78 (0.64–0.92)

0.70 (0.58–0.83)

0.79 (0.67–0.92)

0.80 (0.52-1.00)

0.64 (0.38–0.90)

0.85 (0.63-1.00)

0.77 (0.62–0.93)

0.73 (0.59–0.86)

0.78 (0.62–0.93)

ChatGPT o1_PRO

0.81 (0.69–0.94)

0.81 (0.69–0.94)

1.00 (1.00–1.00)

0.80 (0.52-1.00)

0.80 (0.52-1.00)

1.00 (1.00–1.00)

0.82 (0.68–0.96)

0.82 (0.68–0.96)

1.00 (1.00–1.00)

GrokV2

0.76 (0.64–0.87)

0.79 (0.66–0.91)

0.90 (0.80–0.99)

0.77 (0.56–0.98)

0.80 (0.52-1.00)

0.74 (0.50–0.99)

0.75 (0.61–0.89)

0.80 (0.67–0.94)

0.94 (0.86-1.00)

DeepSeekV3

0.78 (0.66–0.90)

0.75 (0.61–0.90)

0.92 (0.81-1.00)

0.64 (0.38–0.90)

0.80 (0.52-1.00)

0.85 (0.63-1.00)

0.83 (0.71–0.95)

0.76 (0.59–0.93)

0.93 (0.80-1.00)

Gemini2.0 Advanced

0.69 (0.54–0.84)

0.73 (0.62–0.85)

0.80 (0.65–0.94)

0.96 (0.90-1.00)

0.80 (0.52-1.00)

0.85 (0.85-1.00)

0.63 (0.45–0.82)

0.74 (0.60–0.87)

0.80 (0.64–0.95)