Model | Kappa_All_1st Vs Ref_A | Kappa_All_2nd Vs Ref_A | Internal_Consistency All 1st Vs 2nd | Kappa_Female_1st Vs Ref_A | Kappa_Female_2nd Vs Ref_A | Internal_Consistency Female 1st Vs 2nd | Kappa_Male_1st Vs Ref_A | Kappa_Male_2nd Vs Ref_A | Internal_Consistency Male 1st Vs 2nd |
---|---|---|---|---|---|---|---|---|---|
Man | 0.59 (0.42–0.77) | 0.77 (0.63–0.92) | 0.82 (0.72–0.93) | 0.47 (0.15–0.79) | 0.76 (0.44-1.00) | 0.57 (0.25–0.90) | 0.63 (0.44–0.83) | 0.78 (0.61–0.94) | 0.88 (0.80–0.96) |
Claude 3.5 Sonnet | 0.66 (0.51–0.81) | 0.71 (0.54–0.87) | 0.98 (0.94-1.00) | 0.80 (0.52-1.00) | 0.80 (0.52-1.00) | 1.00 (1.00–1.00) | 0.64 (0.47–0.81) | 0.70 (0.51–0.89) | 0.97 (0.92-1.00) |
NoteboookLM | 0.76 (0.64–0.88) | 0.82 (0.71–0.93) | 0.91 (0.81-1.00) | 0.64 (0.38–0.90) | 0.78 (0.56-1.00) | 0.73 (0.41-1.00) | 0.80 (0.67–0.93) | 0.83 (0.71–0.95) | 0.97 (0.91-1.00) |
Gemini1.5 Advanced Ultra | 0.77 (0.63–0.90) | 0.82 (0.71–0.93) | 0.97 (0.92-1.00) | 0.78 (0.58–0.99) | 0.88 (0.70-1.00) | 0.90 (0.73-1.00) | 0.76 (0.59–0.93) | 0.80 (0.67–0.93) | 0.99 (0.98-1.00) |
LlaMA 405B | 0.82 (0.68–0.97) | 0.83 (0.68–0.97) | 0.97 (0.92-1.00) | 0.82 (0.51-1.00) | 0.82 (0.51-1.00) | 1.00 (1.00–1.00) | 0.82 (0.66–0.98) | 0.83 (0.67–0.99) | 0.95 (0.88-1.00) |
ChatGPT o1 | 0.78 (0.64–0.92) | 0.70 (0.58–0.83) | 0.79 (0.67–0.92) | 0.80 (0.52-1.00) | 0.64 (0.38–0.90) | 0.85 (0.63-1.00) | 0.77 (0.62–0.93) | 0.73 (0.59–0.86) | 0.78 (0.62–0.93) |
ChatGPT o1_PRO | 0.81 (0.69–0.94) | 0.81 (0.69–0.94) | 1.00 (1.00–1.00) | 0.80 (0.52-1.00) | 0.80 (0.52-1.00) | 1.00 (1.00–1.00) | 0.82 (0.68–0.96) | 0.82 (0.68–0.96) | 1.00 (1.00–1.00) |
GrokV2 | 0.76 (0.64–0.87) | 0.79 (0.66–0.91) | 0.90 (0.80–0.99) | 0.77 (0.56–0.98) | 0.80 (0.52-1.00) | 0.74 (0.50–0.99) | 0.75 (0.61–0.89) | 0.80 (0.67–0.94) | 0.94 (0.86-1.00) |
DeepSeekV3 | 0.78 (0.66–0.90) | 0.75 (0.61–0.90) | 0.92 (0.81-1.00) | 0.64 (0.38–0.90) | 0.80 (0.52-1.00) | 0.85 (0.63-1.00) | 0.83 (0.71–0.95) | 0.76 (0.59–0.93) | 0.93 (0.80-1.00) |
Gemini2.0 Advanced | 0.69 (0.54–0.84) | 0.73 (0.62–0.85) | 0.80 (0.65–0.94) | 0.96 (0.90-1.00) | 0.80 (0.52-1.00) | 0.85 (0.85-1.00) | 0.63 (0.45–0.82) | 0.74 (0.60–0.87) | 0.80 (0.64–0.95) |