RATING UNIVERSITIES IN HONG KONG, MACAO AND SINGAPORE BY GENERATIVE ARTIFICIAL INTELLIGENCE (AI): ARE VARIOUS AI MODELS CONSISTENT?

This article seeks to study the consistency, among other ancillary comparisons, between popular generative artificial intelligence (AI) models in rating various dimensions of 26 universities in Hong Kong, Macao, and Singapore. Having experimented with six models, only PaLM and Llama ended up being amenable to analysis where the duo were individually requested to award rating scores to the five dimensions (1) Teaching, (2) Research, (3) Citations, (4) International Outlook, and (5) Industry Income of the universities. For each of the two models, the minimum, the maximum, the range, and the standard deviation of the rating scores for each of the five dimensions were computed across all the universities. The rating score difference for each of the five dimensions between the two models was calculated for each university. The mean of the absolute values, the minimum, the maximum, the range, and the standard deviation of the differences for each dimension between the two models were calculated across all universities. A paired sample t-test was then applied to each dimension for the rating score differences between the two models over all the universities. Finally, a correlation coefficient of the rating scores was computed for each dimension between the two models across all the universities. Among other collateral findings, the two models’ ratings were found almost flawlessly consistent for the five dimensions with the correlation coefficients ranging from .851 to .908 (p all at 0.000). Consistency implies at least the likely trustworthiness of both PaLM’s and Llama’s ratings.