Is AI clairvoyant? ChatGPT can make personality tests and predict responses, Israeli study finds

A recent Israeli study conducted by Hebrew University of Jerusalem researchers has discovered that the popular AI language model, ChatGPT, can generate personality assessment questionnaires from any source text in English. The team, consisting of Dr. Rotem Monsa, Prof. Aviv Zohar, and Prof. Shahar Arzy, tested this method using the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) and an astrology textbook.

The researchers found that ChatGPT was able to create and validate questionnaires based on the provided texts. The questionnaires based on the DSM-5 showed high internal consistency within personality clusters, such as dependency and avoidance, which correlated with real-world responses. These results were comparable to those of the Big Five personality questionnaire (BFI), further validating the strength of the questionnaire in measuring real-life patterns of human psychology.

In contrast, the questionnaire generated from the astrology textbook showed weak internal consistency across traits. This discrepancy suggests that the astrological elements do not reflect coherent psychological dimensions.

The most surprising outcome was that ChatGPT could predict population-level responses before the surveys were carried out. The model accurately predicted mean responses and correlations between questions, further suggesting that LLMs (large language models) have an innate understanding of personality dynamics at a population level.

According to lead author Dr. Monsa, the LLMs don't "understand" the language or possess human-like understanding. Instead, they learn statistical patterns from the vast amount of text data they are trained on. The researchers acknowledged that while ChatGPT doesn't inherently understand psychology, it has learned a significant amount of it through its extensive training.