TWON researcher Simon Münker from Universität Trier has published a new paper titled “Fingerprinting LLMs through Survey Item Factor Correlation: A Case Study on Humor Style Questionnaire.” The paper appears in the Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) and was presented at the conference in Suzhou, China.
The paper addresses the growing use of large language models in relation to psychological instruments and examines how LLMs internally represent psychological constructs. Simon Münker introduces a new approach to “fingerprinting” LLMs by analyzing their factor correlation patterns on standardized psychological assessments.
Using the Humor Style Questionnaire as a case study, the study compares how six LLMs represent and correlate humor-related constructs with patterns found among human survey participants. The findings show little similarity between LLM response patterns and human responses, while human participant subsamples demonstrate high internal consistency. Exploratory graph analysis further indicates that none of the analyzed LLMs successfully recover the four constructs of the Humor Style Questionnaire.
The results suggest that, despite advances in natural language capabilities, current LLMs represent psychological constructs in fundamentally different ways than humans. This raises important questions about the validity of using LLMs as human simulacra in psychological and social research.
