Our researcher Sjoerd Stolwijk recently published a deliverable, proposing a set of metrics to determine the deliberative quality of discussions on social media in general, and TWON in particular.
The report lists the key indicators of: (1) Exposure to political content, (2) Engagement with political content, (3) Contributing political content, (4) Diversity of exposure and (5) Quality of exposure.
It is then explained how and why this set of indicators differs from the typical list of deliberative indicators and proposes to view deliberation from a summative rather than an additive perspective. In this view, social media do not need to aim at perfect deliberation within one platform; rather, the goal is to contribute to deliberation at a societal scale via the platform.
We propose that social media can contribute especially by offering an avenue for users (citizens, journalists and politicians alike) to be exposed to political debate, but also to engage and participate in that debate. In addition, social media can connect otherwise unconnected users and expose them to ideas they might otherwise have missed. Ideally, these ideas are substantiated with arguments and evidence.
Our researcher evaluates a large set of automatic classifiers to determine the degree to which social media comments meet several deliberative criteria, specifically whether comments are rational, interactive, diverse and civil.
Results show how more modern techniques like fine-tuned transformers and generative large language models have improved our ability to reproduce manual codings automatically, but also that results vary considerably between models.
We then integrate the aims of Chapter 3 with the results of Chapter 4 and translate them to the case of TWON to arrive at the metrics proposed in Chapter 2. It adds tests of the performance of different classifiers to determine whether a comment is political or not.
Finally, we take a look into the future, beyond what is currently feasible for TWON, to explore whether new techniques can help determine the deliberative quality of online social media debates to the more fine-grained level of specific claims and show some promising first results.