Feeding the troll detection algorithm

Informal flags used as labels in classification models to identify perceived computational propaganda

  • Vlad Achimescu University of Mannheim
  • Dan Sultanescu National University of Political Studies and Public Administration, Bucharest
Keywords: computational propaganda, non-genuine users, trolls, machine learning, computational text analysis

Abstract

The authenticity of public debate is challenged by the emergence of networks of non-genuine users (such as political bots and trolls) employed and maintained by governments to influence public opinion. To tackle this issue, researchers have developed algorithms to automatically detect non-genuine users, but it is not clear how to identify relevant content, what features to use and how often to retrain classifiers. Users of online discussion boards who informally flag other users by calling them out as paid trolls provide potential labels of perceived propaganda in real time. Against this background, we test the performance of supervised machine learning models (regularized regression and random forests) to predict discussion board comments perceived as propaganda by users of a major Romanian online newspaper. Results show that precision and recall are relatively high and stable, and re-training the model on new labels does not improve prediction diagnostics. Overall, metadata (particularly a low comment rating) are more predictive of perceived propaganda than textual features. The method can be extended to monitor suspicious activity in other online environments, but the results should not be interpreted as detecting actual propaganda.

Author Biographies

Vlad Achimescu, University of Mannheim

PhD candidate at the Professorship for Statistics and Methodology at the University of Mannheim.

Dan Sultanescu, National University of Political Studies and Public Administration, Bucharest

Research director of the Center for Civic Participation and Democracy at the National University of Political Studies and Public Administration, Bucharest.

Published
2020-08-19
How to Cite
Achimescu, V., & Sultanescu, D. (2020). Feeding the troll detection algorithm. First Monday, 25(9). https://doi.org/10.5210/fm.v25i9.10604