Ivan Srba. (source: ESET Science Award/Linda Kisková Bohušová)
In recent years, disinformation has become a huge problem worldwide, but countering it is difficult.
This article is supported by the ESET Foundation, whose annual ESET Science Award recognises exceptional scientists.
IVAN SRBA and his colleagues from the Kempelen Institute of Intelligent Technologies (KInIT) are among those leading efforts to increase the quality of online content and prevent hoaxes and untruths being spread. They have created the largest database of checked facts in the world and an app for media professionals to verify information, regardless of what part of the world it originated in.
You help media professionals evaluate the credibility of content. What does that mean?
We aim to research and develop solutions based on artificial intelligence (AI) that can help media professionals to more effectively create and spread credible information. At the same time, we help them with identifying non-credible information, like disinformation or hoaxes. In these solutions, we mainly deal with multilingualism. We will never have as much data in Slovak as there is for example in English. Therefore, we take a language model trained in English and other languages and test whether it can be used in Slovak as well. One of the such solutions is the detection of generated text, that is, whether an AI was used to write a given text found online, a newspaper article or a social media post.
How do you know a text was generated by a language model? Does it leave traces behind?
A language model is based on statistical probabilities. Just as every person has a vocabulary, a way of choosing words, a sentence style coded in their head, a language model encodes its authorship similarly. Such style can be consequently detected. In a machine-generated image or video, you can spot, for example, that a person blinks the wrong way. With text, it is more difficult, because it is a just a sequence of letters that capture much less information. However, at the same time, there are also approaches that can hide a machine's style and authorship. A very simple example is to use similar looking letters from one alphabet in another. Another method is back translation, that is, I take a text in Slovak, translate it into English and then back again, which can hide to an extent the authorship of a language model.
To stay up to date with what scientists in Slovakia or Slovak scientists around the world are doing, subscribe to the Slovak Science newsletter, which will be sent to readers free of charge four times a year.
How do media professionals apply this?
As part of the CEDMO (Central European Digital Media Observatory) project, involving Slovakia, the Czech Republic and Poland, we have collected what is currently the largest database of fact-checks in the world. We have approximately 430,000 claims that have already been checked in more than 50 languages from around the world. We've created an app that allows fact checkers to take a social media post regardless of language and evaluate whether the information contained in it has already been verified somewhere, even if it is in a different language. An article about a global issue may be in Slovak but with this tool they can find out if the facts around this issue have been verified in any language.
Where is it used?
At first, it was tested by fact-checkers in Slovakia, the Czech Republic and Poland. They really appreciated it because before they had to manually verify everything. It is now integrated as a part of the toolkit for online content verification and will be available to more than 100,000 users worldwide. Since it is multilingual, it can also be used by someone in any other country.
You are currently also working on a tool that will perform content screening and credibility analysis and can also detect logical fallacies. How does it work?
These help media professionals pre-screen content without having to read the text thoroughly and evaluate whether it is credible or has signs of unprofessionalism. We call them credibility signals or credibility indicators. There are many signals, such as a clickbait title, emotionally charged text, etc. However, it must be said that the performance of such a model is not 100% and never will be - there will always be false positives or false negatives.
Is it like AI "hallucinating"?
A little different. When I talk about false positives and negatives, it involves an AI programme trying to classify things into categories. Hallucinating means that instead of putting things into boxes, an AI programme generates text by itself and makes things up. Quite often an AI programme is unable to detect fallacies and other persuasion techniques – it’s difficult to do so. Let's be honest, people cannot do it either. Show 10 people a sentence and ask them where a persuasion technique was used, and you won't get a definitive answer. We tried to train multilingual language models so that they could point out appeals to authority and other techniques. It works relatively well, but it is still far from perfect and it will be a challenge to develop the models further.
Do you think AI will one day be able to understand the various hidden layers in language?
Two years ago, we could not have imagined the kind of things big language models are able to do now. As we see large language models growing, their ability to understand text will also grow and they will be able to react even more creatively. I am convinced that in the future they will be able to do so at least to the same level as an ordinary person.
Ivan Srba. (source: ESET Science Award/Linda Kisková Bohušová)
Could artificial intelligence one day verify claims in real time? Is such a thing even possible?
If we're talking about watching a debate with people making claims that are immediately verified, then that would be the Holy Grail. At the moment, this is not possible at a general level, although it can be done for very specific areas and topics. On the other hand, there is the manual verification that journalists do on a daily basis using trusted sources. There is also a middle ground, which we are focused on - we help media professionals by, when a claim is made, pointing out whether it has already been verified by fact-checkers.
Even if there was such technology, would people believe it?
While this is really a question for social scientists, we have looked at behavioural studies and, unfortunately, it is indeed the case that once a person is convinced that something is or is not true, it is very difficult to correct that belief. Studies have analysed, for example, people's reaction to warnings over the veracity of a social media post. In the case such a warning has been generated by AI, the chances of a person then changing their mind over whether to share it or not was lower than when it was evaluated by a person. AI seems less trustworthy than a person. At the same time, it must be said that in most cases even a warning does not stop a person sharing a post.
You also research the use of chatbots such as ChatGPT in spreading disinformation. What have you found?
The danger that they can be misused to generate hate, personal attacks, offensive language, among others, is often talked about, but their creators are trying to stop this. As part of the European research project vera.ai, we looked at the real potential to do so. We studied several large language models, not just ChatGPT, but open-source ones too. We used well-known disinformation narratives that had been previously checked - about medicine, flat Earth, the moon landing, but also regional issues like voter fraud in Nigeria, and asked the models to generate a newspaper article. We also tested what would happen if they were provided with a short abstract explaining context. We found that most models generated a disinformation article willingly, although their creators tried to incorporate the so-called safety filters, such as a notification that a topic was, for instance, a medical issue and that the reader should contact their doctor instead.
But people know how to bypass these.
Of course, but we didn't have to try to bypass them. We were surprised that open-source models were more reluctant to generate such a text. Another conclusion was that even if a given model did not know the given topic, it was able to very quickly - especially if given an abstract - grasp it and generate text. It even came up with its own arguments to support the narrative. For example, when a model lacked medical information, it was able to support the false argument by a non-existing medical study or a non-existing professor from an American university. So, I come back to when someone does this, can we detect it as a machine-generated text? The answer is that success rates vary.
Are there known examples of models being used to spread disinformation?
That's hard to say. Just because the potential is there, it doesn't mean it's actively happening. Also, unlike images and videos, where manipulation can be proven, it is more difficult when it comes to text. Over time it will be even more difficult, because a model's skill will evolve, allowing it to hide its style.
It feels like we are getting the short end of the stick when it comes to combating disinformation. Is there any way to use AI to predict what kind of disinformation will start to spread next?
I don't think AI could successfully predict that in three days certain disinformation would start to be spread. I believe people are more likely to be able to do so. For example, before an election, we can be proactive and warn people that a certain narrative may be spreading. It's called pre-bunking, and provides an opportunity to not just fight against the spread of unreliable content, but to also show how to identify it. Pre-bunking can strengthen media literacy. If a person is not media literate, they will ignore any warning and spread a post. They might think that the AI programme is trained to censor content - the exact opposite of what we want to achieve. People need to be taught that when a text refers to a university professor, they should look them up to see if they actually exist. I am not sure whether this is already taught as part of the school curriculum.
Slovakia is very bad in terms of the spread and belief in disinformation. Teaching media literacy and educating people about disinformation is a start to combating this. What else should also be looked at?
There is a need for a deeper exploration of how social media encloses us into bubbles in which we are surrounded by content we want to see. To some extent, this is natural due to a huge amount of information. The problem is that when a bubble is formed not on the basis of topics we are interested in, such as hobbies, but on beliefs, for example into some well-known disinformation narratives. So far, there has not been much research into auditing social media recommender systems that put us into such undesired bubbles.
Is there any good news?
Yes. Things are starting to move forward. The European Commission has approved the Digital Service Act (aims to create safer digital space - Ed. note), and the Artificial Intelligence Act (regulates AI systems assessed as potentially dangerous - Ed. note).
Will regulation be enough for big companies to change their policies? Are fines sufficient?
What's important is what happens before fines are issued. For example, if a social media company wants access to the European market - a relatively huge number of people - it will be ordered to remove harmful content. If it doesn't comply, it will not be allowed in, or be expelled. The EC has also introduced principles of self-regulation, for example the Code of Practice on Disinformation, which was recently strengthened and prescribes how harmful content should be fought. The aim is to make companies aware of their responsibility. Of course, I'm not saying it's perfect but it has an effect, otherwise these things would not have been talked about.
Ivan Srba
Senior researcher at KInIT, specialising in artificial intelligence, machine learning and natural language processing, with a focus on low resource AI, learning with limited labelled data, multilinguality and social computing systems, especially social media. His current primary research involves supporting media professionals in producing credible content and countering online disinformation. He is a researcher and an executive board member in three EU-funded projects - CEDMO, vera.ai and AI-CODE.
This article is supported by the ESET Foundation, whose annual ESET Science Award recognises exceptional scientists.
