Michal Valko. (source: Christian Morel Photographer)
Until a few months ago, Slovak machine-learning scientist Michal Valko was training AI for Google’s DeepMind. However, after being personally contacted by Mark Zuckerburg, in March this year he moved to Meta where he trained an AI model which was recently shared for free.
This article is supported by the ESET Foundation, whose annual ESET Science Award recognises exceptional scientists.
An envoy of the Eset Science Award, Valko spoke to The Slovak Spectator ahead of the announcements of this year’s winners, describing his work for Meta, and revealing the endgame of developing and training AI.
We last talked a year ago when the Eset Science Award winners were announced. Has anything changed in the research of language models since then?
Since 2017, when we figured out how to scale transformers - a simple neural network architecture - almost nothing major has changed when it comes to the main structure of language models. There have been some small changes though, such as in word encoding, which has increased performance by three or four percent. Also, more data, parameters, flops go into all of it, so the results are better.
In other words, are AI's replies even better than before, closer to the way a human would speak?
Yes, and AI can perform more demanding tasks. OpenAI recently made ChatGPT do calculations while it was being used, which it hadn’t done before. So now, while its AI is answering questions, it's as if it is considering several options and returning back to them just like people do when they're solving a problem. It's nothing new, we've known how to do it for a long time, it just requires performance and that's expensive. OpenAI just figured out how to do it so that it's not as expensive. The race to see who can scale a model is still there.
To stay up to date with what scientists in Slovakia or Slovak scientists around the world are doing, subscribe to the Slovak Science newsletter, which will be sent to readers free of charge four times a year.
Last time we talked, you were working at DeepMind, but you moved to Meta in March. What was behind the change?
Mark Zuckerberg contacted me directly.
What did he tell you?
Basically, that AI is very important to Meta and that he wants us to work together.
That sounds like a show of respect for your work.
