Yandex improved search using the CS YATI neural network, a new model trained on documents for IT specialists and assessments of programming experts. Search results for developers and ML specialists have become better, and query navigation has become more convenient.
The new model takes into account one and a half times more information from the page than its previous version – YATI. The updated transformer neural network analyzed many search queries and sites that are shown for queries related to programming. This helps her better evaluate the quality and relevance of the document to the query. By running through terabytes of programming documents and a history of searching for experts, CS YATI has also learned to predict the clicks of qualified programmers in order to produce the most relevant answer.
Alexey Gusakov, Head of Machine Intelligence and Research Department has been quoted saying:
It is known that the lion’s share of programming requests are requests in English. CS YATI was trained mainly on English-language sources. We didn’t just improve the search for programmers: in the process, we also improved the search for English-language sources.
In addition, Yandex has significantly improved the enriched Stack Overflow answer. Directly in the search results, without going to the site, the user will see additional information: the question itself, the best answer to it and other comments that may be useful to programmers. Yandex also improved the display of snippets for GitHub and NPM by adding useful information there.
In 2020, Yandex launched a text analysis technology based on transformer neural networks, which perfectly solve problems in the field of natural language processing, but require a huge amount of computing resources. Thanks to this technology, Yandex has become much better at assessing the semantic relationship between queries and the content of documents on the Internet – so much so that this launch can be considered the largest search event in the last ten years. This technology was named YATI.
