Yandex, and Microsoft, have announced a new initiative today by the name of IndexNow. This is a protocol that allows (any) search engine to let webmasters have their content (page URLs) indexed instantaneously by the search engine.
At the time of writing, Yandex and (Microsoft) Bing are the only search engines to fully support this initiative.
The goal of the initiative is to make indexing of URLs and content “more efficient”, and with more efficiency this deep into the stack, it also means that the internet has become a little bit “greener”.
By telling search engines whether an URL has been changed, website owners provide a clear signal helping search engines to prioritize crawl for these URLs, thereby limiting the need for exploratory crawl to test if the content has changed.
Yandex Guidelines
The first step is to generate a user key.
These keys are UTF-8 text strings, between 8 and 128 characters in length.
As a .txt file. this should then be placed in the site’s root directory.
It’s important to note that if you use different CMS, you should use a separate key for each. You can use the same key across multiple websites (domains), and you only need to change your key if it becomes public, and then used/abused/known to third-parties.
You should also use separate keys for each subdomain (www., blog., support., etc)
You can then notify Yandex of changes to URLs through both GET and POST methods.
You should use GET for individual URL changes:
GET https://yandex.com/indexnow
? url=
& key=
& [keyLocation=]
And you should use POST for multiple URLs:
POST https://yandex.com/indexnow
And use JSON format:
POST /indexnow HTTP/1.1
Content-Type: application/json; charset=utf-8
Content-Length: 286
Host: https://yandex.com
{
"host": "www.example.com",
"key": "EdD8dkmdNLlxREi2LkhJjYOH2kyQbJqM3cBKT5fX",
"keyLocation": "https://www.example.com/myIndexNowKey63638.html",
"urlList": [
"https://www.example.com/url1",
"https://www.example.com/folder/url2",
"https://www.example.com/url3"
]
}
Abuse Limits & Submission Frequency Advice
It is also worth noting that Yandex has specified limits, and measures, to prevent abuse of these features.
- Via the POST method, you can only submit 10,000 URLs per JSON
- There is no limit to how often you can submit URLs via GET or POST. Yandex have advised however that they have algorithms in place to prevent abuse of this function.
- If you have a page that updates daily/a lot, you can use <changefreq> in the XML sitemap to indicate daily changes, which is supported.
Documentation Links
- Yandex GET documentation
- Yandex POST documentation
