Support

I have explained before that this is something we cannot do. It's not a technical limitation, it's more of a "people are stupid" issue.

If we start adding the missing domains and IPs on update people would be upset we're messing with their site's configuration. Even worse, we've had some loud idiots who in more than one occasion complained for something asinine, e.g. that allowing their public site to be indexed by Baidu is allowing the Communist Party of China to hack their site. Trying to explain the obvious fact that no, this is not the case, to a moron of that magnitude is like trying to fight a pig in a mud pit: pointless, you get dirty, and the pig enjoys it.

As I've said before, what we do instead is document what are the default values. You can always consult the documentation and add whatever you think should be added. Of course, you can also consult the same sources we did —the search engines' documentation on their bots— to add the relevant IPs and domain names yourself. Maybe there's a search engine we missed, or something did change and we missed the change.