Support

Hello,

I'm running very private site for our members, no one shall access the site so I put every module and article and menu link to registered.
I set: NO INDEX, NO FOLLOW in the global config.
and I have set the Robots txt as the following:

https://www.ostraining.com/blog/joomla/how-to-prevent-google-search-engine-from-crawling-your-joomla-urls-using-robots-txt-/

Is there is any extra setting that I can do on Admin tools to make sure to help to achieve that when I create the .htaccess.

Thanks, :)

You can enable the blocking of the list of user agents in .htaccess Maker (and regenerate the .htaccess file). You would need to add legitimate search bots to the list. A Google search found this list of bots: http://www.useragentstring.com/pages/Crawlerlist/.