Search Engine Spider and User Agent Identification with “Ultimate User Agent Blacklist”
A user-agent is software (a software agent) that is acting on behalf of a user.
In many cases, a user-agent acts as a client in a network protocol used in communications within a client–server distributed computing system.
For more information view wiki
http://en.wikipedia.org/wiki/User_agent
Automated Agents is called as Bots.
http://user-agents.org has a complete list of all latest spiders/bots/user agents.
There are some more and those are anonymous (not known and have very different names).
If you develop a website and want to make the site accessible by some specific user agent or bots of a country, then you can update the information in your root .htaccess file.
As, Bot blocking blacklists are useless in some way as some rogue spiders just generate random user-agent strings so we will never have them in our list to start with, but We have tried list out as much as we can in the below zip file.
What you have to done is, unzip the file and paste the code in your root .htaccess file, it will protect your website from unwanted crawling/indexing by anonymous bots.
If your website uses WordPress open-source, then can also use “Better WP Security” plug-in.
Just need to download and install the plug-in and go to “Better WP Security – Ban Users” Tab and enable “Enable Default Banned List”, you can also update the list according to your needs.
But please be careful before doing same, as it may affect your website’s core files and plug-in, So before doing this backup your website’s files and database.
