Get content from the website only open to Search Bots
Ever experienced this? You ask Google to look something up; the engine returns with a number of finds, but if you try to open the ones with the most promising content, you are confronted with a registration page instead, and the stuff you were looking for will not be revealed to you unless you agree to a credit card transaction first….
The lesson you should have learned here is: Obviously Google can go where you can’t.
In fact, many sites that force users to register or even pay in order to search and use their content, leave a backdoor open for the Googlebot, because a prominent presence in Google searches is known to generate sales leads, site hits and exposure.
So, disguise yourself as a Googlebot !!!
