Become a Googlebot, get a Free Pass

[blackpearl]

The Problem:

Sometimes you conduct some search on Google and the engine returns a number of results, but when you try to open the ones that looks the most promising you get a registration page. One thing that you can do is click on the cached content to view the page directly from Google's cache. But what if you want to view another page from the same site, not present in google's cache?

The Solution:

Its evident that there are some sites that allows Google but not you. So why not disguise yourself as Google? Thats exactly what we will do by changing the browser's user agent to Googlebot. Copy the following code into notepad and save it as a .reg file

Windows Registry Editor Version 5.00 
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent] 
@="Googlebot/2.1" 
"Compatible"="+http://www.googlebot.com/bot.html"

Now double-click the file to merge it into registry. In order to change back to the default user agent use this:

Windows Registry Editor Version 5.00 
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent] 
@="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Unfortunately, this only works with IE and not Opera. There is no way to add new user agent in Opera AFAIK. There is some way to do it in Firefox but you have to search that yourself on google. To check whether you have sucessfully changed the user agent, type the follwing into the address bar and hit enter.

javascript(BrowserDetect.browser)

Now for some practice. One good example is http://www.governmentsecurity.org/forum/index.php. Anybody who has tried to visit this website will find that it requires you to register in order to view the forum. But registering is tricky. You need to submit your own article to the site afterwhich they will evaluate it and then decide whether to give you membership or not. Now visit this forum after spoofing your IE's user agent, aaha you are welcomed!!!

Note, that this method might not work with all websites.

Fun Exercise 1:

Use a different browser like Opera or Firefox to login to thinkdigit.com. Then click on Quick Links>Who's online (you will also find this near the bottom of the page). On the page that opens choose "Search Bots" and hit go. No matter what time of the day or night it is, you will find a couple of googlebots, msnbots and yahoobots crawling on the website. Now open IE and view thinkdigit.com, this time as a googlebot. Refresh the page you were viewing in your other browser. This time you will see another googlebot appearing which is actually you!!

Fun Exercise 2:

On the same "Who's online" page you can see what thread the other online members and guest are viewing at that particular time. Speaking of privacy, huh?

[soham]

Nice tutorial. But wont you land up in trouble if you mess with government security sites?

[blackpearl]

It isn't a government website, just the name is.

Anyway, if the mods feel that reference to that website was inappropiate then they might remove the link.

[Chirag]

Alternate way to bypass registration page is find some username and pass at http://bugmenot.com

[AshishSharma]

Simple Get "User Agent Switcher" Add-On for Firefox, whenever a site doesn't allow you in change your user Agent to Goglebot and then refresh ....

[piyush gupta]

gud tutor

[hemant_mathur]

Anything for Opera ??
__________

Originally Posted by AshishSharma

Simple Get "User Agent Switcher" Add-On for Firefox, whenever a site doesn't allow you in change your user Agent to Goglebot and then refresh ....

It only shows IE6, Netscape 4.8 and Opera 8.5 but no googlebot. How to add that ?

[forever]

omfg, this does work, for ff u dont need to create any reg entry

-- go to TOOLS > USER AGENT SWITCHER > OPTIONS > OPTIONS, there add a user agent and in the DESCRIPTION FIELD use "Googlebot/2.1" (without " )
and in USER AGENT type "+http://www.googlebot.com/bot.html", and now use this agent whenever logging into a protected forum, hopefully itll work, thanks a ton blackpearl

[hemant_mathur]

^^ Thanx for the info

[freakitude]

Thanks for the info

See this thread http://www.thinkdigit.com/forum/tutorials/36119-avoid-compulsory-registration-website-post304946.html where i posted some tricks to avoid compulsory registration...

You can download the Firefox user agent switcher extension from http://addons.mozilla.org/firefox/59/ . Then you can switch the user agent so that the reg-only website is fooled to think that you are a web spider.

[Akhil Jain]

really good info

[tuxfan]

Good tips! I added a point to your reputation

[prakash2119]

Iam unable to merge in to registry its giving some error that it cant import ...iam on windows98

[Quiz_Master]

WORKING.....
Cool...

[forever]

@prakash
the reg editor version for 98 is different, i dont remember it , but that is what is needed to be changed i think..

[blackpearl]

Originally Posted by prakash2119

Iam unable to merge in to registry its giving some error that it cant import ...iam on windows98

For Win98 replace the line "Windows Registry Editor Version 5.00 " with "REGEDIT4".

[delivi]

thax dude for this tutiorial

This is the first time I knew that we can change the useragent.

[shaunak]

Not working imm on vista and ie7

[anandk]

pretty impressive thanx !

[debiprasad_sahoo]

Originally Posted by Chirag

Alternate way to bypass registration page is find some username and pass at http://bugmenot.com

Nice Info dude. Thanks!

[alsiladka]

Even IE7Pro has Google Bot as one of the User Agents in the options. Just select and restart IE.

[Zeeshan Quireshi]

or use the "User Agent Switcher" extension if you're on firefox .

https://addons.mozilla.org/en-US/firefox/addon/59

[hullap]

Great yaar