So,who,or what is "Google spider"?
1 - 4 of 4 Posts
It's Googles software search engine spying searching the web to update Googles indexes.
The question is WHY has googles crawling of the web increased so dramatically lately?
Hopefully this current spider is really spotting and flagging sites that are using redirects and other such scummy methods.
The question is WHY has googles crawling of the web increased so dramatically lately?
Hopefully this current spider is really spotting and flagging sites that are using redirects and other such scummy methods.
Joined
·
4,814 Posts
Spiders are just programs that will go to the opening page of any site (usually index.html) and then follow every link off of that page through the entire site, indexing keywords and phrases to be used in the search engines so people can find relevant matches to words or phrases they are searching for.
Generally speaking, spiders are welcomed on nearly every site, because getting you site to have a high hit ratio on keywords puts you closer to the top of the listing when someone requests a search on that keyword. For instance, a site like this would like to be as close to the top of the listing as possible if someone were searching for "guns", "arms", etc.
The fact that a spider was in a user's profile really means nothing at all. It was just a link that the spider was following because it is a valid link within the code of the page it was searching on. Basically any button or link you see on a page is a link that a spider will follow when it gets to that page.
And yes, sometimes spiders can go insane. More then once I have had to reboot my server because a whole nest of spiders had settled into one of my sites and were bogging down the system. FaunaClassifieds once had 470 members and guests logged in, the majority of which were just spiders. So I had to shake them loose with a reboot.
Generally speaking, spiders are welcomed on nearly every site, because getting you site to have a high hit ratio on keywords puts you closer to the top of the listing when someone requests a search on that keyword. For instance, a site like this would like to be as close to the top of the listing as possible if someone were searching for "guns", "arms", etc.
The fact that a spider was in a user's profile really means nothing at all. It was just a link that the spider was following because it is a valid link within the code of the page it was searching on. Basically any button or link you see on a page is a link that a spider will follow when it gets to that page.
And yes, sometimes spiders can go insane. More then once I have had to reboot my server because a whole nest of spiders had settled into one of my sites and were bogging down the system. FaunaClassifieds once had 470 members and guests logged in, the majority of which were just spiders. So I had to shake them loose with a reboot.
1 - 4 of 4 Posts
Join the discussion
Join now to ask and comment!