Being Crawled

[Rus]

I'm dropping this in chit chat because forum admins are most likely to know even if it doesn't have anything to do with TP.

Inktomi Corporation, or Yahoo has been crawling my site with 7-48 guest IP addys at a time for almost a week.  Google ads have all gone to the public service announcements and I'm guessing its because of all the garbage traffic I'm getting from Yahoo.

Does anyone know why this is happening?  I know they need to crawl thoroughly to have relevant links for their search engine but shouldn't they be done for now considering I'm a small site and they've been going so hot and hard on me?

I've banned their IP range to cut this out.  Any thoughts on if thats a good idea or not?

[Zetan]

Yahoo crawls my site relentlessly.. and has done for a long time. I get a huge number of spiders, probably more than necessary. Yahoo and Google are hugely competitive and big rivals.

I just let them get on with it and never really give it much thought on how to limit the number. But I see a huge list of guests here, at SMF and other boards including vBulletin boards and I know for a fact that 90% of all guests listed are spiders. Many are viewing profiles, calendar, register page and general forum posts.

I wouldn't block them unless you don't want to be indexed at all by search engines. I would have a search at SMF and The Admin Zone, you may find that a lot of other people have asked the same thing.

[Rus]

Considering that my site gets very little traffic Yahoo is generating the vast majority of my traffic so the ads are all PSAs now. 

Makes it kinda ugly.  It never was really pretty but doesn't need PSA ugly to boot.

[JPDeni]

I use a robots.txt file that only allows Google. Of course, there are a whole lot of 'bots that don't respect it, but Yahoo does.

I'm attaching my file, which you are welcome to use. Just put it in your main forum directory. It won't stop them all, but it'll stop the big ones except Google.

[Zetan]

I have a robots file.. I will also have a look at yours Deni.. lol, when my site is finally working again 
Google does have the larger search engine.. but Yahoo does have more actual customers across all the sites they own.. which is a lot of sites.

[JPDeni]

I just decided that I was only going to use one search engine and Google is the one I use, so that's the one I chose.

[Zetan]

I just decided that I was only going to use one search engine and Google is the one I use, so that's the one I chose.

I really only use Google as a search engine and Yahoo does seem to pump out more bots than Google does. I get pages of them.

[Rus]

Cool, thanks!