OffBeatMammal

Searching for monkeys in Cyberspace

The spiders finally become an infestation!

clock January 6, 2008 16:09 by author offbeatmammal

Back at the beginning I registered a new domain. Almost immediately Googlebot turned up and indexed it, followed less than a week later by Baiduspider. It's taken until now however for the rest of the gang to turn up - over this weekend the site finally got a visit from Yahoo! Slurp, MSNbot and a surprise visitor from MERL.

What makes the progression interesting (and as someone who works at MS a little worrying that we took so long) is not just the fact that Google and Baidu found the site before it was linked from anywhere (do they have deals with the hosting ISP, the domain registrar or do they watch DNS servers for new sites - and do Baidu get data from Google, which would explain the quick following in the footsteps) but that even when it was in other indexes and had a couple of links from sites that are much longer established they took so long to find the site... and the quality of a search on both Yahoo! and MSN doesn't yet return as much content as Google (can't speak for Baidu until I find a Chinese speaker).

Since the latest upgrade of the Live search engine I've been pretty impressed with the quality of the results and the speed (it's certainly a much stronger contender for Google now) but it seems that there's still a little bit of work do go on agility to make sure it finds sites quicker and gets a better in-depth picture. Older, better established, sites don't seem to have the depth or discovery issues so it's obviously something a few repeat visits would resolve.

Still no sign of Ask (formerly known as AskJeeves) or some of the other second tier players... but except for a few niche uses I've not seen much reason to desert the big two (ah, I remember finally switching from WebCrawler to Google years ago!)



The spiders keep coming

clock December 16, 2007 19:28 by author offbeatmammal

Just over a week ago I registered a new domain (on the Friday night). On Sunday I put the holding site up there and on Monday the logs showed a visit from Googlebot. Well, it's now a full seven days since the holding site went up and we got our second spider hit... this time from Baidu in China - 你好!

It's interesting to see the uptake. I've deliberately not seeded any search engines, or linked to the site from my blog, in fact the only clue to its existence is fleeting messages on the Twitter public timeline.

It'll be interesting to see how it grows and what order the different spiders find me in.



Search

Calendar

<<  May 2013  >>
SuMoTuWeThFrSa
2829301234
567891011
12131415161718
19202122232425
2627282930311
2345678

Sign in

Twitter


    follow OffBeatMammal at http://twitter.com


    Amazon Store


     
    Donate unused CPU cycles with BOINC Stats and Account Management from BOINCStats.com



    Blogroll

    Archive

    Tags

    Categories


    Disclaimer

    The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

    © Copyright 2013