Greg Hewgill (ghewgill) wrote,
Greg Hewgill
ghewgill

search engine spider gone awry

So I happened to look at the log file for one of my web servers today, and noticed lots of requests for the Apache manual, which is installed and served by the default configuration. The requests were all coming from "msnbot/0.3", a search engine spider for MSN. I changed the Apache config to remove the manual, and it seems that the requests are coming in more frequently now that they return 404. I wonder how much time it will take before msnbot gives up.
Tags: web
Subscribe

  • foiled again

    So I finished my earthquake widget about a week or so ago. I was waiting to send it to the Yahoo Widget Gallery so that the screen snapshot they…

  • email address obfuscation

    At the bottom of each of my web pages, I have a footer that looks something like this: Greg Hewgill greg@hewgill.com Having my email…

  • minilink reaches 10k links

    The minilink (lnk.nu) link shortening service first went online in october 2004. Today, a milestone was passed - there have been more than 10,000…

  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    Help
  • 3 comments