Greg Hewgill (ghewgill) wrote,
Greg Hewgill
ghewgill

search engine spider gone awry

So I happened to look at the log file for one of my web servers today, and noticed lots of requests for the Apache manual, which is installed and served by the default configuration. The requests were all coming from "msnbot/0.3", a search engine spider for MSN. I changed the Apache config to remove the manual, and it seems that the requests are coming in more frequently now that they return 404. I wonder how much time it will take before msnbot gives up.
Tags: web
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 3 comments