This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
Re: Don't let search bots look at buglist.cgi
Richard Guenther <richard.guenther@gmail.com> writes:
> On Fri, May 13, 2011 at 7:14 PM, Ian Lance Taylor <iant@google.com> wrote:
>> I noticed that buglist.cgi was taking quite a bit of CPU time. ÂI looked
>> at some of the long running instances, and they were coming from
>> searchbots. ÂI can't think of a good reason for this, so I have
>> committed this patch to the gcc.gnu.org robots.txt file to not let
>> searchbots search through lists of bugs. ÂI plan to make a similar
>> change on the sourceware.org and cygwin.com sides. ÂPlease let me know
>> if this seems like a mistake.
>>
>> Does anybody have any experience with
>> http://code.google.com/p/bugzilla-sitemap/ ? ÂThat might be a slightly
>> better approach.
>
> Shouldn't we keep searchbots way from bugzilla completely? Searchbots
> can crawl the gcc-bugs mailinglist archives.
I don't see anything wrong with crawling bugzilla, though, and the
resulting links should be better.
Ian