https://szeching.com/tags/web-crawler/ Recent content in Web Crawler on 瓶子裡的小狐狸 Hugo zh-Hant Thu, 27 Nov 2014 00:57:49 +0000 https://szeching.com/rare-web-spider/ Thu, 27 Nov 2014 00:57:49 +0000 https://szeching.com/rare-web-spider/ <p>看网站LOG信息有奇怪的东西混入：</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f">1</span><span>130.211.147.7 - - [23/Nov/2014:15:36:20 +0000] &#34;GET / HTTP/1.0&#34; 200 18701 &#34;-&#34; &#34;NerdyBot&#34; </span></span><span style="display:flex;"><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f">2</span><span>107.22.83.14 - - [23/Nov/2014:15:13:36 +0000] &#34;GET /robots.txt HTTP/1.1&#34; 200 345 &#34;-&#34; &#34;HubSpot Crawler 1.0 http://www.hubspot.com/&#34; </span></span><span style="display:flex;"><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f">3</span><span>192.99.107.190 - - [24/Nov/2014:14:49:46 +0000] &#34;GET /robots.txt HTTP/1.1&#34; 200 357 &#34;-&#34; &#34;Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)&#34; </span></span></code></pre></div><p>搜索了一下，得知：</p>