https://szeching.com/tags/web-crawler/ Recent content in Web Crawler on 瓶子裡的小狐狸 Hugo -- gohugo.io zh-Hant © 2026 Y Cheung Thu, 27 Nov 2014 00:57:49 +0000 https://szeching.com/rare-web-spider/ Thu, 27 Nov 2014 00:57:49 +0000 https://szeching.com/rare-web-spider/ <p>看网站LOG信息有奇怪的东西混入：</p> <div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="ln">1</span><span class="cl">130.211.147.7 - - [23/Nov/2014:15:36:20 +0000] &#34;GET / HTTP/1.0&#34; 200 18701 &#34;-&#34; &#34;NerdyBot&#34; </span></span><span class="line"><span class="ln">2</span><span class="cl">107.22.83.14 - - [23/Nov/2014:15:13:36 +0000] &#34;GET /robots.txt HTTP/1.1&#34; 200 345 &#34;-&#34; &#34;HubSpot Crawler 1.0 http://www.hubspot.com/&#34; </span></span><span class="line"><span class="ln">3</span><span class="cl">192.99.107.190 - - [24/Nov/2014:14:49:46 +0000] &#34;GET /robots.txt HTTP/1.1&#34; 200 357 &#34;-&#34; &#34;Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)&#34;</span></span></code></pre></div></div> <p>搜索了一下，得知：</p> <ul> <li>NerdyBot是搜索引擎 (<a href="http://nerdydata.com/" target="_blank" rel="noreferrer">http://nerdydata.com/</a>) 的爬虫，这个主要是面向开发者搜索源代码（source code）的引擎。</li> <li>HubSpot Crawler是数字营销公司的爬虫，已经IPO了，连google都投资它了，传说hubspot的企业文化十分令人艳羡。</li> <li>meanpathbot 是付费搜索引擎 (<a href="http://meanpath.com/" target="_blank" rel="noreferrer">http://meanpath.com/</a>) 的爬虫，它的主要客户群是做语义分析/恶意软件及病毒分析/身份防窃保护之类的，免费试用可以点<a href="https://meanpath.com/f/" target="_blank" rel="noreferrer">这里</a>。</li> </ul> <p>突然觉得，有空写写爬虫也还蛮好玩的&hellip;</p>