獲取最新惡意爬蟲列表配置 Fail2ban Filter

Thu, 25 Nov 2021 11:04:00 +0000

安裝並設置好 Fail2ban 後（可以參考Y Cheung 之前寫的『Fail2ban 配置 Nginx filter』），可以看到在 /etc/fail2ban/filter.d/nginx-badbots.conf 內容如下：

 1# /etc/fail2ban/filter.d/nginx-badbots.conf
 2# Fail2Ban configuration file
 3#
 4# Regexp to catch known spambots and software alike. Please verify
 5# that it is your intent to block IPs which were driven by
 6# above mentioned bots.
 7
 8[Definition]
 9
10badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|(?:Mozilla/\d+\.\d+ )?Jorgee
11
12badbots = Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 \+http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots&#44; \+http\://www\.snap\.com\)|sogou develop spider|Sogou Orion spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00
13
14failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$
15
16ignoreregex =
17
18datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
19              ^[^\[]*\[({DATE})
20              {^LN-BEG}
21      
22# DEV Notes:
23# List of bad bots fetched from http://www.user-agents.org
24# Generated on Thu Nov  7 14:23:35 PST 2013 by files/gen_badbots.
25#
26# Author: Yaroslav Halchenko

如果沒有找到這個文件，請新建一個，或者將 apache-badbots.conf 拷貝並重命名。

Fail2ban 配置 Nginx filter

Mon, 15 Jun 2020 12:00:00 +0000

簡要紀錄一下目前服務器上的 fail2ban 關於 nginx 的 filter 配置。

惡意爬蟲過濾器

 1# /etc/fail2ban/filter.d/nginx-badbots.conf
 2[Definition]
 3
 4badbotscustom = Sogou web spider|DotBot|AhrefsBot|Baiduspider|PetalBot|WOW64|Daum|Barkrowler|360Spider|Buck|Photon|SEOkicks|magpie-crawler|SemrushBot|SeznamBot|MJ12bot|EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|(?:Mozilla/\d+\.\d+ )?Jorgee
 5
 6badbots = Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 \+http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots&#44; \+http\://www\.snap\.com\)|sogou develop spider|Sogou Orion spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00
 7
 8failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*"*(?:%(badbots)s|%(badbotscustom)s).*"$
 9
10ignoreregex =
11
12datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
13              ^[^\[]*\[({DATE})
14              {^LN-BEG}
15
16/etc/fail2ban/filter.d/nginx-badbots.conf

自定義規則

 1# /etc/fail2ban/filter.d/nginx-custom.conf
 2[Definition]
 3failregex = ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+XDEBUG.+$
 4            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+GponForm.+$
 5            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+phpunit.+$
 6            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+ajax-index\.php .+$
 7            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+sellers\.json .+$
 8            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+adminer\.php .+$
 9            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+wp-configuration\.php.+$
10            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+ThinkPHP.+$
11            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+wp-config.+$
12            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+dede\/login\.php .+$
13            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+plus\/recommend\.php .+$
14            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+e\/install\/index.php .+$
15            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+m\/e\/install\/index\.php .+$
16            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+e_bak\/install\/index.php .+$
17            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+\.aspx .+$
18            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+\.act .+$
19            ^<HOST>.*] "(GET|POST|HEAD) .*xmlrpc\.php.*
20            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.Mozilla/5\.0 \(Windows NT 6\.1\; rv\:60\.0\) Gecko\/20100101 Firefox\/60\.0.*"$
21            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.Photon.*"$
22            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.Mozilla\/5\.0 zgrab\/0\.x.*"$
23            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.XTC.*"$
24            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.python-requests.*"$
25            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.bidswitchbot.*"$
26            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.Google-adstxt.*"$
27            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.Apache-HttpClient.*"$
28            ^<HOST>.*] "POST .*HTTP\/1\.1.*
29            ^<HOST> -.*"(GET|POST|HEAD).*HTTP\/1\.1.*"*.Go-http-client\/1\.1.*"$
30            ^<HOST> .* ".*\\x.*" .*$
31            ^<HOST> -.*"(GET|POST|HEAD) .+wp-login\.php .*HTTP\/1\.1.*
32            ^<HOST> -.*"(GET|POST|HEAD) .*wp-includes/wlwmanifest.xml .*HTTP\/1\.1.*
33ignoreregex = ^<HOST>.*] "POST /xmlrpc\.php\?for=jetpack.*
34              ^<HOST>.*] "POST /wp-cron\.php\?doing_wp_cron=.*
35datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
36              ^[^\[]*\[({DATE})
37              {^LN-BEG}

簡要說明一下：

Fail2ban on 瓶子裡的小狐狸

獲取最新惡意爬蟲列表配置 Fail2ban Filter

Fail2ban 配置 Nginx filter

惡意爬蟲過濾器

自定義規則