Changeset 33568 for gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt

Timestamp:

2019-10-14T23:36:54+13:00 (5 years ago)

Author:

ak19

Message:

1. More sites greylisted and blacklisted, discovered as I attempted to crawl them and afterwards learnt to investigate sites first. Should all .ru and .pl domains be on the greylist? 2. Adjusted instruction comments in CCWETProcessor for compiling and running

File:

• gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt (modified) (2 diffs)

gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt

@@ -8 +8 @@
 
 
-# Product sites: unwanted auto-translation pages of online product stores
+# Product sites: unwanted auto-translation pages of online product stores and other websites
 /product/
 /products/
@@ -16 +16 @@
 ledpar64.china-led-lighting.com
 ledwallwasher.china-led-lighting.com
+abacre.com
+cn-huafu.net
+
+# not product stores but autotranslated?
+192-168-1-1l.com
+19216811login.club
+19216811login.club
+1videosmusica.com
+256file.com
+7773033.ru
+abali.ru
+allbeautyone.ru
+
+# if page doesn't load and can't be tested
+1videosmusica.com
+www.kiterewa.pl
+
+# license plate site?
+eba.com.ru
+
+# As per archive.org, there's just a photo on the defunct page at this site
+# And the picture label and filename is probably Japanese
+agri.mine.utsunomiya-u.ac.jp
+
+# seems to be Indonesian or Malaysian Bible rather than in Maori or any Polynesian language
+alkitab.life:2022