Changeset 33569 for gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt

Timestamp:

2019-10-16T20:00:09+13:00 (5 years ago)

Author:

ak19

Message:

1. batchcrawl.sh now does what it should have from the start, which is to move the log.out and UNFINISHED files into the output folder instead of leaving them in the input folder, as the input to_crawl folder can and does get replaced all the time, every time I regenerate it after black/white/greylisting more urls. 2. Blacklisted more adult sites, greylisted more product sites and .ru, .pl and .tk domains with whitelisting in the whitelist file. 3. CCWETProcessor now looks out for additional adult sites based on URL and adds them to its blacklist in memory (not the file) and logs the domain for checking and manually adding to the blacklist file.

File:

• gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt (modified) (4 diffs)

gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt

@@ -18 +18 @@
 abacre.com
 cn-huafu.net
+apteka.social
+
 
 # not product stores but autotranslated?
@@ -25 +27 @@
 1videosmusica.com
 256file.com
-7773033.ru
-abali.ru
-allbeautyone.ru
+# already in greylisting of all .ru 
+#7773033.ru
+#abali.ru
+#allbeautyone.ru
+aqualuz.org
 
 # if page doesn't load and can't be tested
@@ -33 +37 @@
 www.kiterewa.pl
 
-# license plate site?
-eba.com.ru
+
+
+# MANUALLY INSPECTED URLS AND ADDED TO GREYLIST
+
+# license plate site? - already in greylisting of all .ru
+#eba.com.ru
 
 # As per archive.org, there's just a photo on the defunct page at this site
@@ -42 +50 @@
 # seems to be Indonesian or Malaysian Bible rather than in Maori or any Polynesian language
 alkitab.life:2022
+
+# appears defunct
+alixira.com
+
+# single seedURL was not a page in Maori, but global languages.
+# And the rest of the domain appears to be in English
+anglican.org
+
+
+### TLDs that we greylist - any exceptions will be in the whitelist
+# Our list of .ru and .pl domains were not relevant
+.ru/
+.pl/
+.tk/