Changeset 33568

Timestamp:

2019-10-14T23:36:54+13:00 (5 years ago)

Author:

ak19

Message:

1. More sites greylisted and blacklisted, discovered as I attempted to crawl them and afterwards learnt to investigate sites first. Should all .ru and .pl domains be on the greylist? 2. Adjusted instruction comments in CCWETProcessor for compiling and running

Location:

gs3-extensions/maori-lang-detection

Files:

• conf/sites-too-big-to-exhaustively-crawl.txt (modified) (1 diff)
• conf/url-blacklist-filter.txt (modified) (2 diffs)
• conf/url-greylist-filter.txt (modified) (2 diffs)
• src/org/greenstone/atea/CCWETProcessor.java (modified) (1 diff)

gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt

@@ -55 +55 @@
 00.gs,SINGLEPAGE
 
+# May be a large site
+topographic-map.com,SINGLEPAGE
 
 # TOP SITES

gs3-extensions/maori-lang-detection/conf/url-blacklist-filter.txt

@@ -64 +64 @@
 wildsexsluts.com
 xxxblacknudes.com
+bigsexymelons.com
+
+# more adult sites
+acba.osb-land.com
 
 # sounds like some pirating site
 ^http://pirateguides.com/
+fastmp3.ru
 
 # from alexa topsites at https://www.alexa.com/topsites
@@ -76 +81 @@
 xhamster.com
 xnxx.com
+
+
+# not sure about the domain name and/or full url seems like it belongs here
+abcutie.com

gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt

@@ -8 +8 @@
 
 
-# Product sites: unwanted auto-translation pages of online product stores
+# Product sites: unwanted auto-translation pages of online product stores and other websites
 /product/
 /products/
@@ -16 +16 @@
 ledpar64.china-led-lighting.com
 ledwallwasher.china-led-lighting.com
+abacre.com
+cn-huafu.net
+
+# not product stores but autotranslated?
+192-168-1-1l.com
+19216811login.club
+19216811login.club
+1videosmusica.com
+256file.com
+7773033.ru
+abali.ru
+allbeautyone.ru
+
+# if page doesn't load and can't be tested
+1videosmusica.com
+www.kiterewa.pl
+
+# license plate site?
+eba.com.ru
+
+# As per archive.org, there's just a photo on the defunct page at this site
+# And the picture label and filename is probably Japanese
+agri.mine.utsunomiya-u.ac.jp
+
+# seems to be Indonesian or Malaysian Bible rather than in Maori or any Polynesian language
+alkitab.life:2022

gs3-extensions/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java

@@ -53 +53 @@
  *
  * To run, passing the log4j and other properties files in conf/ folder:
- *      maori-lang-detection/src$ java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor <folder containing warc.wet(.gz) files> <outputFolder>
+ *      maori-lang-detection/src$ java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor <folder containing commoncrawls subfolders containing warc.wet(.gz) files> <outputFolder>
  *
- * e.g.
- *    - java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor ../tmp/processWET /Scratch/ak19/gs3-extensions/maori-lang-detection/tmp/processedWET
- *    - java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor ../tmp/processWET /Scratch/ak19/gs3-extensions/maori-lang-detection/tmp/processedWET 2>&1 | less
+ * e.g. (from maori-lang-detection/src)
+ *    
+ *    - java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor ../ccrawl-data /Scratch/ak19/gs3-extensions/maori-lang-detection/to_crawl
+ *    - java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor ../ccrawl-data /Scratch/ak19/gs3-extensions/maori-lang-detection/to_crawl 2>&1 | less
  *
 */