Changeset 33569 for gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh

Timestamp:

2019-10-16T20:00:09+13:00 (5 years ago)

Author:

ak19

Message:

1. batchcrawl.sh now does what it should have from the start, which is to move the log.out and UNFINISHED files into the output folder instead of leaving them in the input folder, as the input to_crawl folder can and does get replaced all the time, every time I regenerate it after black/white/greylisting more urls. 2. Blacklisted more adult sites, greylisted more product sites and .ru, .pl and .tk domains with whitelisting in the whitelist file. 3. CCWETProcessor now looks out for additional adult sites based on URL and adds them to its blacklist in memory (not the file) and logs the domain for checking and manually adding to the blacklist file.

File:

• gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh (modified) (1 diff)

gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh

@@ -74 +74 @@
         echo "CRAWL FAILED." 2>&1 | tee -a ${siteDir}log.out
     fi
-    
+
+
+    # move the peripheral crawl products (the log.out and UNFINISHED files)
+    # from the input to the output folder. This way we can re-run the crawl and
+    # the original output will still have been preserved
+    mv ${siteDir}log.out $outputDir/$crawlId/log.out
+    mv ${siteDir}UNFINISHED $outputDir/$crawlId/UNFINISHED
 }