Changeset 33571 for gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh

Timestamp:

2019-10-16T21:11:26+13:00 (5 years ago)

Author:

ak19

Message:

Adding Dr Bainbridge's suggestion of appending the crawlId of each site to a single file as soon as that site's been crawled. Haven't yet understood how and where to incorporate the other part that Dr Bainbridge mentioned of using echo and file completion using patterns to launch batchcrawl from another script.

File:

• gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh (modified) (1 diff)

gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh

@@ -84 +84 @@
     mv ${siteDir}UNFINISHED $outputDir/$crawlId/UNFINISHED
     fi
+
+    # finally append the current crawlId to siteIDsCrawled.txt
+    echo $crawlId >> $outputDir/siteIDsCrawled.txt
 }