Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/Readme.txt
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/Readme.txt	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/Readme.txt	(revision 33527)
@@ -0,0 +1,386 @@
+----------------------------------------
+INDEX: follow in sequence
+----------------------------------------
+A.  VAGRANT VM WITH HADOOP AND SPARK
+B.  Create IAM role on Amazon AWS to use S3a
+C.  Configure Spark on your vagrant VM with the AWS authentication details
+D.  OPTIONAL? Further configuration for Hadoop to work with Amazon AWS
+E.  Setup cc-index-table git project
+F.  Setup warc-to-wet tools (git projects)
+G.  Getting and running our scripts
+----------------------------------------
+
+----------------------------------------
+A. VAGRANT VM WITH HADOOP AND SPARK
+----------------------------------------
+Set up vagrant with hadoop and spark as follows
+
+1. by following the instructions at
+https://github.com/martinprobson/vagrant-hadoop-hive-spark
+
+This will eventually create the following folder, which will contain Vagrantfile
+/home/<USER>/vagrant-hadoop-hive-spark
+
+2. If there are other vagrant VMs set up according to the same instructions on the same machine, then need to change the forwarded ports (the 2nd column of ports) in the file "Vagrantfile". In the example below, excerpted from my Vagrantfile, I've incremented the forwarded ports by 1:
+
+    config.vm.network "forwarded_port", guest: 8080, host: 8081
+    config.vm.network "forwarded_port", guest: 8088, host: 8089
+    config.vm.network "forwarded_port", guest: 9083, host: 9084
+    config.vm.network "forwarded_port", guest: 4040, host: 4041
+    config.vm.network "forwarded_port", guest: 18888, host: 18889
+    config.vm.network "forwarded_port", guest: 16010, host: 16011
+
+Remember to visit the adjusted ports on the running VM.
+
+3. The most useful vagrant commands:
+vagrant up      # start up the vagrant VM if not already running.
+	        # May need to provide VM's ID if there's more than one vagrant VM
+ssh vagrant     # ssh into the sole vagrant VM, else may need to provide vagrant VM's ID
+
+vagrant halt    # to shutdown the vagrant VM. Provide VM's ID if there's more than one vagrant VM.
+
+(vagrant destroy)   # to get rid of your vagrant VM. Useful if you've edited your Vagrantfile
+
+
+4. Inside the VM, /home/<USER>/vagrant-hadoop-hive-spark will be shared and mounted as /vagrant
+Remember, this is the folder containing Vagrantfile. It's easy to use the shared folder to transfer files between the VM and the actual machine that hosts it.
+
+5. Install EMACS, FIREFOX AND MAVEN on the vagrant VM:
+Start up vagrant machine ("vagrant up") and ssh into it ("ssh vagrant") if you haven't already.
+
+
+a. sudo apt-get -y install firefox
+
+b. sudo apt-get install emacs
+
+c. sudo apt-get install maven
+  (or sudo apt update
+   sudo apt install maven)
+
+Maven is needed for the commoncrawl github projects we'll be working with.
+
+
+6. Although you can edit the Vagrantfile to have emacs and maven automatically installed when the vagrant VM is created, for firefox, you're advised to install it as above.
+
+To be able to view firefox from the machine hosting the VM, use a separate terminal and run:
+   vagrant ssh -- -Y
+[or "vagrant ssh -- -Y node1", if VM ID is node1]
+
+READING ON Vagrant:
+	* Guide: https://www.vagrantup.com/intro/getting-started/index.html
+	* Common cmds: https://blog.ipswitch.com/5-vagrant-commands-you-need-to-know
+	* vagrant reload = vagrant halt + vagrant up https://www.vagrantup.com/docs/cli/reload.html
+	* https://stackoverflow.com/questions/46903623/how-to-use-firefox-ui-in-vagrant-box
+	* https://stackoverflow.com/questions/22651399/how-to-install-firefox-in-precise64-vagrant-box
+	  sudo apt-get -y install firefox
+	* vagrant install emacs: https://medium.com/@AnnaJS15/getting-started-with-virtualbox-and-vagrant-8d98aa271d2a
+
+	* hadoop conf: sudo vi /usr/local/hadoop-2.7.6/etc/hadoop/mapred-site.xml
+	* https://data-flair.training/forums/topic/mkdir-cannot-create-directory-data-name-node-is-in-safe-mode/
+
+-------------------------------------------------
+B. Create IAM role on Amazon AWS to use S3 (S3a)
+-------------------------------------------------
+CommonCrawl (CC) crawl data is stored on Amazon S3, specifically the newest version Amazon s3a which has superceded both s3 and its earlier successor s3n.
+
+In order to have access to cc crawl data, need to create an IAM role on Dr Bainbridge's Amazon AWS account and configure its profile for commoncrawl.
+
+1. Log into Dr Bainbridge's Amazon AWS account
+- In the aws management console:
+davidb@waikato.ac.nz
+lab pwd, capital R and ! (maybe g)
+
+
+2. Create a new "iam" role or user for "commoncrawl(er)" profile
+
+3. You can create the commoncrawl profile while creating the user/role, by following the instructions at https://answers.dataiku.com/1734/common-crawl-s3
+which states
+
+"Even though the bucket is public, if your AWS key does not have your full permissions (ie if it's a restricted IAM user), you need to grant explicit access to the commoncrawl bucket: attach the following policy to your IAM user"
+
+#### START POLICY IN JSON FORMAT ###
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "Stmt1503647467000",
+            "Effect": "Allow",
+            "Action": [
+                "s3:GetObject",
+                "s3:ListBucket"
+            ],
+            "Resource": [
+                "arn:aws:s3:::commoncrawl/*",
+                "arn:aws:s3:::commoncrawl"
+            ]
+        }
+    ]
+}
+#### END POLICY ###
+
+
+--------------------------------------------------------------------------
+C. Configure Spark on your vagrant VM with the AWS authentication details
+--------------------------------------------------------------------------
+Any Spark jobs run against the CommonCrawl data stored on Amazon s3a need to be able to authenticate with the AWS IAM role you created above. In order to do this, you'll want to put the Amazon AWS access key and secret key in the SPARK configuration properties file. (Instead of configuring these values in hadoop's core-site.xml, as in the latter case, the authentication details don't get copied across when distributed jobs are run to other computers in the distributed cluster that also need to know how to authenticate):
+
+1. Inside the vagrant vm:
+
+   sudo emacs /usr/local/spark-2.3.0-bin-hadoop2.7/conf/spark-defaults.conf
+   (sudo emacs $SPARK_HOME/conf/spark-defaults.conf)
+
+2. Edit the spark properties conf file to contain these 3 new properties:
+
+    spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem  
+    spark.hadoop.fs.s3a.access.key=PASTE_IAM-ROLE_ACCESSKEY_HERE
+    spark.hadoop.fs.s3a.secret.key=PASTE_IAM-ROLE_SECRETKEY_HERE
+
+Instructions on which properties to set were taken from:
+- https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark
+- https://community.cloudera.com/t5/Community-Articles/HDP-2-4-0-and-Spark-1-6-0-connecting-to-AWS-S3-buckets/ta-p/245760
+
+[NOTE, inactive alternative: Instead of editing spark's config file to set these properties, these properties can also be set in the bash script that executes the commoncrawl Spark jobs:
+
+$SPARK_HOME/bin/spark-submit \
+   ...
+   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   ...
+
+But better not to hardcode authentication details into code, so I did it the first way.
+]
+
+
+----------------------------------------------------------------------
+D. OPTIONAL? Further configuration for Hadoop to work with Amazon AWS
+----------------------------------------------------------------------
+The following 2 pages state that additional steps are necessary to get hadoop and spark to work with AWS S3a:
+
+- https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark
+- https://community.cloudera.com/t5/Community-Articles/HDP-2-4-0-and-Spark-1-6-0-connecting-to-AWS-S3-buckets/ta-p/245760
+
+I'm not sure whether these steps were really necessary in my case, and if so, whether it was A or B below that got things working for me. However, I have both A and B below set up.
+
+
+A. Check your maven installation for necessary jars:
+
+1. Installing maven may already have got the specifically recommended version of AWS-Java-SDK (aws-java-sdk-1.7.4.jar) and v2.7.6 hadoop-aws matching the vagrant VM's hadoop version (hadoop-aws-2.7.6.jar). Check these locations, as that's where I have them:
+- /home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar
+- /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+
+The specifically recommended v.1.7.4 from the instructions can be found off https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.4 at https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar
+
+2. The script that runs the 2 Spark jobs uses the above paths for one of the spark jobs:
+   $SPARK_HOME/bin/spark-submit \
+       --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+       --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+
+However the other Spark job in the script does not set --jars or --driver-class-path, despite also referring to the s3a://commoncrawl table. So I'm not sure whether the jars are necessary or whether 
+
+B. Download jar files and put them on the hadoop classpath:
+
+1. download the jar files:
+- I obtained aws-java-sdk-1.11.616.jar (v1.11) from https://aws.amazon.com/sdk-for-java/
+
+- I downloaded hadoop-aws 2.7.6 jar, as it goes with my version of hadoop, from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.6
+
+2. The easiest solution is to copy the 2 downloaded jars onto a location in the hadoop classpath.
+
+a. The command that shows the paths present on the Hadoop CLASSPATH:
+	hadoop classpath
+One of the paths this will list is /usr/local/hadoop-2.7.6/share/hadoop/common/
+
+b. SUDO COPY the 2 downloaded jar files, hadoop-aws-2.7.6.jar and aws-java-sdk-1.11.616.jar, to this location:
+
+     sudo cp hadoop-aws-2.7.6.jar /usr/local/hadoop-2.7.6/share/hadoop/common/.
+     sudo cp aws-java-sdk-1.11.616.jar /usr/local/hadoop-2.7.6/share/hadoop/common/.
+
+Any hadoop jobs run will now find these 2 jar files on the classpath.
+
+[NOTE, unused alternative: Instead of copying the 2 jar files into a system location, assuming they were downloaded into /home/vagrant/lib, you can also export a custom folder's jar files into the hadoop classpath from the bash script that runs the spark jobs. This had no effect for me, and was commented out, and is another reason why I'm not sure if the 2 jar files were even necessary.
+#export LIBJARS=/home/vagrant/lib/*                                 
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+]
+
+
+------------------------------------
+E. Setup cc-index-table git project
+------------------------------------
+Need to be inside the vagrant VM.
+
+1. Since you should have already installed maven, you can checkout and compile the cc-index-table git project.
+
+       git clone https://github.com/commoncrawl/cc-index-table.git
+
+2. Modify the toplevel pom.xml file used by maven by changing the spark version used to 2.3.0 and adding a dependency for hadoop-aws 2.7.6, as indicated below:
+
+17c17,18
+< 		<spark.version>2.4.1</spark.version>
+---
+> 		<!--<spark.version>2.4.1</spark.version>-->
+> 		<spark.version>2.3.0</spark.version>
+135a137,143
+> 		<dependency>
+> 		  <groupId>org.apache.hadoop</groupId>
+> 		  <artifactId>hadoop-aws</artifactId>
+> 		  <version>2.7.6</version>
+> 		</dependency>
+> 
+
+3. Although cc-index-table will compile successfully after the above modifications, it will nevertheless throw an exception when it's eventually run. To fix that, edit the file "cc-index-table/src/main/java/org/commoncrawl/spark/examples/CCIndexWarcExport.java" as follows:
+
+a. Set option(header) to false, since the csv file contains no header row, only data rows.
+   Change:
+	sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+                                        .load(csvQueryResult);
+   To
+	sqlDF = sparkSession.read().format("csv").option("header", false).option("inferSchema", true)
+                                        .load(csvQueryResult);
+
+b. The 4 column names are inferred as _c0 to _c3, not as url/warc_filename etc.
+   Comment out:
+	//JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+                                .toJavaRDD();
+   Replace with the default inferred column names:
+	JavaRDD<Row> rdd = sqlDF.select("_c0", "_c1", "_c2", "_c3").rdd()
+                                .toJavaRDD();
+
+// TODO: link svn committed versions of orig and modified CCIndexWarcExport.java here.
+
+4. Now (re)compile cc-index-table with the above modifications:
+
+   cd cc-index-table
+   mvn package
+
+-------------------------------
+F. Setup warc-to-wet tools
+-------------------------------
+To convert WARC files to WET (.warc.wet) files, need to checkout, set up and compile a couple more tools. These instructions are derived from those at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+1. Grab and compile the 2 git projects for converting warc to wet:
+     git clone https://github.com/commoncrawl/ia-web-commons
+     cd ia-web-commons
+     mvn install
+
+     git clone https://github.com/commoncrawl/ia-hadoop-tools
+     cd ia-hadoop-tools
+     # can't compile this yet
+     
+
+2. Add the following into ia-hadoop-tools/pom.xml, in the top of the <dependencies> element, so that maven finds an appropriate version of the org.json package and its JSONTokener (version number found off https://mvnrepository.com/artifact/org.json/json):
+
+   <dependency>
+      <groupId>org.json</groupId>
+      <artifactId>json</artifactId>
+      <version>20131018</version>
+    </dependency>
+
+[
+  UNFAMILAR CHANGES that I don't recollect making and that may have been a consequence of the change in step 1 above:
+  a. These further differences show up between the original version of the file in pom.xml.orig and the modified new pom.xml:
+  ia-hadoop-tools>diff pom.xml.orig pom.xml
+
+  <       <groupId>org.netpreserve.commons</groupId>
+  <       <artifactId>webarchive-commons</artifactId>
+  <       <version>1.1.1-SNAPSHOT</version>
+  ---
+  >       <groupId>org.commoncrawl</groupId>
+  >       <artifactId>ia-web-commons</artifactId>
+  >       <version>1.1.9-SNAPSHOT</version>
+
+  b. I don't recollect changing or manually introducing any java files. I just followed the instructions at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+  However, a diff -rq between the latest "ia-hadoop-tools" gitproject checked out a month after the "ia-hadoop-tools.orig" checkout I ran, shows the following differences in files which are not shown as recently modified in github itself in that same period.
+
+  ia-hadoop-tools> diff -rq ia-hadoop-tools ia-hadoop-tools.orig/
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/io/HDFSTouch.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/io/HDFSTouch.java differ
+  Only in ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs: ArchiveFileExtractor.java
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/CDXGenerator.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/CDXGenerator.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/JobDriver.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/JobDriver.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/WARCMetadataRecordGenerator.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/WARCMetadataRecordGenerator.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/WATGenerator.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/WATGenerator.java differ
+  Only in ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs: WEATGenerator.java
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/mapreduce/ZipNumPartitioner.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/mapreduce/ZipNumPartitioner.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/pig/ZipNumRecordReader.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/pig/ZipNumRecordReader.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/streaming/ZipNumRecordReader.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/streaming/ZipNumRecordReader.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/server/FileBackedInputStream.java and ia-hadoop-tools.orig/src/main/java/org/archive/server/FileBackedInputStream.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/server/GZRangeClient.java and ia-hadoop-tools.orig/src/main/java/org/archive/server/GZRangeClient.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/server/GZRangeServer.java and ia-hadoop-tools.orig/src/main/java/org/archive/server/GZRangeServer.java differ
+]
+
+3. Now can compile ia-hadoop-tools:
+   cd ia-hadoop-tools
+   mvn package
+
+4. Can't run it until guava.jar is on hadoop classpath. Locate a guava.jar and put it into an existing location checked for by hadoop classpath:
+
+   locate guava.jar
+       # found in /usr/share/java/guava.jar and /usr/share/maven/lib/guava.jar
+   diff /usr/share/java/guava.jar /usr/share/maven/lib/guava.jar   
+	# identical/no difference, so can use either
+   sudo cp /usr/share/java/guava.jar /usr/local/hadoop/share/hadoop/common/.
+   	# now guava.jar has been copied into a location on hadoop classpath
+
+
+Having done the above, our bash script will now be able to convert WARC to WET files when it runs:
+    $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz hdfs:///user/vagrant/PATH/TO/warc/*.warc.gz
+Our script expects a specific folder structure: there should be a "warc" folder (containing the warc files), which is supplied as above, but also an empty "wet" and "wat" folder at the same level as the "warc" folder.
+
+
+When the job is running, can visit the Spark Context at http://node1:4040/jobs/ (http://node1:4041/jobs/ for me first time, since I forwarded the vagrant VM's ports at +1. However, subsequent times it was on node1:4040/jobs?)
+
+-----------------------------------
+G. Getting and running our scripts
+-----------------------------------
+
+1. Grab our 1st bash script and put it into the /home/vagrant/cc-index-table/src/script:
+       cd cc-index-table/src/script
+       wget http://svn.greenstone.org/gs3-extensions/maori-lang-detection/bin/script/get_maori_WET_records_for_crawl.sh
+       chmod u+x get_maori_WET_records_for_crawl.sh
+
+RUN AS:
+cd cc-index-table
+./src/script/get_maori_WET_records_for_crawl.sh <crawl-timestamp>
+				where crawl-timestamp of form "CC-MAIN-YYYY-##" >= September 2019
+
+OUTPUT:
+After hours of processing (leave it to run overnight), you should end up with:
+      hdfs dfs -ls /user/vagrant/<crawl-timestamp>
+In particular, the zipped wet records at hdfs:///user/vagrant/<crawl-timestamp>/wet/
+that we want would have been copied into /vagrant/<crawl-timestamp>-wet-files/
+
+
+The script get_maori_WET_records_for_crawl.sh
+- takes a crawl timestamp of the form "CC-MAIN-YYYY-##" from Sep 2018 onwards (before which content_languages were not indexed). The legitimate crawl timestampts are listed in the first column at http://index.commoncrawl.org/
+- runs a spark job against CC's AWS bucket over s3a to create a csv table of MRI language records
+- runs a spark job to download all the WARC records from CC's AWS that are denoted by the csv file's records into zipped warc files
+- converts WARC to WET: locally converts the downloaded warc.gz files into warc.wet.gz (and warc.wat.gz) files
+
+
+2. Grab our 2nd bash script and put it into the top level of the vagrant VM (/home/vagrant):
+
+       cd /home/vagrant
+       wget http://svn.greenstone.org/gs3-extensions/maori-lang-detection/bin/script/get_Maori_WET_records_from_CCSep2018_on.sh
+       chmod u+x get_Maori_WET_records_from_CCSep2018_on.sh
+
+RUN AS:
+    ./get_Maori_WET_records_from_CCSep2018_on.sh
+
+This script just runs the 1st script cc-index-table/src/script/get_maori_WET_records_for_crawl.sh (above) to process all listed common-crawls since September 2018.
+If any fails, then the script will terminate. Else it runs against each common-crawl in sequence.
+
+NOTE: If needed, update the script with more recent crawl timestamps from http://index.commoncrawl.org/
+
+OUTPUT:
+After days of running, will end up with:
+      hdfs:///user/vagrant/<crawl-timestamp>/wet/
+for each crawl-timestamp listed in the script,
+which at present would have got copied into
+      /vagrant/<crawl-timestamp>-wet-files/
+
+Each of these output wet folders can then be processed in turn by CCWETProcessor.java from http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java
+
+
+-----------------------EOF------------------------
+
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/conf/ia-hadoop-tools-pom.xml
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/conf/ia-hadoop-tools-pom.xml	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/conf/ia-hadoop-tools-pom.xml	(revision 33527)
@@ -0,0 +1,173 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>org.archive</groupId>
+  <artifactId>ia-hadoop-tools</artifactId>
+  <version>1.0-SNAPSHOT</version>
+  <packaging>jar</packaging>
+
+  <name>ia-hadoop-tools</name>
+  <url>http://maven.apache.org</url>
+
+  <properties>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+  </properties>
+
+  <dependencies>
+    <!-- START GS TEAM INSERT -->
+    <dependency>
+      <groupId>org.json</groupId>
+      <artifactId>json</artifactId>
+      <version>20131018</version>
+    </dependency>
+    <!-- END GS TEAM INSERT -->
+    <dependency>
+      <groupId>junit</groupId>
+      <artifactId>junit</artifactId>
+      <version>4.10</version>
+      <type>jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.commoncrawl</groupId>
+      <artifactId>ia-web-commons</artifactId>
+      <version>1.1.9-SNAPSHOT</version>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.archive.access-control</groupId>
+      <artifactId>access-control</artifactId>
+      <version>0.1.0-SNAPSHOT</version>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.pig</groupId>
+      <artifactId>pig</artifactId>
+      <version>0.10.0</version>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.mortbay.jetty</groupId>
+      <artifactId>jetty</artifactId>
+      <version>6.1.26</version>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.httpcomponents</groupId>
+      <artifactId>httpclient</artifactId>
+      <version>4.1.3</version>
+      <type>jar</type>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.hbase</groupId>
+      <artifactId>hbase</artifactId>
+      <version>0.90.5</version>
+      <type>jar</type>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.cassandra</groupId>
+      <artifactId>cassandra-all</artifactId>
+      <version>2.0.2</version>
+    </dependency>
+    <dependency>
+      <groupId>com.datastax.cassandra</groupId>
+      <artifactId>cassandra-driver-core</artifactId>
+      <version>2.0.0-rc1</version>
+    </dependency>
+    <dependency>
+      <groupId>org.slf4j</groupId>
+      <artifactId>slf4j-log4j12</artifactId>
+      <version>1.7.5</version>
+    </dependency>
+    <dependency>
+      <groupId>commons-collections</groupId>
+      <artifactId>commons-collections</artifactId>
+      <version>3.1</version>
+      <scope>compile</scope>
+    </dependency>
+
+  </dependencies>
+  <build>
+    <pluginManagement>
+      <plugins>
+        <plugin>
+          <groupId>org.sonatype.plugins</groupId>
+          <artifactId>jarjar-maven-plugin</artifactId>
+          <version>1.5</version>
+        </plugin>
+      </plugins>
+    </pluginManagement>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-compiler-plugin</artifactId>
+        <version>2.3.2</version>
+        <configuration>
+          <source>1.6</source>
+          <target>1.6</target>
+        </configuration>
+      </plugin>
+      <plugin>
+        <artifactId>maven-assembly-plugin</artifactId>
+        <version>2.4</version>
+        <configuration>
+
+          <descriptorRefs>
+            <descriptorRef>jar-with-dependencies</descriptorRef>
+          </descriptorRefs>
+
+          <finalName>ia-hadoop-tools</finalName>
+          <archive>
+            <manifest>
+              <mainClass>org.archive.hadoop.jobs.JobDriver</mainClass>
+            </manifest>
+          </archive>
+        </configuration>
+        <executions>
+          <execution>
+            <phase>package</phase>
+            <goals>
+              <goal>single</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+    <resources>
+      <resource>
+        <directory>src/test/resources</directory>
+        <filtering>true</filtering>
+      </resource>
+    </resources>
+  </build>
+  <repositories>
+    <repository>
+      <id>internetarchive</id>
+      <name>Internet Archive Maven Repository</name>
+      <url>http://builds.archive.org:8080/maven2</url>
+      <layout>default</layout>
+
+      <releases>
+        <enabled>true</enabled>
+        <updatePolicy>daily</updatePolicy>
+        <checksumPolicy>warn</checksumPolicy>
+      </releases>
+      <snapshots>
+        <enabled>true</enabled>
+        <updatePolicy>daily</updatePolicy>
+        <checksumPolicy>fail</checksumPolicy>
+      </snapshots>
+    </repository>
+  </repositories>
+
+  <distributionManagement>
+    <repository>
+      <id>repository</id>
+      <!--Pass as command-line system property to maven -->
+      <url>${repository.url}</url>
+    </repository>
+  </distributionManagement>
+
+</project>
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/conf/spark-defaults.conf.in
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/conf/spark-defaults.conf.in	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/conf/spark-defaults.conf.in	(revision 33527)
@@ -0,0 +1,28 @@
+# Default system properties included when running spark-submit.
+# This is useful for setting default environmental settings.
+
+# Example:
+# spark.master                     spark://master:7077
+# spark.eventLog.enabled           true
+# spark.eventLog.dir               hdfs://namenode:8021/directory
+# spark.serializer                 org.apache.spark.serializer.KryoSerializer
+# spark.driver.memory              5g
+# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
+
+#spark.eventLog.dir=hdfs://10.211.55.101/user/spark/applicationHistory
+#spark.eventLog.dir               hdfs://node1:8021/user/spark/applicationHistory
+spark.eventLog.dir               hdfs://node1/user/spark/applicationHistory
+#spark.eventLog.dir               hdfs:///user/spark/applicationHistory
+spark.yarn.historyServer.address=10.211.55.101:18080
+spark.eventLog.enabled=true
+spark.yarn.archive	hdfs://node1/user/spark/spark-libs.jar
+
+
+## -- START GS TEAM INSERT --- ##
+# Config settings and authentication details
+# for the created IAM role for s3a data that uses the created commoncrawl policy
+# Edit the access and secret keys below:
+spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
+spark.hadoop.fs.s3a.access.key TYPE_AWS_IAM_ROLE_ACCESSKEY_HERE
+spark.hadoop.fs.s3a.secret.key TYPE_AWS_IAM_ROLE_SECRETKEY_HERE
+## -- END GS TEAM INSERT --- ##
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java	(revision 33527)
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.commoncrawl.spark.examples;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.Text;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.commoncrawl.spark.util.WarcFileOutputFormat;
+import org.jets3t.service.S3Service;
+import org.jets3t.service.ServiceException;
+import org.jets3t.service.impl.rest.httpclient.RestS3Service;
+import org.jets3t.service.model.S3Object;
+import org.jets3t.service.utils.ServiceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class CCIndexWarcExport extends CCIndexExport {
+
+	private static final Logger LOG = LoggerFactory.getLogger(CCIndexExport.class);
+
+	protected static final String COMMON_CRAWL_BUCKET = "commoncrawl";
+
+	protected int numRecordsPerWarcFile = 10000;
+	protected String warcPrefix = "COMMON-CRAWL-EXPORT";
+	protected String warcCreator;
+	protected String warcOperator;
+	protected String csvQueryResult;
+
+	protected CCIndexWarcExport() {
+		super();
+	}
+
+	@Override
+	protected void addOptions() {
+		options.getOption("query")
+				.setDescription("SQL query to select rows. Note: the result is required to contain the columns `url', "
+						+ "`warc_filename', `warc_record_offset' and `warc_record_length', make sure they're SELECTed.");
+		options.addOption(new Option(null, "csv", true, "CSV file to load WARC records by filename, offset and length."
+				+ "The CSV file must have column headers and the input columns `url', `warc_filename', "
+				+ "`warc_record_offset' and `warc_record_length' are mandatory, see also option --query. "));
+
+		options.addOption(
+				new Option(null, "numOutputPartitions", true, "repartition data to have <n> output partitions"));
+		options.addOption(new Option(null, "numRecordsPerWarcFile", true, "allow max. <n> records per WARC file. "
+				+ "This will repartition the data so that in average one partition contains not more than <n> rows. "
+				+ "Default is 10000, set to -1 to disable this option."
+				+ "\nNote: if both --numOutputPartitions and --numRecordsPerWarcFile are used, the former defines "
+				+ "the minimum number of partitions, the latter the maximum partition size."));
+
+		options.addOption(new Option(null, "warcPrefix", true, "WARC filename prefix"));
+		options.addOption(new Option(null, "warcCreator", true, "(WARC info record) creator of WARC export"));
+		options.addOption(new Option(null, "warcOperator", true, "(WARC info record) operator of WARC export"));
+	}
+
+	protected int parseOptions(String[] args, List<String> arguments) {
+
+		CommandLineParser parser = new PosixParser();
+		CommandLine cli;
+
+		try {
+			cli = parser.parse(options, args);
+			if (cli.hasOption("numRecordsPerWarcFile")) {
+				numRecordsPerWarcFile = Integer.parseInt(cli.getOptionValue("numRecordsPerWarcFile"));
+			}
+			if (cli.hasOption("warcPrefix")) {
+				warcPrefix = cli.getOptionValue("warcPrefix");
+			}
+			if (cli.hasOption("warcCreator")) {
+				warcCreator = cli.getOptionValue("warcCreator");
+			}
+			if (cli.hasOption("warcOperator")) {
+				warcOperator = cli.getOptionValue("warcOperator");
+			}
+			if (cli.hasOption("csv")) {
+				if (cli.hasOption("query")) {
+					LOG.error("Options --csv and --query are mutually exclusive.");
+					return 1;
+				}
+				csvQueryResult = cli.getOptionValue("csv");
+			}
+		} catch (ParseException e) {
+			// ignore, handled in call to super.parseOptions(...)
+		}
+
+		return super.parseOptions(args, arguments);
+	}
+
+	protected static byte[] getCCWarcRecord(S3Service s3, String filename, int offset, int length) {
+		long from = offset;
+		long to = offset + length - 1;
+		S3Object s3obj = null;
+		try {
+			s3obj = s3.getObject(COMMON_CRAWL_BUCKET, filename, null, null, null, null, from, to);
+			byte[] bytes = ServiceUtils.readInputStreamToBytes(s3obj.getDataInputStream());
+			s3obj.closeDataInputStream();
+			return bytes;
+		} catch (IOException | ServiceException e) {
+			LOG.error("Failed to fetch s3://{}/{} (bytes = {}-{}): {}", COMMON_CRAWL_BUCKET, filename, from, to, e);
+		} finally {
+			if (s3obj != null) {
+				try {
+					s3obj.closeDataInputStream();
+				} catch (IOException e) {
+				}
+			}
+		}
+		return null;
+	}
+
+	@Override
+	protected int run(String tablePath, String outputPath) {
+
+		Dataset<Row> sqlDF;
+		if (csvQueryResult != null) {
+		    // ************* START GS TEAM CHANGES 1 ************ //
+		        //sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+			sqlDF = sparkSession.read().format("csv").option("header", false).option("inferSchema", true)
+					.load(csvQueryResult);
+		    // ************** END GS TEAM CHANGES 1 ************ //
+		} else {
+			loadTable(sparkSession, tablePath, tableName);
+			sqlDF = executeQuery(sparkSession, sqlQuery);
+		}
+		sqlDF.persist();
+
+		long numRows = sqlDF.count();
+		LOG.info("Number of records/rows matched by query: {}", numRows);
+		if (numRecordsPerWarcFile > 0) {
+			int n = 1 + (int) (numRows / numRecordsPerWarcFile);
+			if (n > numOutputPartitions) {
+				numOutputPartitions = n;
+				LOG.info("Distributing {} records to {} output partitions (max. {} records per WARC file)", numRows,
+						numOutputPartitions, numRecordsPerWarcFile);
+			} else {
+				// more output partitions requested
+				LOG.info("Distributing {} records to {} output partitions", numRows, numOutputPartitions);
+			}
+		}
+		if (numOutputPartitions > 0) {
+			LOG.info("Repartitioning data to {} output partitions", numOutputPartitions);
+			sqlDF = sqlDF.repartition(numOutputPartitions);
+		}
+
+		// ************* START GS TEAM CHANGES 2 ************ //
+		//JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+		JavaRDD<Row> rdd = sqlDF.select("_c0", "_c1", "_c2", "_c3").rdd()
+				.toJavaRDD();
+		// ************** END GS TEAM CHANGES 2 ************* //
+
+		// fetch WARC content from s3://commoncrawl/ and map to paired RDD
+		//   <Text url, byte[] warc_record>
+		JavaPairRDD<Text,byte[]> res = rdd.mapPartitionsToPair((Iterator<Row> rows) -> {
+			ArrayList<scala.Tuple2<Text, byte[]>> reslist = new ArrayList<>();
+			S3Service s3 = new RestS3Service(null);
+			while (rows.hasNext()) {
+				Row row = rows.next();
+				String url = row.getString(0);
+				String filename = row.getString(1);
+				int offset = row.getInt(2);
+				int length = row.getInt(3);
+				LOG.info("Fetching WARC record {} {} {} for {}", filename, offset, length, url);
+				byte[] bytes = getCCWarcRecord(s3, filename, offset, length);
+				if (bytes != null) {
+					reslist.add(new scala.Tuple2<Text, byte[]>(new Text(url), bytes));
+				}
+			}
+			return reslist.iterator();
+		}, false);
+
+		// save data as WARC files
+		Configuration conf = sparkSession.sparkContext().hadoopConfiguration();
+		conf.set("warc.export.prefix", warcPrefix);
+		if (warcCreator != null) {
+			conf.set("warc.export.creator", warcCreator);
+		}
+		if (warcOperator != null) {
+			conf.set("warc.export.operator", warcOperator);
+		}
+		conf.set("warc.export.software",
+				getClass().getCanonicalName() + " (Spark " + sparkSession.sparkContext().version() + ")");
+		conf.set("warc.export.description", "Common Crawl WARC export from " + tablePath + " for query: " + sqlQuery);
+
+		res.saveAsNewAPIHadoopFile(outputPath, String.class, byte[].class, WarcFileOutputFormat.class, conf);
+		LOG.info("Wrote {} WARC files with {} records total", numOutputPartitions, numRows);
+		sparkStats.report();
+
+		return 0;
+	}
+
+	public static void main(String[] args) throws IOException {
+		CCIndexExport job = new CCIndexWarcExport();
+		int success = job.run(args);
+		System.exit(success);
+	}
+
+}
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java.orig
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java.orig	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java.orig	(revision 33527)
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.commoncrawl.spark.examples;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.Text;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.commoncrawl.spark.util.WarcFileOutputFormat;
+import org.jets3t.service.S3Service;
+import org.jets3t.service.ServiceException;
+import org.jets3t.service.impl.rest.httpclient.RestS3Service;
+import org.jets3t.service.model.S3Object;
+import org.jets3t.service.utils.ServiceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class CCIndexWarcExport extends CCIndexExport {
+
+	private static final Logger LOG = LoggerFactory.getLogger(CCIndexExport.class);
+
+	protected static final String COMMON_CRAWL_BUCKET = "commoncrawl";
+
+	protected int numRecordsPerWarcFile = 10000;
+	protected String warcPrefix = "COMMON-CRAWL-EXPORT";
+	protected String warcCreator;
+	protected String warcOperator;
+	protected String csvQueryResult;
+
+	protected CCIndexWarcExport() {
+		super();
+	}
+
+	@Override
+	protected void addOptions() {
+		options.getOption("query")
+				.setDescription("SQL query to select rows. Note: the result is required to contain the columns `url', "
+						+ "`warc_filename', `warc_record_offset' and `warc_record_length', make sure they're SELECTed.");
+		options.addOption(new Option(null, "csv", true, "CSV file to load WARC records by filename, offset and length."
+				+ "The CSV file must have column headers and the input columns `url', `warc_filename', "
+				+ "`warc_record_offset' and `warc_record_length' are mandatory, see also option --query. "));
+
+		options.addOption(
+				new Option(null, "numOutputPartitions", true, "repartition data to have <n> output partitions"));
+		options.addOption(new Option(null, "numRecordsPerWarcFile", true, "allow max. <n> records per WARC file. "
+				+ "This will repartition the data so that in average one partition contains not more than <n> rows. "
+				+ "Default is 10000, set to -1 to disable this option."
+				+ "\nNote: if both --numOutputPartitions and --numRecordsPerWarcFile are used, the former defines "
+				+ "the minimum number of partitions, the latter the maximum partition size."));
+
+		options.addOption(new Option(null, "warcPrefix", true, "WARC filename prefix"));
+		options.addOption(new Option(null, "warcCreator", true, "(WARC info record) creator of WARC export"));
+		options.addOption(new Option(null, "warcOperator", true, "(WARC info record) operator of WARC export"));
+	}
+
+	protected int parseOptions(String[] args, List<String> arguments) {
+
+		CommandLineParser parser = new PosixParser();
+		CommandLine cli;
+
+		try {
+			cli = parser.parse(options, args);
+			if (cli.hasOption("numRecordsPerWarcFile")) {
+				numRecordsPerWarcFile = Integer.parseInt(cli.getOptionValue("numRecordsPerWarcFile"));
+			}
+			if (cli.hasOption("warcPrefix")) {
+				warcPrefix = cli.getOptionValue("warcPrefix");
+			}
+			if (cli.hasOption("warcCreator")) {
+				warcCreator = cli.getOptionValue("warcCreator");
+			}
+			if (cli.hasOption("warcOperator")) {
+				warcOperator = cli.getOptionValue("warcOperator");
+			}
+			if (cli.hasOption("csv")) {
+				if (cli.hasOption("query")) {
+					LOG.error("Options --csv and --query are mutually exclusive.");
+					return 1;
+				}
+				csvQueryResult = cli.getOptionValue("csv");
+			}
+		} catch (ParseException e) {
+			// ignore, handled in call to super.parseOptions(...)
+		}
+
+		return super.parseOptions(args, arguments);
+	}
+
+	protected static byte[] getCCWarcRecord(S3Service s3, String filename, int offset, int length) {
+		long from = offset;
+		long to = offset + length - 1;
+		S3Object s3obj = null;
+		try {
+			s3obj = s3.getObject(COMMON_CRAWL_BUCKET, filename, null, null, null, null, from, to);
+			byte[] bytes = ServiceUtils.readInputStreamToBytes(s3obj.getDataInputStream());
+			s3obj.closeDataInputStream();
+			return bytes;
+		} catch (IOException | ServiceException e) {
+			LOG.error("Failed to fetch s3://{}/{} (bytes = {}-{}): {}", COMMON_CRAWL_BUCKET, filename, from, to, e);
+		} finally {
+			if (s3obj != null) {
+				try {
+					s3obj.closeDataInputStream();
+				} catch (IOException e) {
+				}
+			}
+		}
+		return null;
+	}
+
+	@Override
+	protected int run(String tablePath, String outputPath) {
+
+		Dataset<Row> sqlDF;
+		if (csvQueryResult != null) {
+			sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+					.load(csvQueryResult);
+		} else {
+			loadTable(sparkSession, tablePath, tableName);
+			sqlDF = executeQuery(sparkSession, sqlQuery);
+		}
+		sqlDF.persist();
+
+		long numRows = sqlDF.count();
+		LOG.info("Number of records/rows matched by query: {}", numRows);
+		if (numRecordsPerWarcFile > 0) {
+			int n = 1 + (int) (numRows / numRecordsPerWarcFile);
+			if (n > numOutputPartitions) {
+				numOutputPartitions = n;
+				LOG.info("Distributing {} records to {} output partitions (max. {} records per WARC file)", numRows,
+						numOutputPartitions, numRecordsPerWarcFile);
+			} else {
+				// more output partitions requested
+				LOG.info("Distributing {} records to {} output partitions", numRows, numOutputPartitions);
+			}
+		}
+		if (numOutputPartitions > 0) {
+			LOG.info("Repartitioning data to {} output partitions", numOutputPartitions);
+			sqlDF = sqlDF.repartition(numOutputPartitions);
+		}
+
+		JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+				.toJavaRDD();
+
+		// fetch WARC content from s3://commoncrawl/ and map to paired RDD
+		//   <Text url, byte[] warc_record>
+		JavaPairRDD<Text,byte[]> res = rdd.mapPartitionsToPair((Iterator<Row> rows) -> {
+			ArrayList<scala.Tuple2<Text, byte[]>> reslist = new ArrayList<>();
+			S3Service s3 = new RestS3Service(null);
+			while (rows.hasNext()) {
+				Row row = rows.next();
+				String url = row.getString(0);
+				String filename = row.getString(1);
+				int offset = row.getInt(2);
+				int length = row.getInt(3);
+				LOG.info("Fetching WARC record {} {} {} for {}", filename, offset, length, url);
+				byte[] bytes = getCCWarcRecord(s3, filename, offset, length);
+				if (bytes != null) {
+					reslist.add(new scala.Tuple2<Text, byte[]>(new Text(url), bytes));
+				}
+			}
+			return reslist.iterator();
+		}, false);
+
+		// save data as WARC files
+		Configuration conf = sparkSession.sparkContext().hadoopConfiguration();
+		conf.set("warc.export.prefix", warcPrefix);
+		if (warcCreator != null) {
+			conf.set("warc.export.creator", warcCreator);
+		}
+		if (warcOperator != null) {
+			conf.set("warc.export.operator", warcOperator);
+		}
+		conf.set("warc.export.software",
+				getClass().getCanonicalName() + " (Spark " + sparkSession.sparkContext().version() + ")");
+		conf.set("warc.export.description", "Common Crawl WARC export from " + tablePath + " for query: " + sqlQuery);
+
+		res.saveAsNewAPIHadoopFile(outputPath, String.class, byte[].class, WarcFileOutputFormat.class, conf);
+		LOG.info("Wrote {} WARC files with {} records total", numOutputPartitions, numRows);
+		sparkStats.report();
+
+		return 0;
+	}
+
+	public static void main(String[] args) throws IOException {
+		CCIndexExport job = new CCIndexWarcExport();
+		int success = job.run(args);
+		System.exit(success);
+	}
+
+}
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/GS_README
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/GS_README	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/GS_README	(revision 33527)
@@ -0,0 +1,11 @@
+1. Most of the scripts in this folder are based on https://github.com/commoncrawl/cc-index-table/tree/master/src/script/convert_url_index.sh
+
+to set the environment and setup SPARK etc, in order to run cc-index-table related tasks to process data that we're particularly interested in.
+
+Several of the modifications are further based on the instructions and examples at https://github.com/commoncrawl/cc-index-table
+
+2. The third/final phase of the script get_maori_WET_records_for_crawl.sh is based on the instructions at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+3. The script get_Maori_WET_records_from_CCSep2018_on.sh is merely a batch processing script.
+
+4. Script limit10_export_index.sh never really worked, but it was only scripted for testing purposes, and has no other purpose.
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_index_csv.sh
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_index_csv.sh	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_index_csv.sh	(revision 33527)
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+#DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+CRAWL_ID=$1
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-csv"
+OUTPUT_PARENTDIR=hdfs:///user/vagrant/${CRAWL_ID}
+     # or just OUTPUT_PARENTDIR=/user/vagrant/${CRAWL_ID}, since /user/vagrant is on hdfs:
+
+# https://stackoverflow.com/questions/26513861/checking-if-directory-in-hdfs-already-exists-or-not
+hdfs dfs -test -d $OUTPUT_PARENTDIR
+if [ $? == 0 ]; then
+    echo "Directory $OUTPUT_PARENTDIR already exists."        
+else
+    echo "Creating directory $OUTPUT_PARENTDIR..."    
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR
+fi
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+#    --conf spark.hadoop.fs.s3a.access.key=AKIA2EVQBWSTBJ2M4BLM \
+#    --conf spark.hadoop.fs.s3a.secret.key=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 \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+# https://www.patricia-anong.com/blog/2017/11/1/extend-vmdk-on-virtualbox
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --conf spark.sql.parquet.mergeSchema=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length   
+            FROM ccindex                                                        
+            WHERE crawl = '${CRAWL_ID}' AND subset = 'warc' AND content_languages = 'mri'" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    $OUTPUTDIR
+
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset.sh
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset.sh	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset.sh	(revision 33527)
@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+#DATA="$1"
+# Output directory
+CRAWL_ID=$1
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+#OUTPUTDIR="$2"
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-subset"
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+# $SPARK_HOME/bin/spark-submit \
+#     $SPARK_ON_YARN \
+#     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+#     --conf spark.core.connection.ack.wait.timeout=600s \
+#     --conf spark.network.timeout=120s \
+#     --conf spark.task.maxFailures=20 \
+#     --conf spark.shuffle.io.maxRetries=20 \
+#     --conf spark.shuffle.io.retryWait=60s \
+#     --conf spark.driver.memory=$DRIVER_MEM \
+#     --conf spark.executor.memory=$EXECUTOR_MEM \
+#     $SPARK_EXTRA_OPTS \
+#     --num-executors $NUM_EXECUTORS \
+#     --executor-cores $EXECUTOR_CORES \
+#     --executor-memory $EXECUTOR_MEM \
+#     --conf spark.hadoop.parquet.enable.dictionary=true \
+#     --conf spark.sql.parquet.filterPushdown=true \
+#     --conf spark.sql.parquet.mergeSchema=false \
+#     --conf spark.sql.hive.metastorePartitionPruning=true \
+#     --conf spark.hadoop.parquet.enable.summary-metadata=false \
+#     --class org.commoncrawl.spark.CCIndex2Table $_APPJAR \
+#     --outputCompression=$COMPRS \
+#     --outputFormat=$FORMAT $NESTED \
+#     --partitionBy=$PARTITION_BY \
+#     "$DATA" "$OUTPUTDIR"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+   # --conf spark.hadoop.parquet.enable.dictionary=true \
+   # --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   # --conf spark.sql.hive.metastorePartitionPruning=true \
+   # --conf spark.sql.parquet.filterPushdown=true \
+   # --conf spark.sql.parquet.mergeSchema=true \
+
+
+
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --csv hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-unzipped-csv/cc-mri.csv \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-${CRAWL_ID} \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   $OUTPUTDIR
+
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset_from_scratch.sh
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset_from_scratch.sh	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset_from_scratch.sh	(revision 33527)
@@ -0,0 +1,98 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+#DATA="$1"
+# Output directory
+CRAWL_ID=$1
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-subset"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+# $SPARK_HOME/bin/spark-submit \
+#     $SPARK_ON_YARN \
+#     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+#     --conf spark.core.connection.ack.wait.timeout=600s \
+#     --conf spark.network.timeout=120s \
+#     --conf spark.task.maxFailures=20 \
+#     --conf spark.shuffle.io.maxRetries=20 \
+#     --conf spark.shuffle.io.retryWait=60s \
+#     --conf spark.driver.memory=$DRIVER_MEM \
+#     --conf spark.executor.memory=$EXECUTOR_MEM \
+#     $SPARK_EXTRA_OPTS \
+#     --num-executors $NUM_EXECUTORS \
+#     --executor-cores $EXECUTOR_CORES \
+#     --executor-memory $EXECUTOR_MEM \
+#     --conf spark.hadoop.parquet.enable.dictionary=true \
+#     --conf spark.sql.parquet.filterPushdown=true \
+#     --conf spark.sql.parquet.mergeSchema=false \
+#     --conf spark.sql.hive.metastorePartitionPruning=true \
+#     --conf spark.hadoop.parquet.enable.summary-metadata=false \
+#     --class org.commoncrawl.spark.CCIndex2Table $_APPJAR \
+#     --outputCompression=$COMPRS \
+#     --outputFormat=$FORMAT $NESTED \
+#     --partitionBy=$PARTITION_BY \
+#     "$DATA" "$OUTPUTDIR"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+	  
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --conf spark.hadoop.parquet.enable.dictionary=true \
+   --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   --conf spark.sql.hive.metastorePartitionPruning=true \
+   --conf spark.sql.parquet.filterPushdown=true \
+   --conf spark.sql.parquet.mergeSchema=true \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --query "SELECT url, warc_filename, warc_record_offset, warc_record_length
+            FROM ccindex
+            WHERE crawl = '${CRAWL_ID}' AND subset = 'warc' AND content_languages LIKE '%mri%'" \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-${CRAWL_ID} \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   $OUTPUTDIR
+
+#   hdfs:///user/vagrant/cc-mri-subset
+#   s3://commoncrawl/cc-index/table/cc-main/warc/ \
+# Since s3:// deprecated, moved to using s3a:// now
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh	(revision 33527)
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+# crawl_ids are from http://index.commoncrawl.org/
+# We only want the crawl_ids from Sep 2018 and onwards as that's when
+# the content_languages field was included in CommonCrawl's columnar index
+
+# https://www.cyberciti.biz/faq/bash-for-loop-array/
+# (else chain commands as at https://superuser.com/questions/237072/wrapping-long-bash-commands-in-script-files)
+crawl_ids=( "CC-MAIN-2019-35" "CC-MAIN-2019-30" "CC-MAIN-2019-26" \
+      "CC-MAIN-2019-22" "CC-MAIN-2019-18" "CC-MAIN-2019-13" \
+      "CC-MAIN-2019-09" "CC-MAIN-2019-04" "CC-MAIN-2018-51" \
+      "CC-MAIN-2018-47" "CC-MAIN-2018-43" "CC-MAIN-2018-39" )
+
+for crawl_id in "${crawl_ids[@]}"
+do
+    echo "About to start off index and WARC download process for CRAWL ID: $crawl_id"
+    ./cc-index-table/src/script/get_maori_WET_records_for_crawl.sh $crawl_id
+    result=$?
+    if [ $result != 0 ]; then
+	echo "Processing common-crawl $crawl_id failed with exit value: $result"
+	echo "Will cease to process remaining cc crawls. Exitting..."
+	exit $result
+    fi
+done
+
+# Important note from
+# https://www.tldp.org/LDP/abs/html/exit-status.html
+# "When a script ends with an exit that has no parameter, the exit
+# status of the script is the exit status of the last command executed
+# in the script (previous to the exit)."
+
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh	(revision 33527)
@@ -0,0 +1,205 @@
+#!/bin/bash
+
+# This script is BASED ON the cc-index-table github project's convert_url_index script at
+# https://github.com/commoncrawl/cc-index-table/blob/master/src/script/convert_url_index.sh
+# That script is described as
+# "A Spark job converts the Common Crawl URL index files (a sharded gzipped index in CDXJ format)
+# into a table in Parquet or ORC format." (https://github.com/commoncrawl/cc-index-table)."
+# If you want to run that script, then modify its variables to have the following values before
+# running it, in order for it to work on our machine for doing analytics:
+# EXECUTOR_MEM=3g
+# EXECUTOR_CORES=2
+# NUM_EXECUTORS=2
+# DRIVER_MEM=3g
+# Since that script was copied here, as a result, a lot of such variables (like executor
+# and memory related) are unused here, as they were just copied directly across. Such unused
+# variables can probably be removed from this file.
+
+# This script was modified to do the following:
+# SQL query CommonCrawl's distributed cc-index table on Amazon S3 for the parameterised crawl timestamp
+# and get all those records for which the primary language in the content_languages field is MRI for Maori.
+# Only the WARC related fields (url, filename, offset and length fields) of each record are requested.
+# The matching records' fields are then constructed into a distributed csv file on the local hdfs system
+# A second phase then requests the warc files at those offsets and downloads them onto the local hdfs.
+# We still get zipped WARC files, but they only contain the pages of that crawl where the primary language
+# was identified as MRI.
+# A third phase converts those WARC files into WET (and WAT) files and copies these zipped files onto the
+# mounted shared space on vagrant.
+
+
+#---------------------------- START UNUSED VARIABLES---------------------------#
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+#--- Dr Bainbridge modified the above variables in the original script, convert_url_index.sh,
+# as follows in order to get that spark job to run. Not used in this script. ---#
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+#--- VARIABLES PROBABLY ALSO NOT OF USE IN THIS SCRIPT ---#
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+#---------------------------- END UNUSED VARIABLES---------------------------#
+
+
+# The crawl timestamp, of the form CC-MAIN-2019-26
+# Obtain from http://index.commoncrawl.org/
+CRAWL_ID=$1
+if [ "x$CRAWL_ID" == "x" ]; then
+    echo "No crawl timestamp provided. Should be of the form CC-MAIN-YYYY-COUNT."
+    echo "e.g. CC-MAIN-2019-26. Choose a crawl timestamp from http://index.commoncrawl.org/"
+    exit
+fi
+# Output directory
+OUTPUT_PARENTDIR=hdfs:///user/vagrant/${CRAWL_ID}
+     # or just OUTPUT_PARENTDIR=/user/vagrant/${CRAWL_ID}, since /user/vagrant is on hdfs:
+
+# https://stackoverflow.com/questions/26513861/checking-if-directory-in-hdfs-already-exists-or-not
+#hdfs dfs -test -d $OUTPUT_PARENTDIR
+#if [ $? == 0 ]; then
+if $(hdfs dfs -test -d "$OUTPUT_PARENTDIR"); then
+    echo "Directory $OUTPUT_PARENTDIR already exists."        
+else
+    echo "Creating directory $OUTPUT_PARENTDIR..."    
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR
+fi
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+# PHASE 1: querying this crawl's massive index with an SQL query that requests just the references to warc files
+# for those crawled web pages where the content_languages field's primary language is MRI (3 letter code for Maori)
+# The output is a distributed .csv file which will be stored in a "cc-mri-csv" subfolder of the $OUTPUT_PARENTDIR.
+
+#OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-csv"
+OUTPUTDIR="${OUTPUT_PARENTDIR}/cc-mri-csv"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+# https://www.patricia-anong.com/blog/2017/11/1/extend-vmdk-on-virtualbox
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --conf spark.sql.parquet.mergeSchema=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length   
+            FROM ccindex                                                        
+            WHERE crawl = '${CRAWL_ID}' AND subset = 'warc' AND content_languages = 'mri'" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    $OUTPUTDIR
+
+
+# The above generates ZIPPED part*.csv files in $OUTPUTDIR (folder cc-mri-csv).
+# First create a folder and unzip into it:
+#hdfs dfs -test -d $OUTPUT_PARENTDIR/cc-mri-unzipped-csv
+#if [ $? == 0 ]; then
+if $(hdfs dfs -test -d "$OUTPUT_PARENTDIR/cc-mri-unzipped-csv"); then
+    echo "Directory cc-mri-unzipped-csv already exists for crawl ${CRAWL_ID}."
+    echo "Assuming cc-mri.csv also exists inside $OUTPUT_PARENTDIR"
+else
+    echo "Creating directory $OUTPUT_PARENTDIR/cc-mri-unzipped-csv..."
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR/cc-mri-unzipped-csv
+
+    echo "Unzipping ${OUTPUTDIR}/part files into $OUTPUT_PARENTDIR/cc-mri-unzipped-csv/cc-mri.csv"
+    hdfs dfs -cat $OUTPUTDIR/part* | gzip -d | hdfs dfs -put - $OUTPUT_PARENTDIR/cc-mri-unzipped-csv/cc-mri.csv
+fi
+
+
+
+# PHASE 2, which uses the index of MRI warc URLs and offsets,
+# stored in the now unzipped .csv file,
+# to get all the WARC records it specifies at the specified warc offsets.
+# This will get us all the MRI pages of the commoncrawl for CRAWL_ID
+
+# Change OUTPUTDIR
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/warc"
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --csv $OUTPUT_PARENTDIR/cc-mri-unzipped-csv/cc-mri.csv \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-${CRAWL_ID} \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   $OUTPUTDIR
+
+
+# PHASE 3: convert warc files to wet files and copy the wet files into the mounted shared area
+
+#hdfs dfs -test -f $OUTPUTDIR/_SUCCESS
+#if [ $? == 0 ]; then
+if $(hdfs dfs -test -f "$OUTPUTDIR/_SUCCESS"); then
+    # ia-hadoop-tools converts warc files into wet (and wat) files but expects a particular folder structure
+    # Create the expected folder structure: a "wet" and a "wat" folder should exist
+    # at the same level as the "warc" folder ($OUTPUTDIR) that contains the warc gz files.
+    
+    echo "Creating wet (text) and wat (metadata) directories to put converted warc files into..."
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR/wet
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR/wat
+    # move the _SUCCESS file from /warc to its parent folder
+    hdfs dfs -mv $OUTPUTDIR/_SUCCESS $OUTPUT_PARENTDIR/.
+
+    # Move from the "/home/vagrant/cc-index-table" github project to the "ia-hadoop-tools" github project:
+    cd /home/vagrant/ia-hadoop-tools
+    # PWD is now ia-hadoop-tools folder
+    # To run the following, guava.jar should be on the hadoop classpath
+    # This can be achieved with:
+    # vagrant@node1:~/ia-hadoop-tools$ sudo cp /usr/share/java/guava.jar /usr/local/hadoop/share/hadoop/common/.
+    $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz $OUTPUTDIR/*.warc.gz
+
+    # Now copy the zipped wet-files into the mounted folder, so we can scp the files from there to where WETProcessor.java is
+    echo "Copying and tarring up the wet files"
+    mkdir /vagrant/${CRAWL_ID}-wet-files
+    hdfs dfs -get $OUTPUT_PARENTDIR/wet/*.warc.wet.gz /vagrant/${CRAWL_ID}-wet-files/.
+    cd /vagrant
+    #tar -cvf ${CRAWL_ID}.tar /vagrant/${CRAWL_ID}-wet-files
+    #rm -rf /vagrant/${CRAWL_ID}-wet-files
+fi
+
Index: /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/limit10_export_index.sh
===================================================================
--- /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/limit10_export_index.sh	(revision 33527)
+++ /gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/limit10_export_index.sh	(revision 33527)
@@ -0,0 +1,76 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="$2"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+#    --conf spark.hadoop.fs.s3a.access.key=AKIA2EVQBWSTBJ2M4BLM \
+#    --conf spark.hadoop.fs.s3a.secret.key=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 \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+#    --conf spark.sql.parquet.mergeSchema=true \
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT * FROM ccindex LIMIT 10" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    hdfs:///user/vagrant/cc-limit10-csv
+
