Index: gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt
===================================================================
--- gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt	(revision 33540)
+++ gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt	(revision 33541)
@@ -4,74 +4,4 @@
 https://www.quora.com/What-are-some-Web-crawler-tips-to-avoid-crawler-traps
 
-
-ALTERNATIVES TO NUTCH - looking for site mirroring capabilities
-https://anarc.at/services/archive/web/ 
-	Autistici's crawl [https://git.autistici.org/ale/crawl] needs Go:
-	https://medium.com/better-programming/install-go-1-11-on-ubuntu-18-04-16-04-lts-8c098c503c5f	
-	https://guide.freecodecamp.org/go/installing-go/ubuntu-apt-get/
-		https://medium.com/@firebitsbr/how-to-uninstall-from-the-apt-manager-uninstall-just-golang-go-from-universe-debian-ubuntu-82d6a3692cbd
-	https://tecadmin.net/install-go-on-ubuntu/ [our vagrant VMs are Ubuntu 16.04 LTS, as discovered by running the cmd "lsb_release -a"]
-https://alternativeto.net/software/apache-nutch/
-https://alternativeto.net/software/wget/
-https://github.com/ArchiveTeam/grab-site/blob/master/README.md#inspecting-warc-files-in-the-terminal
-https://github.com/ArchiveTeam/wpull
-
----
-Autistici crawl:
----
-1. Install go 1.11 by following instructions at https://medium.com/better-programming/install-go-1-11-on-ubuntu-18-04-16-04-lts-8c098c503c5f
-2. Create go environment:
-#!/bin/bash
-# environment vars for golang               
-export GOROOT=/usr/local/go
-export GOPATH=$HOME/go
-export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
-3. The https://git.autistici.org/ale/crawl/README.md instructions on installing are not very clear and don't work as is at this stage.
-
-These steps work:
-
-cd $GOPATH
-mkdir bin
-mkdir src
-cd src
-
-4. Since trying to go install the crawl url didn't work
-https://stackoverflow.com/questions/14416275/error-cant-load-package-package-my-prog-found-packages-my-prog-and-main
-
-vagrant@node2:~/go/src$
-  mkdir -p git.autistici.org/ale
-  cd git.autistici.org/ale
-  git clone https://git.autistici.org/ale/crawl.git
-
-[Now can run the install command in README.md:]
-  cd $GOPATH/src
-  go install git.autistici.org/ale/crawl/cmd/crawl
-
-Now we should have a $GOPATH/bin folder containing the "crawl" binary
-
-5. Run a crawl:
-  cd $GOPATH/bin
-  ./crawl https://www.cs.waikato.ac.nz/~davidb/
-
-which downloads the site and puts the warc file into the $GOPATH/bin folder.
-
-More options, including output folder, WARC filename pattern for huge sites so that multiple warc files created for one site follow the same pattern are all in the instructions in README.md
-
-6. To view the RAW contents of a WARC file:
-https://github.com/ArchiveTeam/grab-site/blob/master/README.md#viewing-the-content-in-your-warc-archives
-
-zless <warc-file-name>
-
-zless already installed on vagrant file
-
-
-Issues converting to Wet:
-Not the correct warc format: missing elements in header, ordering different.
-But WET is an official format, not CommonCrawl specific:
-
-https://library.stanford.edu/projects/web-archiving/research-resources/data-formats-and-apis
-WET (parsed text)
-
-WARC Encapsulated Text (WET) or parsed text consists of the extracted plaintext, delimited by archived document. Each record retains the associated URL and timestamp. Common Crawl provides details on the format and Internet Archive provides documentation on usage, though they use different names for the format.
 
 -----------
@@ -153,2 +83,20 @@
 
 
+Solution to get a working nutch2:
+get http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/vagrant-for-nutch2.tar.gz
+And follow the instructions in my README file in there.
+
+---------------------------------------------------------------------
+ALTERNATIVES TO NUTCH - looking for site mirroring capabilities
+---------------------------------------------------------------------
+=> https://anarc.at/services/archive/web/ 
+	Autistici's crawl [https://git.autistici.org/ale/crawl] needs Go:
+	https://medium.com/better-programming/install-go-1-11-on-ubuntu-18-04-16-04-lts-8c098c503c5f	
+	https://guide.freecodecamp.org/go/installing-go/ubuntu-apt-get/
+		To uninstall: https://medium.com/@firebitsbr/how-to-uninstall-from-the-apt-manager-uninstall-just-golang-go-from-universe-debian-ubuntu-82d6a3692cbd
+	https://tecadmin.net/install-go-on-ubuntu/ [our vagrant VMs are Ubuntu 16.04 LTS, as discovered by running the cmd "lsb_release -a"]
+https://alternativeto.net/software/apache-nutch/
+https://alternativeto.net/software/wget/
+https://github.com/ArchiveTeam/grab-site/blob/master/README.md#inspecting-warc-files-in-the-terminal
+https://github.com/ArchiveTeam/wpull
+
Index: gs3-extensions/maori-lang-detection/hdfs-cc-work/GS_README.TXT
===================================================================
--- gs3-extensions/maori-lang-detection/hdfs-cc-work/GS_README.TXT	(revision 33540)
+++ gs3-extensions/maori-lang-detection/hdfs-cc-work/GS_README.TXT	(revision 33541)
@@ -13,4 +13,7 @@
 F.  Setup warc-to-wet tools (git projects)
 G.  Getting and running our scripts
+---
+H. Austici's crawl - CLI to download web sites as WARCs, features basics to avoid crawler taps
+
 ----------------------------------------
 
@@ -392,4 +395,203 @@
 Each of these output wet folders can then be processed in turn by CCWETProcessor.java from http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java
 
+-----------------------------------
+H. Austici crawl
+-----------------------------------
+Austici's crawl: CLI to download web sites as WARCs, features basics to avoid crawler taps.
+
+Out of several software to do site mirroring, Autistici's "crawl" seemed promising:
+https://anarc.at/services/archive/web/
+
+- CLI.
+- Can download a website quite simply, though flags for additional settings are available.
+- Coded to prevent common traps.
+- Downloads website as WARC file
+- Now I have the WARC to WET process working for the WARC file it produced for the usual test site (Dr Bainbridge's home page)
+
+Need to have Go installed in order to install and run Autistici's crawl.
+Not a problem, because I can do it on the remote machine (which also hosts the hdfs) where I have sudo powers.
+
+INSTRUCTIONS
+
+1. Install go 1.11 by following instructions at https://medium.com/better-programming/install-go-1-11-on-ubuntu-18-04-16-04-lts-8c098c503c5f
+2. Create go environment:
+#!/bin/bash
+# environment vars for golang               
+export GOROOT=/usr/local/go
+export GOPATH=$HOME/go
+export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
+3. The https://git.autistici.org/ale/crawl/README.md instructions on installing are not very clear and don't work as is at this stage.
+
+These steps work:
+
+cd $GOPATH
+mkdir bin
+mkdir src
+cd src
+
+4. Since trying to go install the crawl url didn't work
+https://stackoverflow.com/questions/14416275/error-cant-load-package-package-my-prog-found-packages-my-prog-and-main
+
+vagrant@node2:~/go/src$
+  mkdir -p git.autistici.org/ale
+  cd git.autistici.org/ale
+  git clone https://git.autistici.org/ale/crawl.git
+
+[Now can run the install command in README.md:]
+  cd $GOPATH/src
+  go install git.autistici.org/ale/crawl/cmd/crawl
+
+Now we should have a $GOPATH/bin folder containing the "crawl" binary
+
+5. Run a crawl:
+  cd $GOPATH/bin
+  ./crawl https://www.cs.waikato.ac.nz/~davidb/
+
+which downloads the site and puts the warc file into the $GOPATH/bin folder.
+
+More options, including output folder, WARC filename pattern for huge sites so that multiple warc files created for one site follow the same pattern are all in the instructions in README.md
+
+6. To view the RAW contents of a WARC file:
+https://github.com/ArchiveTeam/grab-site/blob/master/README.md#viewing-the-content-in-your-warc-archives
+
+zless <warc-file-name>
+
+zless already installed on vagrant file
+
+
+-----------------------------------------------------------------------------------------------
+How to run warc-to-wet conversion on sites downloaded as WARCs by Austici's "crawl" 
+-----------------------------------------------------------------------------------------------
+ISSUES CONVERTING WARC to WET:
+---
+WARC files produced by Autistici crawl are of a somewhat different format to CommonCrawl WARCs.
+- missing elements in header
+- different header elements
+- ordering different (if that matters)
+
+But WET is an official format, not CommonCrawl specific, as indicated by
+
+https://library.stanford.edu/projects/web-archiving/research-resources/data-formats-and-apis
+"WET (parsed text)
+
+WARC Encapsulated Text (WET) or parsed text consists of the extracted plaintext, delimited by archived document. Each record retains the associated URL and timestamp. Common Crawl provides details on the format and Internet Archive provides documentation on usage, though they use different names for the format."
+
+So must be possible to get WARC to WET conversion used for CommonCrawl data to work on Autistici crawl's WARC files.
+
+
+RESOLUTION:
+---
+I made changes to 2 java source files in the 2 github projects ia-web-commons and ia-hadoop-tools, which we use for the WARC to WET processing of CommonCrawl data. These gitprojects (with modifications for commoncrawl) are already on http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/gitprojects.
+
+The changed files are as follows:
+1. patches/WATExtractorOutput.java
+   put into ia-web-commons/src/main/java/org/archive/extract
+   after renaming existing to .orig
+
+THEN RECOMPILE ia-web-commons with:
+   mvn install
+
+2. patches/GZRangeClient.java
+   put into ia-hadoop-tools/src/main/java/org/archive/server
+   after renaming existing to .orig
+  
+THEN RECOMPILE ia-hadoop-tools with:
+   mvn package
+
+Make sure to first compile ia-web-commons, then ia-hadoop-tools.
+
+
+The modifications made to the above 2 files are as follows:
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+1. ia-web-commons/src/main/java/org/archive/extract/WATExtractorOutput.java
+
+[diff src/main/java/org/archive/extract/WATExtractorOutput.orig src/main/java/org/archive/extract/WATExtractorOutput.java]
+
+162,163c162,163
+< 			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Filename");
+< 		} else {
+---
+> 			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Warcinfo-ID");
+> 		} else {
+
+
+2. ia-hadoop-tools/src/main/java/org/archive/server/GZRangeClient.java
+
+[diff src/main/java/org/archive/server/GZRangeClient.orig src/main/java/org/archive/server/GZRangeClient.java]
+
+76,83c76,82
+< 		"WARC/1.0\r\n" +
+< 		"WARC-Type: warcinfo\r\n" +
+< 		"WARC-Date: %s\r\n" +
+< 		"WARC-Filename: %s\r\n" +
+< 		"WARC-Record-ID: <urn:uuid:%s>\r\n" +
+< 		"Content-Type: application/warc-fields\r\n" +
+< 		"Content-Length: %d\r\n\r\n";
+< 
+---
+> 	    "WARC/1.0\r\n" +
+> 	    "Content-Type: application/warc-fields\r\n" +
+> 	    "WARC-Type: warcinfo\r\n" +
+> 	    "WARC-Warcinfo-ID: <urn:uuid:%s>\r\n" +
+> 	    "Content-Length: %d\r\n\r\n" +
+> 	    "WARC-Record-ID: <urn:uuid:%s>\r\n" +	    
+> 	    "WARC-Date: %s\r\n";
+115,119c114,119
+< 	private static String DEFAULT_WARC_PATTERN = "software: %s Extractor\r\n" +
+< 	"format: WARC File Format 1.0\r\n" +
+< 	"conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf\r\n" +
+< 	"publisher: Internet Archive\r\n" +
+< 	"created: %s\r\n\r\n";
+---
+> 	private static String DEFAULT_WARC_PATTERN = "Software: crawl/1.0\r\n" +
+> 	"Format: WARC File Format 1.0\r\n" +
+> 	    "Conformsto: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf\r\n\r\n";
+>     // +
+>     //"publisher: Internet Archive\r\n" +
+>     //"created: %s\r\n\r\n";
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+
+3. To run WARC to WET, the warc needs to live on hdfs in a warc folder and there should be wet and wat folders at the same level.
+
+For example, assume that running Autistici's crawl generated $GOPATH/bin/crawl.warc.gz
+(default location and filename unless you pass flags to crawl CLI to control these)
+
+a. Ensure you get crawl.warc.gz onto the vagrant VM with the WARC to WET git projects installed, recompiled with the above modifications.
+
+b. Now, create the folder structure needed for warc-to-wet conversion:
+   hdfs dfs -mkdir /user/vagrant/warctest
+   hdfs dfs -mkdir /user/vagrant/warctest/warc
+   hdfs dfs -mkdir /user/vagrant/warctest/wet
+   hdfs dfs -mkdir /user/vagrant/warctest/wat
+
+c. Put crawl.warc.gz into the warc folder on hfds:
+   hdfs dfs -put crawl.warc.gz /user/vagrant/warctest/warc/.
+
+d. Finally, time to run the actual warc-to-wet conversion from ia-hadoop-tools:
+   cd ia-hadoop-tools
+   WARC_FOLDER=/user/vagrant/warctest/warc
+   $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz $WARC_FOLDER/crawl*.warc.gz
+
+More meaningful when the WARC_FOLDER contains multiple *.warc.gz files,
+as the above will use map-reduce to generate the *.warc.wet.gz files in the output wet folder.
+
+e. Copy the generated wet files across from /user/vagrant/warctest/wet/:
+
+   (cd /vagrant or else
+   cd /home/vagrant
+   )
+   hdfs dfs -get /user/vagrant/warctest/wet/crawl.warc.wet.gz .
+
+or, when dealing with multiple input warc files, we'll have multiple wet files:
+    hdfs dfs -get /user/vagrant/warctest/wet/*.warc.wet.gz
+
+
+f. Now can view the contents of the WET files to confirm they are what we want:
+   gunzip crawl.warc.wet.gz
+   zless crawl.warc.wet
+
+The wet file contents should look good now: the web pages as WET records without html tags.
+
 
 -----------------------EOF------------------------
Index: gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/GZRangeClient.java
===================================================================
--- gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/GZRangeClient.java	(revision 33541)
+++ gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/GZRangeClient.java	(revision 33541)
@@ -0,0 +1,417 @@
+package org.archive.server;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.net.URLConnection;
+import java.nio.charset.Charset;
+import java.text.ParseException;
+import java.util.List;
+import java.util.UUID;
+import java.util.logging.Logger;
+import java.net.URI;
+import java.net.URISyntaxException;
+
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.archive.util.HMACSigner;
+import org.archive.format.gzip.GZIPFormatException;
+import org.archive.format.gzip.GZIPMemberSeries;
+import org.archive.format.gzip.GZIPMemberWriter;
+import org.archive.format.gzip.GZIPSeriesMember;
+import org.archive.streamcontext.SimpleStream;
+import org.archive.util.IAUtils;
+import org.archive.util.DateUtils;
+import org.archive.util.FileNameSpec;
+import com.google.common.io.ByteStreams;
+
+public class GZRangeClient {
+
+	private final static Logger LOGGER = 
+		Logger.getLogger(GZRangeClient.class.getName());
+
+	private static final Charset UTF8 = Charset.forName("UTF-8");
+	private static int CR = 13;
+	private static int LF = 10;
+	private static final long DEFAULT_MAX_ARC_SIZE = 1024 * 1024 * 100;
+	private static final long DEFAULT_MAX_WARC_SIZE = 1024 * 1024 * 1024;
+
+	private File targetDir;
+	private long maxArcSize;
+	private long maxWarcSize;
+	private String timestamp14;
+	private String timestampZ;
+	private FileNameSpec warcNamer;
+	private FileNameSpec arcNamer;
+	private File currentArc;
+	private File currentArcTmp;
+
+	private String hmacName = "";
+	private String hmacSignature = "";
+
+	protected static FileSystem hdfsSys = null;
+
+	private File currentWarc;
+	private File currentWarcTmp;
+
+	private FileOutputStream currentArcOS;
+	private long currentArcSize = 0;
+	private FileOutputStream currentWarcOS;
+	private long currentWarcSize = 0;
+	private byte[] warcHeaderContents;
+	private boolean exitOnError = false;
+
+	private final static String ARC_PATTERN = 
+		"filedesc://%s 0.0.0.0 %s text/plain 76\n" +
+		"1 0 InternetArchive\n" +
+		"URL IP-address Archive-date Content-type Archive-length\n\n";
+
+	private final static String WARC_PATTERN =
+	    "WARC/1.0\r\n" +
+	    "Content-Type: application/warc-fields\r\n" +
+	    "WARC-Type: warcinfo\r\n" +
+	    "WARC-Warcinfo-ID: <urn:uuid:%s>\r\n" +
+	    "Content-Length: %d\r\n\r\n" +
+	    "WARC-Record-ID: <urn:uuid:%s>\r\n" +	    
+	    "WARC-Date: %s\r\n";
+
+	/*
+filedesc://IQ-125-20061126082604-03075-crawling08.us.archive.org.arc 0.0.0.0 20061126082604 text/plain 1447
+1 1 InternetArchive
+URL IP-address Archive-date Content-type Archive-length
+
+
+
+WARC-Date: 2009-10-10T21:33:10Z
+WARC-Filename: LOC-WEEKLY-008-20091010213310-06162-crawling110.us.archive.org.warc.gz
+WARC-Record-ID: <urn:uuid:776a760b-f456-48f1-97e3-1d29967c75d2>
+Content-Type: application/warc-fields
+Content-Length: 599
+
+software: Heritrix/1.15.4 http://crawler.archive.org
+ip: 207.241.235.29
+hostname: crawling110.us.archive.org
+format: WARC File Format 0.17
+conformsTo: http://crawler.archive.org/warc/0.17/WARC0.17ISO.doc
+operator: Vinay Goel
+publisher: Internet Archive
+audience: Library of Congress
+isPartOf: LOC-WEEKLY-008-RECOVER
+created: 2009-09-30T12:21:49Z
+description: Library of Congress Monthly Harvest
+robots: ignore
+http-header-user-agent: Mozilla/5.0 (compatible; archive.org_bot/1.5.0 +http://www.loc.gov/minerva/crawl.html)
+http-header-from: archive-crawler-agent@lists.sourceforge.net
+
+	 */
+
+	private static String DEFAULT_WARC_PATTERN = "Software: crawl/1.0\r\n" +
+	"Format: WARC File Format 1.0\r\n" +
+	    "Conformsto: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf\r\n\r\n";
+    // +
+    //"publisher: Internet Archive\r\n" +
+    //"created: %s\r\n\r\n";
+
+	private static final String defaultWarcHeaderString = String.format(
+			DEFAULT_WARC_PATTERN, 
+			IAUtils.COMMONS_VERSION, 
+			DateUtils.getLog17Date(System.currentTimeMillis()));
+
+	private static final byte[] DEFAULT_WARC_HEADER_BYTES = 
+		defaultWarcHeaderString.getBytes(UTF8);
+
+	public GZRangeClient(File targetDir, String prefix, String timestamp14)
+	throws ParseException {
+
+		this.targetDir = targetDir;
+		arcNamer = new FileNameSpec(prefix, ".arc.gz");
+		warcNamer = new FileNameSpec(prefix, ".warc.gz");
+		this.timestamp14 = timestamp14;
+		long msse = DateUtils.parse14DigitDate(timestamp14).getTime();
+		this.timestampZ = DateUtils.getLog17Date(msse);
+		maxArcSize = DEFAULT_MAX_ARC_SIZE;
+		maxWarcSize = DEFAULT_MAX_WARC_SIZE;
+		warcHeaderContents = DEFAULT_WARC_HEADER_BYTES;
+	}
+	
+	public GZRangeClient(File targetDir, String prefix, String timestamp14, String hmacName, String hmacSignature)
+	throws ParseException {
+
+		this.targetDir = targetDir;
+		arcNamer = new FileNameSpec(prefix, ".arc.gz");
+		warcNamer = new FileNameSpec(prefix, ".warc.gz");
+		this.timestamp14 = timestamp14;
+		long msse = DateUtils.parse14DigitDate(timestamp14).getTime();
+		this.timestampZ = DateUtils.getLog17Date(msse);
+		maxArcSize = DEFAULT_MAX_ARC_SIZE;
+		maxWarcSize = DEFAULT_MAX_WARC_SIZE;
+		warcHeaderContents = DEFAULT_WARC_HEADER_BYTES;
+		this.hmacName = hmacName;
+		this.hmacSignature = hmacSignature;
+	}
+
+	public void finish() throws IOException {
+		closeArc();
+		closeWarc();
+	}
+
+	private long getGZLength(InputStream is) 
+	throws IOException, GZIPFormatException {
+
+		SimpleStream s = new SimpleStream(is);
+		GZIPMemberSeries gzs = new GZIPMemberSeries(s,"range",0,true);
+		GZIPSeriesMember m = gzs.getNextMember();
+		m.skipMember();
+		return m.getCompressedBytesRead();
+	}
+
+	public void append(long offset, List<String> urls) throws IOException {
+		boolean isArc = false;
+		String first = urls.get(0);
+		if(first.endsWith(".arc.gz")) {
+			isArc = true;
+		} else if(first.endsWith(".warc.gz")) {
+
+		} else {
+			throw new IOException("URL (" + first +
+					") must end with '.arc.gz' or '.warc.gz'");
+		}
+		
+		HMACSigner signer = null;
+		if(hmacName != null && hmacSignature != null && !hmacName.isEmpty() && !hmacSignature.isEmpty())
+			signer = new HMACSigner(hmacSignature, hmacName);
+		
+		for(String url : urls) {
+			FileBackedInputStream fbis = null;
+			InputStream is = null;
+			try {
+				if(url.startsWith("http://")) {
+					URL u = new URL(url);
+					URLConnection conn = u.openConnection();
+					conn.setRequestProperty("Range", String.format("bytes=%d-", offset));
+					if(signer != null)	
+						conn.setRequestProperty("Cookie", signer.getHMacCookieStr(1000));
+					LOGGER.info(String.format("Attempting(%d) from(%s)",offset,url));
+					conn.connect();
+					is = conn.getInputStream();
+				} else if(url.startsWith("hdfs://")){
+					URI u = new URI(url);
+					//only initialize the FS once
+					if (hdfsSys == null) {
+						Configuration conf = new Configuration();
+						URI defaultURI = new URI(u.getScheme() + "://" + u.getHost() + ":"+ u.getPort() + "/");
+						hdfsSys = FileSystem.get(defaultURI, conf);
+					}
+					Path path = new Path(u.getPath());
+					FSDataInputStream fis = hdfsSys.open(path);
+					fis.seek(offset);
+					is = fis;
+				}
+				fbis = new FileBackedInputStream(is);
+				long length = getGZLength(fbis);
+				InputStream orig = fbis.getInputStream();
+				if(isArc) {
+					writeARCRecord(orig, length);
+				} else {
+					writeWARCRecord(orig, length);					
+				}
+				LOGGER.info(String.format("Wrote record(%d) from(%s)",
+						offset,url));
+				return;
+			} catch (IOException e) {
+				LOGGER.warning("FAILED URL-OFFSET("+url+")(" + offset+")");
+			} catch (URISyntaxException e) { 
+				LOGGER.warning("FAILED URL-OFFSET("+url+")(" + offset+")");
+			} finally {
+				if(is != null) {
+					is.close();
+				}
+				if(fbis != null) {
+					fbis.resetBacker();
+				}
+			}
+		}
+		StringBuilder sb = new StringBuilder();
+		for(String u : urls) {
+			if(sb.length() != 0) {
+				sb.append(",");
+			}
+			sb.append(u);
+		}
+		String errMsg = String.format("Unable to get offset(%d) from (%s)",
+				offset,sb.toString());
+		if(exitOnError) {
+			throw new IOException(errMsg);
+		} else {
+			LOGGER.severe(errMsg);
+		}
+	}
+
+
+
+	private String getWARCRecordID() {
+		return "urn:uuid:" + UUID.randomUUID().toString();
+	}
+	private byte[] getARCHeader(String name) {
+		return String.format(ARC_PATTERN,name,timestamp14).getBytes(UTF8);
+	}
+	private byte[] getWARCHeader(String name) {
+		String t = String.format(WARC_PATTERN,
+				timestampZ,name,getWARCRecordID(),warcHeaderContents.length + 4);
+		byte[] b = t.getBytes(UTF8);
+		ByteArrayOutputStream baos = new ByteArrayOutputStream();
+		try {
+			baos.write(b);
+			baos.write(warcHeaderContents);
+		} catch(IOException e) {
+			// not gonna happen
+		}
+		baos.write(CR);
+		baos.write(LF);
+		baos.write(CR);
+		baos.write(LF);
+		return baos.toByteArray();
+	}
+
+	private void writeWARCRecord(InputStream is, long length) throws IOException {
+		if(currentWarcSize == 0) {
+			nextWarc();
+		}
+		InputStream lis = ByteStreams.limit(is, length);
+		ByteStreams.copy(lis, currentWarcOS);
+		currentWarcSize += length;
+		if(currentWarcSize > maxWarcSize) {
+			closeWarc();
+		}
+	}
+	private void writeARCRecord(InputStream is, long length) throws IOException {
+		if(currentArcSize == 0) {
+			nextArc();
+		}
+		InputStream lis = ByteStreams.limit(is, length);
+		ByteStreams.copy(lis, currentArcOS);
+		currentArcSize += length;
+		if(currentArcSize > maxArcSize) {
+			closeArc();
+		}
+	}
+
+	private void closeArc() throws IOException {
+		if(currentArcOS == null) {
+			return;
+		}
+		currentArcOS.close();
+		if(!currentArcTmp.renameTo(currentArc)) {
+			throw new IOException(String.format("Failed rename(%s) to (%s)",
+					currentArcTmp.getAbsolutePath(),
+					currentArc.getAbsolutePath()));
+		}
+		currentArcOS = null;
+		currentArcSize = 0;
+		LOGGER.info(String.format("Closed(%s)",currentArc.getAbsolutePath()));
+	}
+	private void closeWarc() throws IOException {
+		if(currentWarcOS == null) {
+			return;
+		}
+		currentWarcOS.close();
+		if(!currentWarcTmp.renameTo(currentWarc)) {
+			throw new IOException(String.format("Failed rename(%s) to (%s)",
+					currentWarcTmp.getAbsolutePath(),
+					currentWarc.getAbsolutePath()));
+		}
+		currentWarcOS = null;
+		currentWarcSize = 0;
+		LOGGER.info(String.format("Closed(%s)",currentWarc.getAbsolutePath()));
+	}
+	private void nextArc() throws IOException {
+		String newArcName = arcNamer.getNextName();
+		currentArc = new File(targetDir,newArcName);
+		String tmpArcName = newArcName + ".OPEN";
+		currentArcTmp = new File(targetDir,tmpArcName);
+		currentArcOS = new FileOutputStream(currentArcTmp);
+		byte[] header = getARCHeader(newArcName);
+		GZIPMemberWriter w = new GZIPMemberWriter(currentArcOS);
+                w.write(new ByteArrayInputStream(header));
+		currentArcSize = w.getBytesWritten();
+		LOGGER.info(String.format("Openned(%s)",currentArc.getAbsolutePath()));
+	}
+
+	private void nextWarc() throws IOException {
+		String newWarcName = warcNamer.getNextName();
+		currentWarc = new File(targetDir,newWarcName);
+		String tmpWarcName = newWarcName + ".OPEN";
+		currentWarcTmp = new File(targetDir,tmpWarcName);
+		currentWarcOS = new FileOutputStream(currentWarcTmp);
+
+		byte[] header = getWARCHeader(newWarcName);
+		GZIPMemberWriter w = new GZIPMemberWriter(currentWarcOS);
+                w.write(new ByteArrayInputStream(header));
+		currentWarcSize = w.getBytesWritten();
+		LOGGER.info(String.format("Openned(%s)",currentWarc.getAbsolutePath()));
+	}
+
+	/**
+	 * @return the warcHeaderContents
+	 */
+	public byte[] getWarcHeaderContents() {
+		return warcHeaderContents;
+	}
+
+	/**
+	 * @param warcHeaderContents the warcHeaderContents to set
+	 */
+	public void setWarcHeaderContents(byte[] warcHeaderContents) {
+		this.warcHeaderContents = warcHeaderContents;
+	}
+
+	/**
+	 * @return the maxArcSize
+	 */
+	public long getMaxArcSize() {
+		return maxArcSize;
+	}
+
+	/**
+	 * @param maxArcSize the maxArcSize to set
+	 */
+	public void setMaxArcSize(long maxArcSize) {
+		this.maxArcSize = maxArcSize;
+	}
+
+	/**
+	 * @return the maxWarcSize
+	 */
+	public long getMaxWarcSize() {
+		return maxWarcSize;
+	}
+
+	/**
+	 * @param maxWarcSize the maxWarcSize to set
+	 */
+	public void setMaxWarcSize(long maxWarcSize) {
+		this.maxWarcSize = maxWarcSize;
+	}
+
+
+	/**
+	 * @return the exitOnError
+	 */
+	public boolean isExitOnError() {
+		return exitOnError;
+	}
+
+
+	/**
+	 * @param exitOnError the exitOnError to set
+	 */
+	public void setExitOnError(boolean exitOnError) {
+		this.exitOnError = exitOnError;
+	}
+}
Index: gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/WATExtractorOutput.java
===================================================================
--- gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/WATExtractorOutput.java	(revision 33541)
+++ gs3-extensions/maori-lang-detection/hdfs-cc-work/patches/WATExtractorOutput.java	(revision 33541)
@@ -0,0 +1,211 @@
+package org.archive.extract;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.nio.charset.Charset;
+import java.text.ParseException;
+import java.net.UnknownHostException;
+import java.util.Date;
+
+import org.archive.format.gzip.GZIPMemberWriter;
+import org.archive.format.gzip.GZIPMemberWriterCommittedOutputStream;
+import org.archive.format.http.HttpHeaders;
+import org.archive.format.json.JSONUtils;
+import org.archive.format.warc.WARCRecordWriter;
+import org.archive.resource.MetaData;
+import org.archive.resource.Resource;
+import org.archive.util.IAUtils;
+import org.archive.util.DateUtils;
+import org.archive.util.StreamCopy;
+import org.archive.util.io.CommitedOutputStream;
+import com.github.openjson.JSONException;
+
+import java.net.InetAddress;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+
+import java.util.logging.Logger;
+
+public class WATExtractorOutput implements ExtractorOutput {
+	WARCRecordWriter recW;
+	private boolean wroteFirst;
+	private GZIPMemberWriter gzW;
+	private static int DEFAULT_BUFFER_RAM = 1024 * 1024;
+	private int bufferRAM = DEFAULT_BUFFER_RAM;
+	private final static Charset UTF8 = Charset.forName("UTF-8");
+	private String outputFile;
+	
+	private static final Logger LOG = Logger.getLogger(WATExtractorOutput.class.getName());
+	
+	public WATExtractorOutput(OutputStream out) {
+		this(out, null);
+	}
+
+	public WATExtractorOutput(OutputStream out, String outputFile) {
+		gzW = new GZIPMemberWriter(out);
+		recW = new WARCRecordWriter();
+		wroteFirst = false;
+		this.outputFile = outputFile;
+	}
+
+	private CommitedOutputStream getOutput() {
+		return new GZIPMemberWriterCommittedOutputStream(gzW,bufferRAM);
+	}
+
+	public void output(Resource resource) throws IOException {
+		StreamCopy.readToEOF(resource.getInputStream());
+		MetaData top = resource.getMetaData().getTopMetaData();
+		CommitedOutputStream cos;
+		if(!wroteFirst) {
+			cos = getOutput();
+			writeWARCInfo(cos,top);
+			cos.commit();
+			wroteFirst = true;
+		}
+		String envelopeFormat = JSONUtils.extractSingle(top, "Envelope.Format");
+		if(envelopeFormat == null) {
+			// hrm...
+			throw new IOException("Missing Envelope.Format");
+		}
+
+		// remove the text extracts if it exists
+		JSONUtils.removeObject(top, "Envelope.Payload-Metadata.HTTP-Response-Metadata.HTML-Metadata", "Text");
+
+		cos = getOutput();
+		if(envelopeFormat.startsWith("ARC")) {
+			writeARC(cos,top);
+		} else if(envelopeFormat.startsWith("WARC")) {
+			writeWARC(cos,top);
+		} else {
+			// hrm...
+			throw new IOException("Unknown Envelope.Format");
+		}
+		cos.commit();
+	}
+
+	private void writeWARCInfo(OutputStream recOut, MetaData md) throws IOException {
+		// filename is given in the command line
+		String filename = outputFile;
+		if (filename == null || filename.length() == 0) {
+			// if no filename by command line, we construct a default filename base on container filename
+			filename = JSONUtils.extractSingle(md, "Container.Filename");
+			if (filename == null) {
+				throw new IOException("No Container.Filename...");
+			}
+			if (filename.endsWith(".warc") || filename.endsWith(".warc.gz")) {
+				filename = filename.replaceFirst("\\.warc$", ".warc.wat.gz");
+				filename = filename.replaceFirst("\\.warc\\.gz$", ".warc.wat.gz");
+			} else if (filename.endsWith(".arc") || filename.endsWith(".arc.gz")) {
+				filename = filename.replaceFirst("\\.arc$", ".arc.wat.gz");
+				filename = filename.replaceFirst("\\.arc\\.gz$", ".arc.wat.gz");
+			}
+		}
+		// removing path from filename
+		File tmpFile = new File(filename);
+		filename = tmpFile.getName();
+		HttpHeaders headers = new HttpHeaders();
+		headers.add("Software-Info", IAUtils.COMMONS_VERSION);
+		headers.addDateHeader("Extracted-Date", new Date());
+
+		// add ip, hostname
+		try {
+			InetAddress host = InetAddress.getLocalHost();
+			headers.add("ip", host.getHostAddress());
+			headers.add("hostname", host.getCanonicalHostName());
+		} catch (UnknownHostException e) {
+			LOG.warning("unable to obtain local crawl engine host :\n"+e.getMessage());
+        }
+
+		headers.add("format", IAUtils.WARC_FORMAT);
+		headers.add("conformsTo", IAUtils.WARC_FORMAT_CONFORMS_TO);
+		// optional arguments
+		if(IAUtils.OPERATOR != null && IAUtils.OPERATOR.length() > 0) {
+			headers.add("operator", IAUtils.OPERATOR);
+		}
+		if(IAUtils.PUBLISHER != null && IAUtils.PUBLISHER.length() > 0) {
+			headers.add("publisher", IAUtils.PUBLISHER);
+		}
+		if(IAUtils.WAT_WARCINFO_DESCRIPTION != null && IAUtils.WAT_WARCINFO_DESCRIPTION.length() > 0) {
+			headers.add("description", IAUtils.WAT_WARCINFO_DESCRIPTION);
+		}
+		
+		ByteArrayOutputStream baos = new ByteArrayOutputStream();
+		headers.write(baos);
+                recW.writeWARCInfoRecord(recOut,filename,baos.toByteArray());
+	}
+
+	private String extractOrIO(MetaData md, String path) throws IOException {
+		String value = JSONUtils.extractSingle(md, path);
+		if(value == null) {
+			throw new IOException("No "+path+" found.");
+		}
+		return value;
+	}
+
+	private void writeARC(OutputStream recOut, MetaData md) throws IOException {
+		String targetURI = extractOrIO(md, "Envelope.ARC-Header-Metadata.Target-URI");
+		String capDateString = extractOrIO(md, "Envelope.ARC-Header-Metadata.Date");
+		String filename = extractOrIO(md, "Container.Filename");
+		String offset = extractOrIO(md, "Container.Offset");
+		String recId = String.format("<urn:arc:%s:%s>",filename,offset);
+		writeWARCMDRecord(recOut,md,targetURI,capDateString,recId);
+	}
+
+	private void writeWARC(OutputStream recOut, MetaData md) throws IOException {
+		String warcType = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Type");
+		String targetURI;
+		if(warcType.equals("warcinfo")) {
+			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Warcinfo-ID");
+		} else {		
+			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Target-URI");
+		}
+		// handle date of generation in WARC format
+		DateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");
+		String capDateString = dateFormat.format(new Date());
+		String recId = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Record-ID");
+		writeWARCMDRecord(recOut,md,targetURI,capDateString,recId);
+	}
+
+	private void writeWARCMDRecord(OutputStream recOut, MetaData md, 
+			String targetURI, String capDateString, String recId)
+	throws IOException {
+
+                ByteArrayOutputStream bos = new ByteArrayOutputStream();
+
+		OutputStreamWriter osw = new OutputStreamWriter(bos, UTF8);
+		String contents = md.toString();
+		osw.write(contents, 0, contents.length());
+		osw.flush();
+//		ByteArrayInputStream bais = new ByteArrayInputStream(md.toString().getBytes("UTF-8"));
+		Date capDate;
+		try {
+			capDate = DateUtils.getSecondsSinceEpoch(capDateString);
+
+		} catch (ParseException e) {
+			e.printStackTrace();
+			// TODO... not the write thing...
+			capDate = new Date();
+		}
+		
+		recW.writeJSONMetadataRecord(recOut, bos.toByteArray(),
+				targetURI, capDate, recId);
+	}
+
+	private static String transformWARCDate(final String input) {
+		
+		StringBuilder output = new StringBuilder(14);
+		
+		output.append(input.substring(0,4));
+		output.append(input.substring(5,7));
+		output.append(input.substring(8,10));
+		output.append(input.substring(11,13));
+		output.append(input.substring(14,16));
+		output.append(input.substring(17,19));
+		
+		return output.toString();
+	}
+}
