Index: /other-projects/maori-lang-detection/MoreReading/CommonCrawl.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/CommonCrawl.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/MoreReading/CommonCrawl.txt	(revision 33635)
@@ -0,0 +1,297 @@
+WARC, WET, WAT FILES
+https://pypi.org/project/warc3-wet/
+https://gist.github.com/Smerity/afe7430fdb4371015466
+https://github.com/commoncrawl/commoncrawl/issues/11
+WARC TO WET:
+https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+Sebastian Nagel 	
+05/07/2017
+Hi,
+
+unfortunately, we do not provide (yet) text extracts of the CC-NEWS archives.
+
+But it's easy to run the WET extractor on the WARC files, see:
+  https://groups.google.com/d/topic/common-crawl/imv4hlLob4s/discussion
+  https://groups.google.com/d/topic/common-crawl/b6yDG7EmnhM/discussion
+
+That's what you have to do:
+
+# download the WARC files and place them in a directory "warc/"
+# create sibling folders wat and wet
+# |
+# |-- warc/
+# |   |-- CC-NEWS-20161001224340-00008.warc.gz
+# |   |-- CC-NEWS-20161017145313-00000.warc.gz
+# |   `-- ...
+# |
+# |-- wat/
+# |
+# `-- wet/
+
+git clone https://github.com/commoncrawl/ia-web-commons
+cd ia-web-commons
+mvn install
+
+cd ..
+git clone https://github.com/commoncrawl/ia-hadoop-tools
+cd ia-hadoop-tools
+mvn package
+
+java -jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator \
+   -strictMode -skipExisting batch-id-xyz .../warc/*.warc.gz
+
+The folders wat/ and wet/ will then contain the exports.
+
+Best,
+Sebastian
+
+====================
+wharariki:[239]/Scratch/ak19/gs3-extensions/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.WETProcessor ../tmp/processWET /Scratch/ak19/gs3-extensions/maori-lang-detection/tmp/processedWET
+
+wharariki:[188]/Scratch/ak19/gs3-extensions/maori-lang-detection/tmp/processedWET>ls keep | wc
+   4090    4090   65440
+wharariki:[189]/Scratch/ak19/gs3-extensions/maori-lang-detection/tmp/processedWET>ls discard | wc
+   1515    1515   24240
+
+We keep 4090 WET records and are discarding 1515.
+
+=======================
+Latest version of the index's schema:
+https://commoncrawl.s3.amazonaws.com/cc-index/table/cc-main/index.html
+
+
+At http://commoncrawl.org/2018/10/september-2018-crawl-archive-now-available/, it says
+"The September crawl contains 500 million new URLs, not contained in any crawl archive before. New URLs stem from
+
+    the continued seed donation of URLs from mixnode.com
+    ..."
+
+https://www.mixnode.com/
+"The entire web, in your hands
+
+Mixnode turns the web into a database that you can run queries against. Say goodbye to web crawling, forget about web scraping, never run a spider again: get all the web data that you need using simple SQL queries."
+
+--------------
+https://commoncrawl.github.io/cc-crawl-statistics/plots/languages
+http://commoncrawl.org/2018/08/august-2018-crawl-archive-now-available/
+
+	The JSON for the index files (that we downloaded for .nz) already contained a "languages:" field. The above page mentions that this shows the primary, upto 3, detected languages of the document.
+
+"Language Annotations
+
+We now run the Compact Language Detector 2 (CLD2) on HTML pages to identify the language of a document. CLD2 is able to identify 160 different languages and up to 3 languages per document. The detected languages resp. the ISO-639-3 code are shown in the URL index as a new field, e.g., "languages": "zho,eng". The WARC metadata records contain the full CLD2 response including scores and text coverage:
+
+languages-cld2: {"reliable":true,"text-bytes":3783,"languages":[{"code":"zh","code-iso-639-3":"zho","text-covered":0.93,"score":1943.0,"name":"Chinese"},{"code":"en","code-iso-639-3":"eng","text-covered":0.05,"score":523.0,"name":"ENGLISH"}]}
+
+On github youâll find the Java bindings to the CLD2 native library and the distribution of the primary document languages as part of our crawl statistics.
+
+Please note that the columnar index does not contain the detected languages for now. "
+
+	
+http://commoncrawl.org/2018/10/september-2018-crawl-archive-now-available/
+"the columnar index contains the content language of a web page as a new field. Please read the instructions below how to upgrade your tools to read newly added fields."
+
+http://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/
+
+SPARK (Spark SQL): https://github.com/commoncrawl/cc-index-table
+	with example on selecting languages
+https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-30/indexes/cluster.idx
+
+./convert_url_index.sh https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-30/indexes/cdx-00000.gz hdfs:///user/vagrant/cc-index-table
+---
+
+https://www.aclweb.org/anthology/L16-1443 (2016, as per https://pbn.nauka.gov.pl/sedno-webapp/getReport/38108)
+
+https://dkpro.github.io/dkpro-c4corpus/
+"DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate removal, language detection, and near-duplicate removal."
+
+https://zoidberg.ukp.informatik.tu-darmstadt.de/jenkins/job/DKPro%20C4Corpus/org.dkpro.c4corpus$dkpro-c4corpus-doc/doclinks/1/#_including_c4corpustools_in_your_java_projects
+- Including C4CorpusTools in your Java projects
+- Working with C4Corpus - Word count example
+
+https://github.com/farhansiddiqui/webscale_nlp
+
+https://github.com/commoncrawl/language-detection-cld2
+---------
+There's already python code for getting text:
+
+https://spark-in.me/post/parsing-common-crawl-in-two-simple-commands
+https://gist.github.com/Smerity/afe7430fdb4371015466
+
+https://spark-in.me/post/parsing-common-crawl-in-two-simple-commands
+
+"But it turns out - it is not. This can be attributed to the effort that has been made to make the CC more accessible. The killer feature for me was the presence of their index weighting only ~200Gb, that also features a language detection option, i.e. you do not need to analyze top-level-domains or do any significant data mining."
+
+What does the "language detection option" discussion above mean?
+
+------------
+Skipping CrawlDiagnostics (see below) and robots.txt gz files:
+
+http://commoncrawl.org/2018/08/august-2018-crawl-archive-now-available/
+
+"HTTP 304 notmodified" responses are now stored as WARC revisit records in the "crawldiagnostics" subset along with 404s, redirects and other non-200 responses. For now the revisit records contain a payload digest although there is no payload sent together with HTTP 304 responses. The stupid reason is that the columnar index requires the digest field and we want to make sure that all tools continue to work as expected. The SHA-1 digest of an empty payload (zero bytes) is used for the revisit records.
+
+http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#revisit
+ârevisitâ
+General
+
+A ârevisitâ record describes the revisitation of content already archived, and might include only an abbreviated content body which has to be interpreted relative to a previous record. Most typically, a ârevisitâ record is used instead of a âresponseâ or âresourceâ record to indicate that the content visited was either a complete or substantial duplicate of material previously archived.
+...
+
+-------
+
+WET FILES:
+
+https://stackoverflow.com/questions/16649535/access-a-common-crawl-aws-public-dataset/25297965#25297965
+
+
+http://commoncrawl.org/2019/07/june-2019-crawl-archive-now-available/
+			File List	#Files	Total Size Compressed (TiB)
+	WET files	CC-MAIN-2019-26/wet.paths.gz	56000	7.59
+
+
+http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/
+(Instructions)
+
+https://gist.github.com/svemir/4207353
+(Hadoop related) A Common Crawl Experiment
+
+https://gist.github.com/Smerity/afe7430fdb4371015466
+
+ Extract just the text from Common Crawl WARC WET files
+ 
+https://stackoverflow.com/tags/common-crawl/hot?filter=all
+
+https://stackoverflow.com/questions/45920527/get-offset-and-length-of-a-subset-of-a-wat-archive-from-common-crawl-index-serve/46152773#46152773
+
+
+"The Common Crawl index does not contain offsets into WAT and WET files. So, the only way is to search the whole WAT/WET file for the desired record/URL. Eventually, it would be possible to estimate the offset because the record order in WARC and WAT/WET files is the same."
+
+https://dmorgan.info/posts/common-crawl-python/
+https://groups.google.com/forum/#!topic/common-crawl/pdI3w09AAbQ
+
+Example:
+WARC:
+tikauka:[142]/Scratch/anupama/maori-lang-detection>wget https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/segments/1563195526237.47/crawldiagnostics/CC-MAIN-20190719115720-20190719141720-00077.warc.gz
+WET:
+tikauka:[142]/Scratch/anupama/maori-lang-detection>wget https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/segments/1563195526237.47/wet/CC-MAIN-20190719115720-20190719141720-00508.warc.wet.gz
+tikauka:[142]/Scratch/anupama/maori-lang-detection>gunzip CC-MAIN-20190719115720-20190719141720-00508.warc.wet.gz 
+
+vagrant@node1:~/ia-hadoop-tools$ hdfs dfs -get /user/vagrant/cc-mri-subset/wet/MAORI-CC-2019-30-20190902100139-000000.warc.wet.gz /home/vagrant/.
+
+--------------------------------------------
+http://webdatacommons.org/
+
+https://dzone.com/articles/need-billions-of-web-pages-dont-bother-crawling
+
+	Ran Geva 2017-04-09
+	Like (0)
+
+	Excellent article! CommonCrawl is an amazing resourouce. You should also check out webdatacommons.org that is using their data and extract structured data (using RDFa, Microdata..)
+
+	If I may add a shameless plug here and tell you about Webhose.io [PAYWARE/SERVICES]. We provide an API to structured web data. The idea is the same as the one you presented. Instead of crawling the web, we already crawl millions of sites, download the data, structure and organize it so anyone can easily consume it and plug into their own system. 
+	Reply
+
+
+https://stackoverflow.com/questions/12097848/finding-all-domains-of-a-country
+
+	-> http://urlsearch.commoncrawl.org/
+	-> http://index.commoncrawl.org/
+	-> INSTRUCTIONS: https://groups.google.com/forum/#!msg/common-crawl/3QmQjFA_3y4/vTbhGqIBBQAJ
+
+
+Go to: http://index.commoncrawl.org/
+Grab the newest gzipped archive file.
+Then open it and find the cluster.idx file listed in it.
+Copy its relative URL, prefix with https://commoncrawl.s3.amazonaws.com/
+
+THEN:
+
+wharariki:[101]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>wget https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cluster.idx
+	--2019-07-29 17:40:45--  https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cluster.idx
+	Resolving commoncrawl.s3.amazonaws.com (commoncrawl.s3.amazonaws.com)... 52.216.8.171
+	Connecting to commoncrawl.s3.amazonaws.com (commoncrawl.s3.amazonaws.com)|52.216.8.171|:443... connected.
+	HTTP request sent, awaiting response... 200 OK
+	Length: 125059234 (119M) [binary/octet-stream]
+	Saving to: âcluster.idxâ
+
+	cluster.idx                       100%[============================================================>] 119.27M  8.51MB/s    in 15s     
+
+	2019-07-29 17:41:01 (7.83 MB/s) - âcluster.idxâ saved [125059234/125059234]
+
+wharariki:[102]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>grep '^nz,' cluster.idx | cut -f2 | uniq
+cdx-00237.gz
+cdx-00238.gz
+
+Prefix "https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/" to the listed gz files and wget them:
+
+	https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cdx-00237.gz
+	https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cc-index/collections/CC-MAIN-2019-26/indexes/cdx-00238.gz
+
+
+
+Unzip those, and we have all URLs with TLD .nz:
+	wharariki:[131]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>gunzip cdx-00237.gz 
+	wharariki:[132]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>gunzip cdx-00238.gz 
+
+
+The first of these files includes Norwegian TLDs (start with "no,") and the second gz file includes TLDs that start with "org,".
+So extract just those that start with "^nz," [https://www.unix.com/shell-programming-and-scripting/176608-how-copy-lines-starts-either-3-4-into-new-file.html].
+
+	wharariki:[107]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>egrep "^nz," cdx-00238 > nz-only-TLDs-from-237-238.txt
+	wharariki:[108]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>egrep "^nz," cdx-00238 >> nz-only-TLDs-from-237-238.txt
+
+
+Checking the abacusinstitute.ac.nz is also in the current June 2019 list:
+	egrep "ac,abacusinstitute" nz-only-TLDs-from-237-238.txt
+
+
+OTHER:
+https://www.tutorialspoint.com/hadoop/hadoop_mapreduce
+http://stormcrawler.net/
+http://storm.apache.org/getting-help.html
+
+
+https://dzone.com/articles/need-billions-of-web-pages-dont-bother-crawling
+Basically, each release is split into 100 segments. Each segment has three types of files: WARC, WAT, and WET. As explained on the Get Started page:
+
+    WARC files store the raw crawl data.
+    WAT files store computed metadata for the data stored in the WARC.
+    WET files store extracted plaintext from the data stored in the WARC.
+
+Note that WAT and WET are in the WARC format too! In fact, the WARC format is nothing more than an envelope with metadata and content. In the case of the WARC files, that content is the HTTP requests and responses, whereas, for the WET files, it is simply the plain text extracted from the WARCs. The WAT files contain a JSON representation of metadata extracted from the WARCs, e.g. title, links etc.
+
+
+Resources
+
+The Get Started page on the CommonCrawl website contains useful pointers to libraries and code in various programming languages to process the datasets. There is also a list of tutorials and presentations.
+
+It is also worth noting that CommonCrawl provides an index per release, allowing you to search for URLs (including wildcards) and retrieve the segment and offset therein where the content of the URL is stored, e.g.:
+
+	{ "urlkey": "org,apache)/", "timestamp": "20170220105827", "status": "200", "url": "http://apache.org/", "filename": "crawl-data/CC-MAIN-2017-09/segments/1487501170521.30/warc/CC-MAIN-20170219104610-00206-ip-10-171-10-108.ec2.internal.warc.gz", "length": "13315", "mime": "text/html", "offset": "14131184", "digest": "KJREISJSKKGH6UX5FXGW46KROTC6MBEM" }
+
+
+This is useful but only if you are interested in a limited number of URLs which you know in advance. In many cases, what you know in advance is what you want to extract, not where it will be extracted from. For situations such as these, you will need distributed batch-processing using MapReduce in Apache Hadoop or Apache Spark.
+
+
+https://www.forbes.com/sites/kalevleetaru/2017/09/28/common-crawl-and-unlocking-web-archives-for-research/#7067d4313b83
+One large web archive has bucked this trend and stood alone among its peers: Common Crawl. Similar to other large web archiving initiatives like the Internet Archive, Common Crawl conducts regular web wide crawls of the open web and preserves all of the content it downloads in the standard WARC file format. Unlike many other archives, it focuses primarily on preserving HTML web pages and does not archive images, videos, JavaScript files, CSS stylesheets, etc. Its goal is not to preserve the exact look and feel of a website on a given snapshot in time, but rather to collect a vast cross section of HTML web pages from across the web in a single place to enable large-scale data mining at web scale.
+...
+The project excludes sites which have robots.txt exclusion policies, following the historical policy of many other web archives, though it is worth noting that the Internet Archive earlier this year began slowly phasing out its reliance on such files due to their detrimental effect on preservation completeness. Common Crawl also allows sites to request removal from their index. Other than these cases, Common Crawl attempts to crawl as much of the remaining web as possible, aiming for a representative sample of the open web.
+...
+Ms. Crouse [Director of Common Crawl] noted the risk adverse nature of the web archiving community as a whole (historically many adhered and still adhere to a strict âopt inâ policy requiring prior approval before crawling a site) and the unwillingness of many archives to modernize their thinking on copyright and to engage more closely with the legal community in ways that could help them expand fair use horizons. In particular, she noted âsince we [in the US] are beholden to the Copyright Act, while living in a digital age, many well-intentioned organizations devoted to web science, archiving, and information provision may benefit from a stronger understanding of how copyright is interpreted in present day, and its hard boundariesâ and that âmany talented legal advisers and groups are interested in the precedent-setting nature of this topic; some are willing to work Pro Bono.â
+...
+Returning to the difference between Common Crawlâs datasets and traditional preservation-focused web archiving, Ms. Crouse emphasized that they capture only HTML pages and exclude multimedia content like images, video and other dynamic content.
+
+She noted that a key aspect of their approach to fair use is that web pages are intended for consumption by human beings one at a time using a web browser, while Common Crawl concatenates billions of pages together in the specialized WARC file format designed for machine data mining. Specifically, âCommon Crawl does not offer separate/individual web pages for easy consumption. The three data formats that are provided include text, metadata, and raw data, and the data is concatenatedâ and âthe format of the output is not a downloaded web page. The output is in WARC file format which contains the components of a page that are beneficial to machine-level analysis and make for space- efficient archiving (essentially: header, text, and some metadata).â
+
+As Ms. Crouse put it, âthis is big data intended for machine learning/readability. Further, our intention for its use is for public benefit i.e. to encourage research and innovation, not direct consumption.â She noted that âfrom the laypersonâs perspective, it is not at all trivial at present to extract a specific websiteâs content (that is, text) from a Common Crawl dataset. This task generally requires one to know how to install and run a Hadoop cluster, among other things. This is not structured data. Further it is likely that not all pages of that website will be included (depending on the parameters for depth set for the specific crawl).â This means that âthe bulk of [Common Crawlâs] users are from the noncommercial, educational, and research sectors. At a higher level, itâs important to note that we provide a broad and representative sample of the web, in the form of web crawl data, each month. No one really knows how big the web is, and at present, we limit our monthly data publication to approximately 3 billion pages.â
+
+
+Common Crawl believes it addresses this through the fact that its archive represents only a sample of each website crawled, rather than striving for 100% coverage. Specifically, Ms. Crouse noted that âat present, [crawls are] in monthly increments that are discontinuous month-to-month. We do only what is reasonable, necessary, and economical to achieve a representative sample. For instance, we limit the number of pages crawled from any given domain so, for large content owners, it is highly probable that their content, if included in a certain monthâs crawl data, is not wholly represented and thus not ideal for mining for comprehensive results âŠ if the content owner is not a large site, or in a niche market, their URL is less likely to be included in the seeds in the frontier, and, since we limit depth (# of links followed) for the sake of both economy and broader representative web coverage, 'niche' content may not even appear in a given monthâs dataset.â
+
+To put it another way, Common Crawlâs mission is to create a ârepresentative sampleâ of the web at large by crawling a sampling of pages and limiting the number of pages from each site they capture. Thus, their capture of any given site will represent a discontinuous sampling of pages that can change from month to month. A researcher wishing to analyze a single web site in its entirety would therefore not be able to turn to Common Crawl and would instead have to conduct their own crawl of the site or turn to a commercial aggregator that partners with the content holder to license the complete contents of the site.
+
+In Common Crawlâs view this is a critical distinction that sets it apart from both traditional web archiving and the commercial content aggregators that generate data mining revenue for content owners. By focusing on creating a ârepresentative sampleâ of the web at large, rather than attempting to capture a single site in its entirety (and in fact ensuring that it does not include more than a certain number of pages per site), the crawl self-limits itself to being applicable only to macro-level research examining web scale questions. Such âweb scaleâ questions cannot be answered through any existing open dataset and by incorporating specific design features Common Crawl ensures that more traditional research questions, like data mining the entirety of a single site, which might be viewed as redistribution of that site or competing with its ownerâs ability to license its content for data mining, is simply not possible.
+------------
Index: /other-projects/maori-lang-detection/MoreReading/Heritrix-and-WCT.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/Heritrix-and-WCT.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/MoreReading/Heritrix-and-WCT.txt	(revision 33635)
@@ -0,0 +1,369 @@
+
+
+RUN:
+export HERITRIX_HOME=/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT
+$HERITRIX_HOME/bin/heritrix -a admin:admin
+Visit: https://localhost:8443/engine
+
+MORE READING:
+http://open-s.com/en/content/heritrix-configuration
+http://open-s.com/en/content/wget
+	https://superuser.com/questions/655346/wget-execute-script-after-download
+https://www.gnu.org/software/wget/manual/wget.html
+	â--execute command
+
+    	Execute command as if it were a part of .wgetrc (see Startup File). A command thus invoked will be executed after the commands in .wgetrc, thus taking precedence over them.
+	If you need to specify more than one wgetrc command, use multiple instances of â-eâ.
+
+"crawler TRAPS": https://www.contentkingapp.com/academy/crawler-traps/
+	https://www.billhartzer.com/internet-marketing/crawl-thousands-urls/
+
+
+
+-----------------------------------------
+ Scope DecideReject Configuration rules
+-----------------------------------------
+Issues in H3.docx
+https://sbforge.org/download/attachments/.../Issues%20in%20H3.docx?version=1...
+
+By using maxTransHops and maxSpeculativeHops we thought that we could manage how long our discovery path 'X' should be, but we see different results and ...
+DOCX: https://sbforge.org/download/attachments/21856421/Issues%20in%20H3.docx?version=1&modificationDate=1465912557659&api=v2&usg=AOvVaw03EWO6Xy0XiMRITvFwR2v0
+[ https://webcache.googleusercontent.com/search?q=cache:bEM1AdjQR2cJ:https://sbforge.org/download/attachments/21856421/Issues%2520in%2520H3.docx%3Fversion%3D1%26modificationDate%3D1465912557659%26api%3Dv2+&cd=10&hl=en&ct=clnk&gl=nz&client=ubuntu ]
+
+
+
+    Speculative Hops
+
+        No really effect using maxTransHops or maxSpeculativeHops
+
+    By using maxTransHops and maxSpeculativeHops we thought that we could manage how long our discovery path âXâ should be, but we see different results and we still harvest several âXXâ or more.
+    To accomplish this we use HopsPathMatchesDecideRule but we havenât found a specific path that we certainly can say we want excluded from our harvest. We tried to make regex with R, E and X. Does anyone have experience with this?
+	
+
+	
+    Path seeds
+
+        Harvesting specific URI paths that doesnât end with slashes
+
+    When adding path seeds to only harvest from a specific place on a domain we sometimes have problems with redirections.
+    We always end path seeds with a slash, but sometimes we are redirected e.g. HTTP 301 http://www.bt.dk/plus/ â http://www.bt.dk/plus.
+    If we have following as a seed http://www.bt.dk/plus we will harvest the whole site because H3 harvest from the last slash in the seed?
+    Any experience with path seeds?
+
+
+--------------------------------------------------------------------------------
+
+TODO:
+
+https://www.stat.auckland.ac.nz/~paul/Reports/maori/maori.html
+ The macron is the only accent required for written MÄori and the accent can only be applied to vowels, so the full set of accented characters are:
+lower case a, with macron 	Ä 		upper case A, with macron 	Ä
+lower case e, with macron 	Ä 		upper case E, with macron 	Ä
+lower case i, with macron 	Ä« 		upper case I, with macron 	Äª
+lower case o, with macron 	Å 		upper case O, with macron 	Å
+lower case u, with macron 	Å« 		upper case U, with macron 	Åª 
+
+http://emacs.1067599.n8.nabble.com/Entering-vowels-with-macrons-td72136.html
+	Ctrl+\
+	type rfc1345 (and enter)
+	type &a- to get a-macron
+	Then Ctrl+\ to toggle back to default input
+	(Can thereafter toggle with Ctrl+\ to get back to rfc1345 input method)
+
+
+
+https://sachachua.com/blog/2011/04/writing-macrons-linux-latin-pronunciation/
+To add macrons: Ctrl-\ "latin-alt-postfix". But doesn't have all macronised vowels as used in te reo.
+Then Ctrl-\ to get default input editor back.
+https://www.gnu.org/software/emacs/manual/html_node/emacs/Select-Input-Method.html
+
+
+DOWNLOAD HERITRIX:
+BINARY: http://builds.archive.org/maven2/org/archive/heritrix/heritrix/3.4.0-SNAPSHOT/
+CODE, STATIC: https://github.com/internetarchive/heritrix3
+https://github.com/internetarchive/heritrix3/wiki/How%20To%20Crawl
+
+WEB CURATOR TOOL:
+http://dia-nz.github.io/webcurator/
+https://webcuratortool.readthedocs.io/en/latest/guides/quick-start-guide.html
+https://webcuratortool.readthedocs.io/en/latest/guides/overview-history.html
+
+
+Crawl as much of the nz domains as we can
+and run the language detection for Maori on the pages that come through
+and only save those pages and configure (somehow) that it knows that it doesn't need to redownload pages already *inspected* for language (not just that it detects it doesn't need to redownload the *stored* pages, since we only store mri language pages and not all pages inspected)
+
+Then break up the pages by sentences using our SentenceDetector model.
+
+
+SURT urls:
+http://crawler.archive.org/apidocs/org/archive/util/SURT.html
+
+WARC
+https://en.wikipedia.org/wiki/Web_ARChive
+
+Q:
+"The harvested material is captured in ARC/WARC format which has strong storage and archiving characteristics." at https://webcuratortool.readthedocs.io/en/latest/guides/user-manual.html
+
+https://blogs.loc.gov/thesignal/2013/11/anatomy-of-a-web-archive/
+
+ Nicholas Taylor
+November 13, 2013 at 2:49 pm
+
+Hi Ross, thanks for the comment. The tools for personal archiving of web pages and websites to WARC format are getting better, with the capture side further along than the replay side. Archive Ready (http://archiveready.com/) and WARCreate (http://warcreate.com/) can both be used to create a WARC containing all of the objects that make up an individual web page. GNU Wget 1.14+ (http://www.archiveteam.org/index.php?title=Wget_with_WARC_output) and WAIL (http://matkelly.com/wail/) can both be used to capture entire websites to WARC. WAIL also bundles a standalone Wayback Machine that runs locally, which is the easiest way I know of for users to view the content theyâve collected in WARC format.
+
+https://webcuratortool.readthedocs.io/en/latest/guides/user-manual.html
+Doesn't mention https
+
+"How targets work
+
+Targets consist of several important elements, including a name and description for internal use; a set of Seed URLs, ******a web harvester profile that controls the behaviour of the web crawler during the harvest******, one or more schedules that specify when the Target will be harvested, and (optionally) a set of descriptive metadata for the Target."
+
+Harvestor Configuration section:
+"The remaining tabs Pre-fetchers, Fetchers, Extractors, Writers, and Post-Processors are a series of processors that a URI passes through when it is crawled."
+
+
+http://crawler.archive.org/articles/user_manual/config.html
+Look for post-process*. Found under:
+6.1.3. Processing Chains
+
+6.1.2. Frontier
+
+The Frontier is a pluggable module that maintains the internal state of the crawl. What URIs have been discovered, crawled etc. As such its selection greatly effects, for instance, the order in which discovered URIs are crawled.
+
+There is only one Frontier per crawl job.
+
+Multiple Frontiers are provided with Heritrix, each of a particular character.
+6.1.2.1. BdbFrontier
+
+The default Frontier in Heritrix as of 1.4.0 and later is the BdbFrontier(Previously, the default was the Section 6.1.2.2, âHostQueuesFrontierâ). The BdbFrontier visits URIs and sites discovered in a generally breadth-first manner, it offers configuration options controlling how it throttles its activity against particular hosts, and whether it has a bias towards finishing hosts in progress ('site-first' crawling) or cycling among all hosts with pending URIs.
+
+Discovered URIs are only crawled once, except that robots.txt and DNS information can be configured so that it is refreshed at specified intervals for each host.
+
+The main difference between the BdbFrontier and its precursor, Section 6.1.2.2, âHostQueuesFrontierâ, is that BdbFrontier uses BerkeleyDB Java Edition to shift more running Frontier state to disk.
+
+6.1.2.2. HostQueuesFrontier
+
+The forerunner of the Section 6.1.2.1, âBdbFrontierâ. Now deprecated mostly because its custom disk-based data structures could not move as much Frontier state out of main memory as the BerkeleyDB Java Edition approach. Has same general characteristics as the Section 6.1.2.1, âBdbFrontierâ.
+
+
+https://webcuratortool.readthedocs.io/en/latest/guides/quick-start-guide.html
+You can use OpenWayback to view harvests from within WCT, see the wiki on the WCT Github page: https://github.com/DIA-NZ/webcurator/wiki/Wayback-Integration
+
+https://webarchive.jira.com/wiki/spaces/Heritrix/overview
+https://github.com/internetarchive/heritrix3/wiki/Heritrix3
+
+	"Unlike with previous releases, the web control interface is only made available via secure-socket HTTPS, and corresponding to this change the default port has changed to 8443. Additionally, unless you supply a compatible keystore via the new optional '-s' command-line switch, an 'ad-hoc' keystore with a new locally-generated SSL-capable certificate will be created (and then reused on future launches).
+
+	To then contact the web interface from a browser running on the same machine, visit the URL:
+
+	https://localhost:8443/
+	"
+
+	LOCALHOST WITH HTTPS is possible??? vs https://letsencrypt.org/docs/certificates-for-localhost/
+
+	"For local development
+
+	If youâre developing a web app, itâs useful to run a local web server like Apache or Nginx, and access it via http://localhost:8000/ in your web browser. However, web browsers behave in subtly different ways on HTTP vs HTTPS pages. The main difference: On an HTTPS page, any requests to load JavaScript from an HTTP URL will be blocked. So if youâre developing locally using HTTP, you might add a script tag that works fine on your development machine, but breaks when you deploy to your HTTPS production site. To catch this kind of problem, itâs useful to set up HTTPS on your local web server. However, you donât want to see certificate warnings all the time. How do you get the green lock locally?
+
+	The best option: Generate your own certificate, either self-signed or signed by a local root, and trust it in your operating systemâs trust store. Then use that certificate in your local web server. See below for details."
+
+	[Googled: https certificate localhost
+		https://www.freecodecamp.org/news/how-to-get-https-working-on-your-local-development-environment-in-5-minutes-7af615770eec/
+	]
+
+
+https://github.com/internetarchive/heritrix3/wiki/Heritrix%20Output#HeritrixOutput-WARCfiles
+* source-report.txt
+
+This report contains a line item for each host, which includes the seed from which the host was reached.
+Note
+
+    The sourceTagSeeds property of the TextSeedModule bean must be set to true for this report to be generated. 
+
+* WARC files
+
+Assuming you are using the WARC writer that comes with Heritrix, a number of WARC files will be generated containing crawled content.
+
+You can specify the storage location of WARC files by setting the directory value of the WARCWriterProcessor bean.
+
+
+https://github.com/internetarchive/heritrix3/wiki/Archiving%20Rich-Media%20Content
+Large File Sizes
+
+Rich-media content, such as Flash and video, is usually much larger than standard text/html pages.  Crawling such content requires large investments in storage and bandwidth.  To mitigate these issues, deduplication is recommended for rich-media crawls.  Deduplication detects previously collected content that is redundant and skips the download of such content.  Pointers to the duplicate content allow it to appear in subsequent crawls.  For details see Configuring Heritrix for Deduplication.
+
+
+Excessive Memory and CPU Usage
+
+Downloading rich-media content can often cause excessive load to be placed on the crawling computers memory and CPU.  For example, extracting links from Flash and other rich-media resources requires extensive data parsing, which is CPU intensive.   Atypical input patterns can also cause excessive CPU usage when regular expressions used by Heritrix are run.  It is therefore recommended that rich-media crawls be allocated more memory and CPU than "normal" crawls.  The memory allocated to Heritrix is set from the command line.  The following example shows the command line option to allocate 1 GB of memory to Heritrix, which should be sufficient for most rich-media crawls.
+
+export JAVA_OPTS=-Xmx1024M
+
+Multi-core processors are also recommended for rich-media crawls.
+
+
+Streaming media
+and
+Social Networking Sites
+Many social networking sites make use of rich-media to enhance their user-experience.  For specific guidelines on archiving social media sites see Archiving Social Networking Sites with Archive-It .  These instructions apply to the Archive-It application, which is built on top of Heritrix.
+
+
+Q: https://github.com/internetarchive/heritrix3/wiki/Avoiding%20Too%20Much%20Dynamic%20Content
+"To allow both foo.org and www.foo.org to be captured, you could add two seeds: http://www.foo.org/ and http://foo.org/. To allow every subdomain of foo.org to be crawled, you can add the seed http://foo.org.  Note the absence of a trailing slash."
+(Does the latter encompass both of the former?)
+
+
+Delete the TranclusionDecideRule, since this rule has the potential to lead Heritrix onto another host.  For example, if a URI returns a 301 response code (move permanently) or 302 (found) response code as well as a URI that contains a different host name than the seeds, Heritrix would accept this URI using the TransclusionDecideRule.  Removing this rule will keep Heritrix from straying off of our www.foo.org host.
+...
+Alternately, you can add the MatchesFilePatternDecideRule.  Set usePresetPattern to CUSTOM and set the regexp to something like: .foo\.org(?!/calendar).|.*foo\.org/calendar?year=200[56].*
+
+
+https://github.com/internetarchive/heritrix3/wiki/Mirroring%20HTML%20Files%20Only
+Mirroring HTML Files Only
+Alex Osborne edited this page on Jul 4, 2018 Â· 2 revisions
+
+Suppose you only want to crawl URIs that match http://foo.org/bar/\*.html.  Also, you would like to save the crawled files in a file/directory format instead of saving them in WARC files.  Also, assume the web server is case-sensitive.  For example, http://foo.org/bar/abc.html and http://foo.org/bar/ABC.HTML are pointing to two different resources.
+
+
+!! [If Heritrix needs to] be configured to differentiate between abc.html and ABC.HTML.  Do this by removing the LowercaseRule from the canonicalizationPolicy bean.
+
+https://github.com/internetarchive/heritrix3/wiki/Only%20Store%20Successful%20HTML%20Pages
+
+
+https://github.com/internetarchive/heritrix3/wiki/Jobs
+[multiple URLs to crawl can be specified. But what is the separator]
+
+Look up: Spring framework, Spring beans
+
+
+https://localhost:8443/engine/job/pinky/jobdir/crawler-beans.cxml?format=textedit
+ <!-- 
+   PROCESSING CHAINS
+    Much of the crawler's work is specified by the sequential 
+    application of swappable Processor modules. These Processors
+    are collected into three 'chains'. The CandidateChain is applied 
+    to URIs being considered for inclusion, before a URI is enqueued
+    for collection. The FetchChain is applied to URIs when their 
+    turn for collection comes up. The DispositionChain is applied 
+    after a URI is fetched and analyzed/link-extracted.
+  -->
+
+
+https://github.com/internetarchive/heritrix3/wiki/Fetch%20Chain%20Processors
+fetchHttp
+	
+
+This processor fetches HTTP URIs.  As of Heritrix 3.1, the crawler will now properly decode 'chunked' Transfer-Encoding -- even if encountered when it should not be used, as in a response to an HTTP/1.0 request. Additionally, the fetchHttp processor now includes the parameter 'useHTTP11', which if true, will cause Heritrix to report its requests as 'HTTP/1.1'.  This allows sites to use the 'chunked' Transfer-Encoding. (The default for this parameter is false for now, and Heritrix still does not reuse a persistent connection for more than one request to a site.)
+fetchHttp also includes the parameter 'acceptCompression', which if true, will cause Heritrix requests to include an "Accept-Encoding: gzip,deflate" header, which offers to receive compressed responses. (The default for this parameter is false for now.)
+
+extractorHttp
+	
+
+This processor extracts outlinks from HTTP headers.  As of Heritrix 3.1, the extractorHttp processor now considers any URI on a hostname to imply that the '/favicon.ico' from the same host should be fetched.  Also, as of Heritrix 3.1, the "inferRootPage" property has been added to the extractorHttp bean.  If this property is "true", Heritrix infers the '/' root page from any other URI on the same hostname.  The default for this setting is "false", which means the pre-3.1 behavior of only fetching the root page if it is a seed or otherwise discovered and in-scope remains in effect. Discovery via these new heuristics is considered to be a new 'I' (inferred) hop-type, and is treated the same in scoping/transclusion decisions as an 'E' (embed).
+
+
+https://github.com/internetarchive/heritrix3/wiki/Processor%20Settings
+
+fetchHttp:
+
+timeoutSeconds - This setting determines how long an HTTP request will wait for a resource to respond.  This setting should be set to a high value.
+ defaultEncoding - The character encoding to use for files that do not have one specified in the HTTP response headers. The default is ISO-8859â-1.
+
+soTimeoutMs - If the socket is unresponsive for this number of milliseconds, the request is cancelled.  Setting the value to zero (no timeout) is not recommended as it could hang a thread on an unresponsive server. This timeout is used to time out socket opens and socket reads. Make sure this value is less than timeoutSeconds for optimal configuration.  This ensures at least one retry read.
+
+sendIfModifiedSince - Send If-Modified-Since header, if previous Last-Modified fetch history information is available in URI history.
+
+sendIfNoneMatch - Send If-None-Match header, if previous Etag fetch history information is available in URI history.
+
+sendConnectionClose - Send Connection: close header with every request.
+    w3.org connection header documentation
+
+sendRange- Send the Range header when there is a limit on the retrieved document size.  This is for politeness purposes.  The Range header states
+that only the first n bytes are of interest.  It is only pertinent if maxLengthBytes is greater than zero.  Sending the Range header results in a
+206 Partial Content status response, which is better than cutting the response mid-download. On rare occasion, sending the Range header will
+generate 416 Request Range Not Satisfiable response.
+
+
+acceptHeaders - Accept Headers to include in each request. Each must be the complete header, e.g., Accept-Language: en.
+(mi is the 2 letter code for Maori, see https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)
+	https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
+
+
+
+ExtractorHtml:
+
+    extractJavascript - If true, in-page Javascript is scanned for strings that appear to be URIs. This typically finds both valid and invalid URIs.  Attempts to fetch the invalid URIs can generate webmaster concern over odd crawler behavior. Default is true.
+
+extractValueAttributes- If true, strings that look like URIs found in unusual places (such as form VALUE attributes) will be extracted. This typically finds both valid and invalid URIs.  Attempts to fetch the invalid URIs may generate webmaster concerns over odd crawler behavior. Default is true.
+
+
+ ignoreFormActionUrls - If true, URIs appearing as the ACTION attribute in HTML FORMs are ignored. Default is false.
+extractOnlyFormGets - If true, only ACTION URIs with a METHOD of GET (explicit or implied) are extracted. Default is true. 
+
+
+     
+
+candidates
+
+    seedsRedirectNewSeeds - If enabled, any URI found because a seed redirected to it (original seed returned 301 or 302), will also be treated as a seed.
+
+
+https://github.com/internetarchive/heritrix3/wiki/Statistics%20Tracking
+Statistics Tracking
+Alex Osborne edited this page on Jul 4, 2018 Â· 2 revisions
+
+Any number of statistics tracking modules can be attached to a crawl.  Currently only one is provided with Heritrix.  The statisticsTracker Spring bean that comes with Heritrix creates the progress-statistics.log file and provides the WUI with data to display progress information about the crawl.  It is strongly recommended that any crawl run through the WUI use this bean.
+
+
+
+https://github.com/internetarchive/heritrix3/wiki/Configuring-Crawl-Scope-Using-DecideRules
+
+------------
+REST API: https://heritrix.readthedocs.io/en/latest/api.html
+	Execute Script in Job
+
+	POST https://(heritrixhost):8443/engine/job/(jobname)/script
+
+	    Executes a script. The script can be written as Beanshell, ECMAScript, Groovy, or AppleScript.
+
+
+https://github.com/beanshell/beanshell
+https://github.com/internetarchive/heritrix3/wiki/BeanShell%20Script%20For%20Downloading%20Video
+https://github.com/internetarchive/heritrix3/wiki/Heritrix3-Useful-Scripts
+
+-----------
+
+LOGGING
+
+https://github.com/internetarchive/heritrix3/wiki/Configuring%20Crawl%20Scope%20Using%20DecideRules
+
+"DecideRuleSequence Logging
+
+Enable FINEST logging on the class org.archive.crawler.deciderules.DecideRuleSequence to watch each DecideRule's evaluation of the processed URI. This can be done in the logging.properties file
+
+logging.properties
+
+	org.archive.modules.deciderules.DecideRuleSequence.level = FINEST
+
+in conjunction with the -Dsysprop VM argument
+	-Djava.util.logging.config.file=/path/to/heritrix3/dist/src/main/conf/logging.properties
+
+"
+
+I couldn't get the above logging instructions to work, but here's what I did. 
+
+a. I modified conf/logging.properties by adding:
+
+# PINKY
+# DecideRuleSequence Logging
+# https://github.com/internetarchive/heritrix3/wiki/Configuring-Crawl-Scope-Using-DecideRules
+org.archive.modules.deciderules.DecideRuleSequence.level = FINEST
+
+b. I opened the bin/heritrix script and edited in the following into the 2 locations using $JAVACMD:
+
+	CLASSPATH=${CP} nohup $JAVACMD -Djava.util.logging.config.file=/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT/conf/logging.properties -Dheritrix.home=${HERITRIX_HOME}
+
+
+
+
Index: /other-projects/maori-lang-detection/MoreReading/Vagrant-Spark-Hadoop.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/Vagrant-Spark-Hadoop.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/MoreReading/Vagrant-Spark-Hadoop.txt	(revision 33635)
@@ -0,0 +1,728 @@
+Hadoop/Map-reduce
+
+https://www.guru99.com/create-your-first-hadoop-program.html
+
+Some Hadoop commands
+* https://community.cloudera.com/t5/Support-Questions/Closed-How-to-store-output-of-shell-script-in-HDFS/td-p/229933
+* https://stackoverflow.com/questions/26513861/checking-if-directory-in-hdfs-already-exists-or-not
+--------------
+To run firefox/anything graphical inside the VM run by vagrant, have to ssh -Y onto both analytics and then to the vagrant VM from analytics:
+1. ssh analytics -Y
+2. [anupama@analytics vagrant-hadoop-hive-spark]$ vagrant ssh -- -Y
+or
+vagrant ssh -- -Y node1
+(the -- flag tells the vagrant command that the subsequent -Y flag should be passed to the ssh cmd that vagrant runs)
+
+Only once ssh-ed with vagrant into the VM whose hostname is "node1", do you have access to node1's assigned IP: 10.211.55.101
+- Connecting machines, like analytics, must access node1 or use port forwarding to view the VM's servers on localhost. For example, on analytics, can view Yarn pages at http://localhost:8088/
+- If firefox is launched inside the VM (so inside node1), then can access pages off their respective ports at any of localhost|10.211.55.101|node1.
+
+===========================================
+		WARC TO WET
+===========================================
+https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+Sebastian Nagel 	
+05/07/2017
+Hi,
+
+unfortunately, we do not provide (yet) text extracts of the CC-NEWS archives.
+
+But it's easy to run the WET extractor on the WARC files, see:
+  https://groups.google.com/d/topic/common-crawl/imv4hlLob4s/discussion
+  https://groups.google.com/d/topic/common-crawl/b6yDG7EmnhM/discussion
+
+That's what you have to do:
+
+# download the WARC files and place them in a directory "warc/"
+# create sibling folders wat and wet
+# |
+# |-- warc/
+# |   |-- CC-NEWS-20161001224340-00008.warc.gz
+# |   |-- CC-NEWS-20161017145313-00000.warc.gz
+# |   `-- ...
+# |
+# |-- wat/
+# |
+# `-- wet/
+
+git clone https://github.com/commoncrawl/ia-web-commons
+cd ia-web-commons
+mvn install
+
+cd ..
+git clone https://github.com/commoncrawl/ia-hadoop-tools
+cd ia-hadoop-tools
+mvn package
+
+java -jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator \
+   -strictMode -skipExisting batch-id-xyz .../warc/*.warc.gz
+
+The folders wat/ and wet/ will then contain the exports.
+
+Best,
+Sebastian
+
+---
+
+1. So following the above instructions, I first made a warc subfolder in hdfs:///user/vagrant/cc-mri-subset/
+Then moved all the downloaded *warc.gz into there.
+Then created wat and wet subfolders in there alongside the warc folder.
+
+2. Next, I did the 2 git clone and mvn compile operations above.
+The first, ia-web-commons, successfully compiled (despite some test failures)
+
+3. BUT THE 2ND GIT PROJECT, ia-hadoop-tools, DIDN'T COMPILE AT FIRST, with the mvn package command failing:
+
+git clone https://github.com/commoncrawl/ia-hadoop-tools
+cd ia-hadoop-tools
+mvn package
+
+Compile failed with a message about the JSONTokener constructor not taking a String object. It turned out that the JSONTokener used was a version of the class that was too old. Whereas the necessary constructor is present in the most recent version, as seen in the API at https://docs.oracle.com/cd/E51273_03/ToolsAndFrameworks.110/apidoc/assembler/com/endeca/serialization/json/JSONTokener.html
+
+So instead, I opened up ia-hadoop-tools/pom.xml for editing and added the newest version of the org.json package's json (see http://builds.archive.org/maven2/org/json/json/ for <version>) into the pom.xml's <dependencies> element, based on how it this was done at https://bukkit.org/threads/problem-loading-libraries-with-maven-java-lang-noclassdeffounderror-org-json-jsonobject.428354/:
+
+   <dependency>
+      <groupId>org.json</groupId>
+      <artifactId>json</artifactId>
+      <version>20131018</version>
+    </dependency>
+
+Then I was able to run "mvn package" successfully.
+(Maybe I could also have added in a far more recent version, as seen in the version numbers at https://mvnrepository.com/artifact/org.json/json,
+but didn't want to go too far ahead in case there was other incompatibility.)
+
+4. Next, I wanted to finally run the built executable to convert the warc files to wet files.
+
+I had the warc files on the hadoop filesystem. The original instructions at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to however were apparently for working with warcs stored on the local filesystem, as those instructions did not run the hadoop command but the regular java command. The regular java command did not work with the files being on the hadoop system (attempt #1 below).
+
+ATTEMPTS THAT DIDN'T WORK:
+1. vagrant@node1:~/ia-hadoop-tools$ java -jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz hdfs:///user/vagrant/cc-mri-subset/warc/*.warc.gz
+2. vagrant@node1:~/ia-hadoop-tools$ $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz hdfs:///user/vagrant/cc-mri-subset/warc/*.warc.gz
+
+
+The 2nd attempt, which uses a proper hadoop command, I based off https://www.tutorialspoint.com/map_reduce/implementation_in_hadoop.htm
+It produced lots of errors and the output wet (and wat) .gz files were all corrupt as gunzip could not successfully run over them:
+
+vagrant@node1:~/ia-hadoop-tools$ $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz hdfs:///user/vagrant/cc-mri-subset/warc/*.warc.gz
+19/09/05 05:57:22 INFO Configuration.deprecation: mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902100139-000000.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902100141-000001.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902100451-000002.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902100453-000003.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902100805-000004.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902100809-000005.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902101119-000006.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902101119-000007.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902101429-000008.warc.gz
+19/09/05 05:57:23 INFO jobs.WEATGenerator: Add input path: hdfs://node1/user/vagrant/cc-mri-subset/warc/MAORI-CC-2019-30-20190902101429-000009.warc.gz
+19/09/05 05:57:23 INFO client.RMProxy: Connecting to ResourceManager at node1/127.0.0.1:8032
+19/09/05 05:57:23 INFO client.RMProxy: Connecting to ResourceManager at node1/127.0.0.1:8032
+19/09/05 05:57:23 INFO mapred.FileInputFormat: Total input paths to process : 10
+19/09/05 05:57:24 INFO mapreduce.JobSubmitter: number of splits:10
+19/09/05 05:57:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1567397114047_0001
+19/09/05 05:57:24 INFO impl.YarnClientImpl: Submitted application application_1567397114047_0001
+19/09/05 05:57:24 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1567397114047_0001/
+19/09/05 05:57:24 INFO mapreduce.Job: Running job: job_1567397114047_0001
+19/09/05 05:57:31 INFO mapreduce.Job: Job job_1567397114047_0001 running in uber mode : false
+19/09/05 05:57:31 INFO mapreduce.Job:  map 0% reduce 0%
+19/09/05 05:57:44 INFO mapreduce.Job:  map 10% reduce 0%
+19/09/05 05:57:44 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000002_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+Container killed by the ApplicationMaster.
+Container killed on request. Exit code is 143
+Container exited with a non-zero exit code 143
+
+19/09/05 05:57:45 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000004_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+Container killed by the ApplicationMaster.
+Container killed on request. Exit code is 143
+Container exited with a non-zero exit code 143
+
+19/09/05 05:57:45 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000001_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:57:45 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000005_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:57:45 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000000_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:57:45 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000003_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:57:46 INFO mapreduce.Job:  map 0% reduce 0%
+19/09/05 05:57:54 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000007_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:57:55 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000006_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:57:57 INFO mapreduce.Job:  map 10% reduce 0%
+19/09/05 05:57:57 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000009_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+Container killed by the ApplicationMaster.
+Container killed on request. Exit code is 143
+Container exited with a non-zero exit code 143
+
+19/09/05 05:57:58 INFO mapreduce.Job:  map 20% reduce 0%
+19/09/05 05:57:58 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000008_0, Status : FAILED
+Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+19/09/05 05:58:06 INFO mapreduce.Job:  map 30% reduce 0%
+19/09/05 05:58:08 INFO mapreduce.Job:  map 60% reduce 0%
+19/09/05 05:58:09 INFO mapreduce.Job:  map 70% reduce 0%
+19/09/05 05:58:10 INFO mapreduce.Job:  map 80% reduce 0%
+19/09/05 05:58:12 INFO mapreduce.Job:  map 90% reduce 0%
+19/09/05 05:58:13 INFO mapreduce.Job:  map 100% reduce 0%
+19/09/05 05:58:13 INFO mapreduce.Job: Job job_1567397114047_0001 completed successfully
+19/09/05 05:58:13 INFO mapreduce.Job: Counters: 32
+	File System Counters
+		FILE: Number of bytes read=0
+		FILE: Number of bytes written=1239360
+		FILE: Number of read operations=0
+		FILE: Number of large read operations=0
+		FILE: Number of write operations=0
+		HDFS: Number of bytes read=1430
+		HDFS: Number of bytes written=0
+		HDFS: Number of read operations=30
+		HDFS: Number of large read operations=0
+		HDFS: Number of write operations=0
+	Job Counters 
+		Failed map tasks=10
+		Launched map tasks=20
+		Other local map tasks=10
+		Data-local map tasks=10
+		Total time spent by all maps in occupied slots (ms)=208160
+		Total time spent by all reduces in occupied slots (ms)=0
+		Total time spent by all map tasks (ms)=208160
+		Total vcore-milliseconds taken by all map tasks=208160
+		Total megabyte-milliseconds taken by all map tasks=213155840
+	Map-Reduce Framework
+		Map input records=10
+		Map output records=0
+		Input split bytes=1430
+		Spilled Records=0
+		Failed Shuffles=0
+		Merged Map outputs=0
+		GC time elapsed (ms)=1461
+		CPU time spent (ms)=2490
+		Physical memory (bytes) snapshot=1564528640
+		Virtual memory (bytes) snapshot=19642507264
+		Total committed heap usage (bytes)=1126170624
+	File Input Format Counters 
+		Bytes Read=0
+	File Output Format Counters 
+		Bytes Written=0
+vagrant@node1:~/ia-hadoop-tools$ 
+
+
+5. The error messages are all the same but not very informative
+   19/09/05 05:57:45 INFO mapreduce.Job: Task Id : attempt_1567397114047_0001_m_000001_0, Status : FAILED
+   Error: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;
+
+All the references I could find on google indicated that the full version of the error message was that this method (com.google.common.io.ByteStreams.limit(...)) could not be located.
+The page at http://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3C7AD5ED91-F8BA-43C9-9030-049BAB360BD4@gmail.com%3E
+revealed that guava.jar contains the com.google.common.io.ByteStreams class.
+
+
+TO GET THE EXECUTABLE TO WORK:
+I located guava.jar, found there were 2 identical ones on the filesystem but that neither was on the hadoop classpath yet, so I copied it into one of the Hadoop Classpath locations. Then I was able to successfully run the executable and produce meaningful WET files at last from the WARC input files:
+ 
+
+vagrant@node1:~$ locate guava.jar
+/usr/share/java/guava.jar
+/usr/share/maven/lib/guava.jar
+vagrant@node1:~$ jar -tvf /usr/share/maven/lib/guava.jar | less
+vagrant@node1:~$ jar -tvf /usr/share/java/guava.jar | less
+# both contained the ByteStreams class
+
+vagrant@node1:~$ cd -
+/home/vagrant/ia-hadoop-tools
+vagrant@node1:~/ia-hadoop-tools$ find . -name "guava.jar"
+# None in the git project
+
+vagrant@node1:~/ia-hadoop-tools$ hadoop classpath
+/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar
+# guava.jar not on hadoop classpath yet
+
+vagrant@node1:~/ia-hadoop-tools$ diff /usr/share/java/guava.jar /usr/share/maven/lib/guava.jar
+# no differences, identical
+
+vagrant@node1:~/ia-hadoop-tools$ hdfs dfs -put /usr/share/java/guava.jar /usr/local/hadoop/share/hadoop/common/.
+put: `/usr/local/hadoop/share/hadoop/common/.': No such file or directory
+# hadoop classpath locations are not on the hdfs filesystem, but on the regular fs
+
+vagrant@node1:~/ia-hadoop-tools$ sudo cp /usr/share/java/guava.jar /usr/local/hadoop/share/hadoop/common/.
+vagrant@node1:~/ia-hadoop-tools$ hadoop classpath
+/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar
+# Copied guava.jar to somewhere on existing hadoop classpath
+
+vagrant@node1:~/ia-hadoop-tools$ $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz hdfs:///user/vagrant/cc-mri-subset/warc/*.warc.gz
+# Successful run
+
+vagrant@node1:~$ hdfs dfs -get /user/vagrant/cc-mri-subset/wet/MAORI-CC-2019-30-20190902100139-000000.warc.wet.gz /home/vagrant/.
+vagrant@node1:~$ cd ..
+vagrant@node1:~$ gunzip MAORI-CC-2019-30-20190902100139-000000.warc.wet.gz
+vagrant@node1:~$ less MAORI-CC-2019-30-20190902100139-000000.warc.wet
+# Copied a WET output file from the hadoop filesystem to local filesystem and inspected its contents. Works!
+
+-----------------------------------
+ VIEW THE MRI-ONLY INDEX GENERATED
+-----------------------------------
+hdfs dfs -cat hdfs:///user/vagrant/cc-mri-csv/part* | tail -5
+
+(gz archive, binary file)
+
+vagrant@node1:~/cc-index-table/src/script$ hdfs dfs -mkdir hdfs:///user/vagrant/cc-mri-unzipped-csv
+
+# https://stackoverflow.com/questions/34573279/how-to-unzip-gz-files-in-a-new-directory-in-hadoop
+XXX vagrant@node1:~/cc-index-table/src/script$ hadoop fs -cat hdfs:///user/vagrant/cc-mri-csv/part* | gzip -d | hadoop fs -put - hdfs:///user/vagrant/cc-mri-unzipped-csv
+
+
+vagrant@node1:~/cc-index-table/src/script$ hdfs dfs -cat hdfs:///user/vagrant/cc-mri-csv/part* | gzip -d | hdfs dfs -put - hdfs:///user/vagrant/cc-mri-unzipped-csv/cc-mri.csv
+vagrant@node1:~/cc-index-table/src/script$ hdfs dfs -ls hdfs:///user/vagrant/cc-mri-unzipped-csv
+Found 1 items
+-rw-r--r--   1 vagrant supergroup   71664603 2019-08-29 04:47 hdfs:///user/vagrant/cc-mri-unzipped-csv/cc-mri.csv
+
+# https://stackoverflow.com/questions/14925323/view-contents-of-file-in-hdfs-hadoop
+vagrant@node1:~/cc-index-table/src/script$ hdfs dfs -cat hdfs:///user/vagrant/cc-mri-unzipped-csv/cc-mri.csv | tail -5
+
+# url, warc_filename, warc_record_offset, warc_record_length
+http://paupauocean.com/page91?product_id=142&brd=1,crawl-data/CC-MAIN-2019-30/segments/1563195526940.0/warc/CC-MAIN-20190721082354-20190721104354-00088.warc.gz,115081770,21404
+https://cookinseln-reisen.de/cook-inseln/rarotonga/,crawl-data/CC-MAIN-2019-30/segments/1563195526799.4/warc/CC-MAIN-20190720235054-20190721021054-00289.warc.gz,343512295,12444
+http://www.halopharm.com/mi/profile/,crawl-data/CC-MAIN-2019-30/segments/1563195525500.21/warc/CC-MAIN-20190718042531-20190718064531-00093.warc.gz,219160333,10311
+https://www.firstpeople.us/pictures/green/Touched-by-the-hand-of-Time-1907.html,crawl-data/CC-MAIN-2019-30/segments/1563195526670.1/warc/CC-MAIN-20190720194009-20190720220009-00362.warc.gz,696195242,5408
+https://www.sos-accessoire.com/programmateur-programmateur-module-electronique-whirlpool-481231028062-27573.html,crawl-data/CC-MAIN-2019-30/segments/1563195527048.80/warc/CC-MAIN-20190721144008-20190721170008-00164.warc.gz,830087190,26321
+
+# https://stackoverflow.com/questions/32612867/how-to-count-lines-in-a-file-on-hdfs-command
+vagrant@node1:~/cc-index-table/src/script$ hdfs dfs -cat hdfs:///user/vagrant/cc-mri-unzipped-csv/cc-mri.csv | wc -l
+345625
+
+
+ANOTHER WAY (DR BAINBRIDGE'S WAY) TO CREATE SINGLE .CSV FILE FROM /part* FILES AND VIEW ITS CONTENTS:
+vagrant@node1:~/cc-index-table$ hdfs dfs -cat hdfs:///user/vagrant/cc-mri-csv/part* > file.csv.gz
+vagrant@node1:~/cc-index-table$ less file.csv.gz 
+
+
+https://www.patricia-anong.com/blog/2017/11/1/extend-vmdk-on-virtualbox
+
+
+When not using LIKE '%mri%' but = 'mri' instead:
+vagrant@node1:~/cc-index-table$ hdfs dfs -cat hdfs:///user/vagrant/cc-mri-unzipped-csv/cc-mri.csv | wc -l
+5767
+
+
+For a month later, the August 2019 crawl:
+vagrant@node1:~$ hdfs dfs -cat hdfs:///user/vagrant/CC-MAIN-2019-35/cc-mri-unzipped-csv/cc-mri.csv | wc -l
+9318
+
+-----------------------------------------
+Running export_mri_subset.sh
+-----------------------------------------
+
+The export_mri_subset.sh script is set up run on the csv input file produced by running export_mri_index_csv.sh
+
+Running this initially produced the following exception:
+
+
+2019-08-29 05:48:52 INFO  CCIndexExport:152 - Number of records/rows matched by query: 345624
+2019-08-29 05:48:52 INFO  CCIndexExport:157 - Distributing 345624 records to 70 output partitions (max. 5000 records per WARC file)
+2019-08-29 05:48:52 INFO  CCIndexExport:165 - Repartitioning data to 70 output partitions
+Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`url`' given input columns: [http://176.31.110.213:600/?p=287, crawl-data/CC-MAIN-2019-30/segments/1563195527531.84/warc/CC-MAIN-20190722051628-20190722073628-00547.warc.gz, 1215489, 15675];;
+'Project ['url, 'warc_filename, 'warc_record_offset, 'warc_record_length]
++- AnalysisBarrier
+      +- Repartition 70, true
+         +- Relation[http://176.31.110.213:600/?p=287#10,crawl-data/CC-MAIN-2019-30/segments/1563195527531.84/warc/CC-MAIN-20190722051628-20190722073628-00547.warc.gz#11,1215489#12,15675#13] csv
+
+	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
+	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:88)
+	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:85)
+	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
+	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
+	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
+	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:95)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:95)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:106)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$1.apply(QueryPlan.scala:120)
+	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
+	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
+	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
+	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
+	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
+	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:120)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:125)
+	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:125)
+	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:95)
+	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
+	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:80)
+	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
+	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:80)
+	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91)
+	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:104)
+	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
+	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
+	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
+	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
+	at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:3295)
+	at org.apache.spark.sql.Dataset.select(Dataset.scala:1307)
+	at org.apache.spark.sql.Dataset.select(Dataset.scala:1325)
+	at org.apache.spark.sql.Dataset.select(Dataset.scala:1325)
+	at org.commoncrawl.spark.examples.CCIndexWarcExport.run(CCIndexWarcExport.java:169)
+	at org.commoncrawl.spark.examples.CCIndexExport.run(CCIndexExport.java:192)
+	at org.commoncrawl.spark.examples.CCIndexWarcExport.main(CCIndexWarcExport.java:214)
+	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
+	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
+	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
+	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
+	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
+	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
+2019-08-29 05:48:52 INFO  SparkContext:54 - Invoking stop() from shutdown hook
+
+
+
+Hints to solve it were at https://stackoverflow.com/questions/45972929/scala-dataframereader-keep-column-headers
+The actual solution is to edit the CCIndexWarcExport.java as follows:
+1. set option(header) to false since the csv file contains no header row, only data rows. You can confirm the csv has no header row by doing
+   hdfs dfs -cat hdfs:///user/vagrant/cc-mri-csv/part* | head -5
+   
+2. The 4 column names are inferred as _c0 to _c3, not as url/warc_filename etc.
+
+emacs src/main/java/org/commoncrawl/spark/examples/CCIndexWarcExport.java
+
+Change:
+	sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+                                        .load(csvQueryResult);
+To
+	sqlDF = sparkSession.read().format("csv").option("header", false).option("inferSchema", true)
+                                        .load(csvQueryResult);
+
+And comment out:
+	//JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+                                .toJavaRDD();
+Replace with the default inferred column names:
+	JavaRDD<Row> rdd = sqlDF.select("_c0", "_c1", "_c2", "_c3").rdd()
+                                .toJavaRDD();
+
+
+Now recompile:
+    mvn package
+
+And run:
+    ./src/script/export_mri_subset.sh
+
+-------------------------
+
+WET example from https://github.com/commoncrawl/cc-warc-examples
+
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -mkdir /user/vagrant/data
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -put data/CC-MAIN-20190715175205-20190715200159-00000.warc.wet.gz hdfs:///user/vagrant/data/.
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -ls data
+Found 1 items
+-rw-r--r--   1 vagrant supergroup  154823265 2019-08-19 08:23 data/CC-MAIN-20190715175205-20190715200159-00000.warc.wet.gz
+vagrant@node1:~/cc-warc-examples$ hadoop jar target/cc-warc-examples-0.3-SNAPSHOT-jar-with-dependencies.jar org.commoncrawl.examples.mapreduce.WETWordCount
+
+<ONCE FINISHED:>
+
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -cat /tmp/cc/part*
+
+
+
+INFO ON HADOOP/HDFS:
+https://www.bluedata.com/blog/2016/08/hadoop-hdfs-upgrades-painful/
+
+SPARK:
+configure option example: https://stackoverflow.com/questions/46152526/how-should-i-configure-spark-to-correctly-prune-hive-metastore-partitions
+
+
+
+LIKE '%isl%'
+
+cd cc-index-table
+APPJAR=target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+> $SPARK_HOME/bin/spark-submit \
+#    $SPARK_ON_YARN \
+   --conf spark.hadoop.parquet.enable.dictionary=true \
+   --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   --conf spark.sql.hive.metastorePartitionPruning=true \
+   --conf spark.sql.parquet.filterPushdown=true \
+   --conf spark.sql.parquet.mergeSchema=true \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $APPJAR \
+   --query "SELECT url, warc_filename, warc_record_offset, warc_record_length
+            FROM ccindex
+            WHERE crawl = 'CC-MAIN-2019-30' AND subset = 'warc' AND content_languages LIKE '%mri%'" \
+   --numOutputPartitions 12 \
+   --numRecordsPerWarcFile 20000 \
+   --warcPrefix ICELANDIC-CC-2018-43 \
+   s3://commoncrawl/cc-index/table/cc-main/warc/ \
+   .../my_output_path/
+
+
+----
+TIME
+----
+1. https://dzone.com/articles/need-billions-of-web-pages-dont-bother-crawling
+http://digitalpebble.blogspot.com/2017/03/need-billions-of-web-pages-dont-bother_29.html
+
+"So, not only have CommonCrawl given you loads of web data for free, theyâve also made your life easier by preprocessing the data for you. For many tasks, the content of the WAT or WET files will be sufficient and you wonât have to process the WARC files.
+
+This should not only help you simplify your code but also make the whole processing faster. We recently ran an experiment on CommonCrawl where we needed to extract anchor text from HTML pages. We initially wrote some MapReduce code to extract the binary content of the pages from their WARC representation, processed the HTML with JSoup and reduced on the anchor text. Processing a single WARC segment took roughly 100 minutes on a 10-node EMR cluster. We then simplified the extraction logic, took the WAT files as input and the processing time dropped to 17 minutes on the same cluster. This gain was partly due to not having to parse the web pages, but also to the fact that WAT files are a lot smaller than their WARC counterparts."
+
+2. https://spark-in.me/post/parsing-common-crawl-in-two-simple-commands
+"Both the parsing part and the processing part take just a couple of minutes per index file / WET file - the bulk of the âcomputeâ lies within actually downloading these files.
+
+Essentially if you have some time to spare and an unlimited Internet connection, all of this processing can be done on one powerful machine. You can be fancy and go ahead and rent some Amazon server(s) to minimize the download time, but that can be costly.
+
+In my experience - parsing the whole index for Russian websites (just filtering by language) takes approximately 140 hours - but the majority of this time is just downloading (my speed averaged ~300-500 kb/s)."
+
+----
+CMDS
+----
+https://stackoverflow.com/questions/29565716/spark-kill-running-application
+
+=========================================================
+Configuring spark to work on Amazon AWS s3a dataset:
+=========================================================
+https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark
+http://deploymentzone.com/2015/12/20/s3a-on-spark-on-aws-ec2/
+https://answers.dataiku.com/1734/common-crawl-s3
+https://stackoverflow.com/questions/2354525/what-should-be-hadoop-tmp-dir
+https://stackoverflow.com/questions/40169610/where-exactly-should-hadoop-tmp-dir-be-set-core-site-xml-or-hdfs-site-xml
+
+https://stackoverflow.com/questions/43759896/spark-truncated-the-string-representation-of-a-plan-since-it-was-too-large-w
+
+
+https://sparkour.urizone.net/recipes/using-s3/
+Configuring Spark to Use Amazon S3
+"Some Spark tutorials show AWS access keys hardcoded into the file paths. This is a horribly insecure approach and should never be done. Use exported environment variables or IAM Roles instead, as described in Configuring Amazon S3 as a Spark Data Source."
+
+"No FileSystem for scheme: s3n
+
+java.io.IOException: No FileSystem for scheme: s3n
+
+This message appears when dependencies are missing from your Apache Spark distribution. If you see this error message, you can use the --packages parameter and Spark will use Maven to locate the missing dependencies and distribute them to the cluster. Alternately, you can use --jars if you manually downloaded the dependencies already. These parameters also works on the spark-submit script."
+
+===========================================
+IAM Role (or user) and commoncrawl profile
+===========================================
+
+"iam" role or user for commoncrawl(er) profile
+
+
+aws management console:
+davidb@waikato.ac.nz
+lab pwd, capital R and ! (maybe g)
+
+commoncrawl profile created while creating the user/role, by following the instructions at: https://answers.dataiku.com/1734/common-crawl-s3
+
+https://answers.dataiku.com/1734/common-crawl-s3
+Even though the bucket is public, if your AWS key does not have your full permissions (ie if it's a restricted IAM user), you need to grant explicit access to the commoncrawl bucket: attach the following policy to your IAM user:
+#### START JSON (POLICY) ###
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "Stmt1503647467000",
+            "Effect": "Allow",
+            "Action": [
+                "s3:GetObject",
+                "s3:ListBucket"
+            ],
+            "Resource": [
+                "arn:aws:s3:::commoncrawl/*",
+                "arn:aws:s3:::commoncrawl"
+            ]
+        }
+    ]
+}
+#### END ###
+
+<!--
+    <property>
+      <name>fs.s3a.awsAccessKeyId</name>
+      <value>XXX</value>
+    </property>
+    <property>
+      <name>fs.s3a.awsSecretAccessKey</name>
+      <value>XXX</value>
+    </property>
+-->
+
+
+[If accesskey and secret were specified in hadoop core-site.xml and not in spark conf props file, then running export_maori_index_csv.sh produced the following error:
+
+2019-08-29 06:16:38 INFO  StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
+2019-08-29 06:16:40 WARN  FileStreamSink:66 - Error while looking for metadata directory.
+Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
+	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
+	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
+	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
+	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
+	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
+]
+
+Instead of putting the access and secret keys in hadoop's core-site.xml as above (with sudo emacs /usr/local/hadoop-2.7.6/etc/hadoop/core-site.xml)
+
+you'll want to put the Amazon AWS access key and secret key in the spark properties file:
+
+       sudo emacs /usr/local/spark-2.3.0-bin-hadoop2.7/conf/spark-defaults.conf
+
+
+The spark properties conf file above should contain:
+
+spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem  
+spark.hadoop.fs.s3a.access.key=PASTE_IAM-ROLE_ACCESSKEY  
+spark.hadoop.fs.s3a.secret.key=PASTE_IAM-ROLE_SECRETKEY  
+
+
+
+When the job is running, can visit the Spark Context at http://node1:4040/jobs/ (http://node1:4041/jobs/ for me first time, since I forwarded the vagrant VM's ports at +1. However, subsequent times it was on node1:4040/jobs?)
+
+-------------
+
+APPJAR=target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+$SPARK_HOME/bin/spark-submit \
+	--conf spark.hadoop.parquet.enable.dictionary=true \
+	--conf spark.hadoop.parquet.enable.summary-metadata=false \
+	--conf spark.sql.hive.metastorePartitionPruning=true \
+	--conf spark.sql.parquet.filterPushdown=true \
+	--conf spark.sql.parquet.mergeSchema=true \
+	--class org.commoncrawl.spark.examples.CCIndexExport $APPJAR \
+        --query "SELECT url, warc_filename, warc_record_offset, warc_record_length   
+            FROM ccindex                                                        
+            WHERE crawl = 'CC-MAIN-2019-30' AND subset = 'warc' AND content_languages LIKE '%mri%'" \
+	--outputFormat csv \
+	--numOutputPartitions 10 \
+	--outputCompression gzip \
+	s3://commoncrawl/cc-index/table/cc-main/warc/ \
+	hdfs:///user/vagrant/cc-mri-csv
+
+----------------
+Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
+
+
+https://stackoverflow.com/questions/39355354/spark-no-filesystem-for-scheme-https-cannot-load-files-from-amazon-s3
+https://stackoverflow.com/questions/33356041/technically-what-is-the-difference-between-s3n-s3a-and-s3
+"2018-01-10 Update Hadoop 3.0 has cut its s3: and s3n implementations: s3a is all you get. It is now significantly better than its predecessor and performs as least as good as the Amazon implementation. Amazon's "s3:" is still offered by EMR, which is their closed source client. Consult the EMR docs for more info."
+
+1. https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark
+
+"Having experienced first hand the difference between s3a and s3n - 7.9GB of data transferred on s3a was around ~7 minutes while 7.9GB of data on s3n took 73 minutes [us-east-1 to us-west-1 unfortunately in both cases; Redshift and Lambda being us-east-1 at this time] this is a very important piece of the stack to get correct and it's worth the frustration.
+
+Here are the key parts, as of December 2015:
+
+    Your Spark cluster will need a Hadoop version 2.x or greater. If you use the Spark EC2 setup scripts and maybe missed it, the switch for using something other than 1.0 is to specify --hadoop-major-version 2 (which uses CDH 4.2 as of this writing).
+
+    You'll need to include what may at first seem to be an out of date AWS SDK library (built in 2014 as version 1.7.4) for versions of Hadoop as late as 2.7.1 (stable): aws-java-sdk 1.7.4. As far as I can tell using this along with the specific AWS SDK JARs for 1.10.8 hasn't broken anything.
+
+    You'll also need the hadoop-aws 2.7.1 JAR on the classpath. This JAR contains the class org.apache.hadoop.fs.s3a.S3AFileSystem.
+
+    In spark.properties you probably want some settings that look like this:
+
+    spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem  
+    spark.hadoop.fs.s3a.access.key=ACCESSKEY  
+    spark.hadoop.fs.s3a.secret.key=SECRETKEY
+
+I've detailed this list in more detail on a post I wrote as I worked my way through this process. In addition I've covered all the exception cases I hit along the way and what I believe to be the cause of each and how to fix them."
+
+
+2. The classpath used by hadoop can be found by running the command (https://stackoverflow.com/questions/26748811/setting-external-jars-to-hadoop-classpath):
+    hadoop classpath
+
+
+3. Got hadoop-aws 2.7.6 jar
+from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.6
+and put it into /home/vagrant
+
+
+4. https://stackoverflow.com/questions/26748811/setting-external-jars-to-hadoop-classpath
+https://stackoverflow.com/questions/28520821/how-to-add-external-jar-to-hadoop-job/54459211#54459211
+vagrant@node1:~$ export LIBJARS=/home/vagrant/hadoop-aws-2.7.6.jar
+vagrant@node1:~$ export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+vagrant@node1:~$ hadoop classpath
+
+5. https://community.cloudera.com/t5/Community-Articles/HDP-2-4-0-and-Spark-1-6-0-connecting-to-AWS-S3-buckets/ta-p/245760
+"Download the aws sdk for java https://aws.amazon.com/sdk-for-java/ Uploaded it to the hadoop directory. You should see the aws-java-sdk-1.10.65.jar in /usr/hdp/2.4.0.0-169/hadoop/"
+
+I got version 1.11
+
+[Can't find a spark.properties file, but this seems to contain spark specific properties:
+$SPARK_HOME/conf/spark-defaults.conf
+
+https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-properties.html
+"The default Spark properties file is $SPARK_HOME/conf/spark-defaults.conf that could be overriden using spark-submit with the --properties-file command-line option."]
+
+Can SUDO COPY the 2 jar files hadoop-aws-2.7.6.jar and aws-java-sdk-1.11.616.jar to:
+/usr/local/hadoop/share/hadoop/common/
+(else /usr/local/hadoop/share/hadoop/hdfs/hadoop-aws-2.7.6.jar)
+
+--------
+schema
+https://commoncrawl.s3.amazonaws.com/cc-index/table/cc-main/index.html
+
+---------------
+More examples to try:
+https://github.com/commoncrawl/cc-warc-examples
+
+
+A bit outdated?
+https://www.journaldev.com/20342/apache-spark-example-word-count-program-java
+https://www.journaldev.com/20261/apache-spark
+
+--------
+
+sudo apt-get install maven
+(or sudo apt update
+sudo apt install maven)
+git clone https://github.com/commoncrawl/cc-index-table.git
+cd cc-index-table
+mvn package
+vagrant@node1:~/cc-index-table$ ./src/script/convert_url_index.sh https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-30/indexes/cdx-00000.gz hdfs:///user/vagrant/cc-index-table
+
+
+
+
+spark:
+https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-shell.html
+
+============
+Dr Bainbridge found the following vagrant file that will set up hadoop and spark presumably for cluster computing:
+
+https://github.com/martinprobson/vagrant-hadoop-hive-spark
+
+Vagrant:
+	* Guide: https://www.vagrantup.com/intro/getting-started/index.html
+	* Common cmds: https://blog.ipswitch.com/5-vagrant-commands-you-need-to-know
+	* vagrant reload = vagrant halt + vagrant up https://www.vagrantup.com/docs/cli/reload.html
+	* https://stackoverflow.com/questions/46903623/how-to-use-firefox-ui-in-vagrant-box
+	* https://stackoverflow.com/questions/22651399/how-to-install-firefox-in-precise64-vagrant-box
+	  sudo apt-get -y install firefox
+	* vagrant install emacs: https://medium.com/@AnnaJS15/getting-started-with-virtualbox-and-vagrant-8d98aa271d2a
+
+	* hadoop conf: sudo vi /usr/local/hadoop-2.7.6/etc/hadoop/mapred-site.xml
+	* https://data-flair.training/forums/topic/mkdir-cannot-create-directory-data-name-node-is-in-safe-mode/
+---
+==> node1: Forwarding ports...
+    node1: 8080 (guest) => 8081 (host) (adapter 1)
+    node1: 8088 (guest) => 8089 (host) (adapter 1)
+    node1: 9083 (guest) => 9084 (host) (adapter 1)
+    node1: 4040 (guest) => 4041 (host) (adapter 1)
+    node1: 18888 (guest) => 18889 (host) (adapter 1)
+    node1: 16010 (guest) => 16011 (host) (adapter 1)
+    node1: 22 (guest) => 2200 (host) (adapter 1)
+==> node1: Running 'pre-boot' VM customizations...
+
+
+==> node1: Checking for guest additions in VM...
+    node1: The guest additions on this VM do not match the installed version of
+    node1: VirtualBox! In most cases this is fine, but in rare cases it can
+    node1: prevent things such as shared folders from working properly. If you see
+    node1: shared folder errors, please make sure the guest additions within the
+    node1: virtual machine match the version of VirtualBox you have installed on
+    node1: your host and reload your VM.
+    node1: 
+    node1: Guest Additions Version: 5.1.38
+    node1: VirtualBox Version: 5.2
+
+------------
Index: /other-projects/maori-lang-detection/MoreReading/WebScraping.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/WebScraping.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/MoreReading/WebScraping.txt	(revision 33635)
@@ -0,0 +1,10 @@
+
+http://www.basicsbehind.com/extract-text-webpage/
+	http://www.l3s.de/~kohlschuetter/publications/wsdm187-kohlschuetter.pdf
+
+
+https://jsoup.org/
+https://uhack-guide.readthedocs.io/en/latest/technical/scraping/
+
+https://blog.ouseful.info/2015/02/09/getting-text-of-anything-docs-pdfs-images-using-apache-tika/
+	https://tika.apache.org/1.20/examples.html
Index: /other-projects/maori-lang-detection/MoreReading/crawling-Nutch.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/crawling-Nutch.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/MoreReading/crawling-Nutch.txt	(revision 33635)
@@ -0,0 +1,510 @@
+https://codereview.stackexchange.com/questions/198343/crawl-and-gather-all-the-urls-recursively-in-a-domain
+http://lucene.472066.n3.nabble.com/Using-nutch-just-for-the-crawler-fetcher-td611918.html
+
+https://www.quora.com/What-are-some-Web-crawler-tips-to-avoid-crawler-traps
+
+https://cwiki.apache.org/confluence/display/nutch/
+https://cwiki.apache.org/confluence/display/NUTCH/Nutch2Crawling
+https://cwiki.apache.org/confluence/display/nutch/ReaddbOptions
+
+https://moz.com/top500
+-----------
+NUTCH
+-----------
+https://stackoverflow.com/questions/35449673/nutch-and-solr-indexing-blacklist-domain
+	https://nutch.apache.org/apidocs/apidocs-1.6/org/apache/nutch/urlfilter/domainblacklist/DomainBlacklistURLFilter.html
+	
+https://lucene.472066.n3.nabble.com/blacklist-for-crawling-td618343.html
+https://lucene.472066.n3.nabble.com/Content-of-size-X-was-truncated-to-Y-td4003517.html
+
+
+Google: nutch mirror web site
+https://stackoverflow.com/questions/33354460/nutch-clone-website
+[https://stackoverflow.com/questions/35714897/nutch-not-crawling-entire-website
+fetch -all seems to be a nutch v2 thing?]
+
+Google (30 Sep): site mirroring with nutch
+https://grokbase.com/t/nutch/user/125sfbg0pt/using-nutch-for-web-site-mirroring
+https://lucene.472066.n3.nabble.com/Using-nutch-just-for-the-crawler-fetcher-td611918.html
+http://www.cs.ucy.ac.cy/courses/EPL660/lectures/lab6.pdf
+	slide p.5 onwards
+
+crawler softw options: https://repositorio.iscte-iul.pt/bitstream/10071/2871/1/Building%20a%20Scalable%20Index%20and%20Web%20Search%20Engine%20for%20Music%20on.pdf
+See also p.20. HTTrack
+
+
+Google: nutch performance tuning
+* https://stackoverflow.com/questions/24383212/apache-nutch-performance-tuning-for-whole-web-crawling
+* https://stackoverflow.com/questions/4871972/how-to-speed-up-crawling-in-nutch
+* https://cwiki.apache.org/confluence/display/nutch/OptimizingCrawls
+
+NUTCH INSTALLATION:
+* Nutch v1: https://cwiki.apache.org/confluence/display/nutch/NutchTutorial#NutchTutorial-SetupSolrforsearch
+
+Nutch v2 installation and set up:
+* https://cwiki.apache.org/confluence/display/NUTCH/Nutch2Tutorial
+* https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781783286850/1/ch01lvl1sec09/installing-and-configuring-apache-nutch
+
+
+Nutch doesn't work with spark (yet):
+https://stackoverflow.com/questions/29950299/distributed-web-crawling-using-apache-spark-is-it-possible
+
+SOLR:
+* Query syntax: http://www.solrtutorial.com/solr-query-syntax.html
+* Deleting a core: https://factorpad.com/tech/solr/reference/solr-delete.html
+
+
+* If you change a nutch 2 configuration, https://stackoverflow.com/questions/16401667/java-lang-classnotfoundexception-org-apache-gora-hbase-store-hbasestore
+explains you can rebuild nutch with:
+	 cd <apache-nutch>
+	 ant clean
+	 ant runtime
+----------------------------------
+Apache Nutch 2 with newer HBase
+
+hbase-common-1.4.8.jar
+
+1. hbase jar files need to go into runtime/local/lib
+
+But not slf4j-log4j12-1.7.10.jar (there's already a slf4j-log4j12-1.7.5.jar) - so remove that from runtime/local/lib after copying it over.
+
+2. https://stackoverflow.com/questions/46340416/how-to-compile-nutch-2-3-1-with-hbase-1-2-6
+	https://stackoverflow.com/questions/39834423/apache-nutch-fetcherjob-throws-nosuchelementexception-deep-in-gora/39837926#39837926
+
+Unfortunately, the page https://paste.apache.org/jjqz referred to above that contains patches for using Gora 0.7 is no longer available.
+
+http://mail-archives.apache.org/mod_mbox/nutch-user/201602.mbox/%3C56B2EA23.8080801@cisinlabs.com%3E
+
+https://www.mail-archive.com/user@nutch.apache.org/msg14245.html
+
+------------------------------------------------------------------------------
+Other way: Nutch on its own vagrant with specified hbase or nutch with mongodb
+------------------------------------------------------------------------------
+* https://lobster1234.github.io/2017/08/14/search-with-nutch-mongodb-solr/
+* https://waue0920.wordpress.com/2016/08/25/nutch-2-3-1-hbase-0-98-hadoop-2-5-solr-4-10-3/
+
+The older but recommended hbase 0.98.21 for hadoop 2 can be downloaded from https://archive.apache.org/dist/hbase/0.98.21/
+
+-----
+HBASE commands
+/usr/local/hbase/bin/hbase shell
+https://learnhbase.net/2013/03/02/hbase-shell-commands/
+http://dwgeek.com/read-hbase-table-using-hbase-shell-get-command.html/
+dropping tables: https://www.tutorialspoint.com/hbase/hbase_drop_table.htm
+
+> list
+
+davidbHomePage_webpage is a table
+
+> get 'davidbHomePage_webpage', '1'
+
+Solution to get a working nutch2:
+get http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/vagrant-for-nutch2.tar.gz
+And follow the instructions in my README file in there.
+
+---------------------------------------------------------------------
+ALTERNATIVES TO NUTCH - looking for site mirroring capabilities
+---------------------------------------------------------------------
+=> https://anarc.at/services/archive/web/ 
+	Autistici's crawl [https://git.autistici.org/ale/crawl] needs Go:
+	https://medium.com/better-programming/install-go-1-11-on-ubuntu-18-04-16-04-lts-8c098c503c5f	
+	https://guide.freecodecamp.org/go/installing-go/ubuntu-apt-get/
+		To uninstall: https://medium.com/@firebitsbr/how-to-uninstall-from-the-apt-manager-uninstall-just-golang-go-from-universe-debian-ubuntu-82d6a3692cbd
+	https://tecadmin.net/install-go-on-ubuntu/ [our vagrant VMs are Ubuntu 16.04 LTS, as discovered by running the cmd "lsb_release -a"]
+https://alternativeto.net/software/apache-nutch/
+https://alternativeto.net/software/wget/
+https://github.com/ArchiveTeam/grab-site/blob/master/README.md#inspecting-warc-files-in-the-terminal
+https://github.com/ArchiveTeam/wpull
+
+-------------------
+
+Running nutch 2.x
+
+-------------------
+
+LINKS
+
+https://lucene.472066.n3.nabble.com/Nutch-2-x-readdb-command-dump-td4033937.html
+https://cwiki.apache.org/confluence/display/nutch/ReaddbOptions
+
+
+https://lobster1234.github.io/2017/08/14/search-with-nutch-mongodb-solr/ ## most useful for running nutch 2.x crawls
+
+https://www.mobomo.com/2017/06/the-basics-working-with-nutch-2-x/
+	"Fetch
+
+	This is where the magic happens.  During the fetch step, Nutch crawls the urls selected in the generate step.  The most important argument you need is -threads: this sets the number of fetcher threads per task.  Increasing this will make crawling faster, but setting it too high can overwhelm a site and it might shut out your crawler, as well as take up too much memory from your machine.  Run it like this:
+	$ nutch fetch -threads 50"
+
+
+https://examples.javacodegeeks.com/enterprise-java/apache-hadoop/apache-hadoop-nutch-tutorial/
+https://www.yegor256.com/2019/04/17/nutch-from-java.html
+
+http://nutch.sourceforge.net/docs/en/tutorial.html
+
+Intranet: Configuration
+To configure things for intranet crawling you must:
+
+    Create a flat file of root urls. For example, to crawl the nutch.org site you might start with a file named urls containing just the Nutch home page. All other Nutch pages should be reachable from this page. The urls file would thus look like:
+
+    http://www.nutch.org/
+
+    Edit the file conf/crawl-urlfilter.txt and replace MY.DOMAIN.NAME with the name of the domain you wish to crawl. For example, if you wished to limit the crawl to the nutch.org domain, the line should read:
+
+    +^http://([a-z0-9]*\.)*nutch.org/
+
+    This will include any url in the domain nutch.org.
+
+Intranet: Running the Crawl
+Once things are configured, running the crawl is easy. Just use the crawl command. Its options include:
+
+    -dir dir names the directory to put the crawl in.
+    -depth depth indicates the link depth from the root page that should be crawled.
+    -delay delay determines the number of seconds between accesses to each host.
+    -threads threads determines the number of threads that will fetch in parallel.
+
+For example, a typical call might be:
+
+bin/nutch crawl urls -dir crawl.test -depth 3 >& crawl.log
+
+Typically one starts testing one's configuration by crawling at low depths, and watching the output to check that desired pages are found. Once one is more confident of the configuration, then an appropriate depth for a full crawl is around 10. <===========
+
+Once crawling has completed, one can skip to the Searching section below.
+
+
+-----------------------------------
+Actually running nutch 2.x - steps
+-----------------------------------
+MANUALLY GOING THROUGH THE CYCLE 3 TIMES:
+
+cd ~/apache-nutch-2.3.1/runtime/local
+
+./bin/nutch inject urls
+
+./bin/nutch generate -topN 50
+./bin/nutch fetch -all
+./bin/nutch parse -all
+./bin/nutch updatedb -all
+
+./bin/nutch generate -topN 50
+./bin/nutch fetch -all
+./bin/nutch parse -all
+./bin/nutch updatedb -all
+
+./bin/nutch generate -topN 50
+./bin/nutch fetch -all
+./bin/nutch parse -all
+./bin/nutch updatedb -all
+
+Dump output on local filesystem:
+	rm -rf /tmp/bla
+	./bin/nutch readdb -dump /tmp/bla [-crawlId ID -text]
+	less /tmp/bla/part-r-00000
+
+To dump output on local filesystem:
+   Need hdfs host name if sending/dumping nutch crawl output to a location on hdfs
+   Host is defined in /usr/local/hadoop/etc/hadoop/core-site.xml for property fs.defaultFS, (https://stackoverflow.com/questions/27956973/java-io-ioexception-incomplete-hdfs-uri-no-host)
+   host is hdfs://node2/ in this case.
+   So:
+
+	hdfs dfs -rmdir /user/vagrant/dump
+	XXX ./bin/nutch readdb -dump user/vagrant/dump -text ### won't work
+	XXX ./bin/nutch readdb -dump hdfs:///user/vagrant/dump -text ### won't work
+	./bin/nutch readdb -dump hdfs://node2/user/vagrant/dump -text
+
+
+USING THE SCRIPT TO ATTEMPT TO CRAWL A SITE
+* Choosing to repeat the cycle 10 times because, as per http://nutch.sourceforge.net/docs/en/tutorial.html
+
+"Typically one starts testing one's configuration by crawling at low depths, and watching the output to check that desired pages are found. Once one is more confident of the configuration, then an appropriate depth for a full crawl is around 10." 
+
+* Use the ./bin/crawl script, provide the seed urls dir, the crawlId and number of times to repeat = 10 
+vagrant@node2:~/apache-nutch-2.3.1/runtime/local$ ./bin/crawl urls davidbHomePage 10
+
+
+* View the downloaded crawls.
+This time need to provide crawlId to readdb, in order to get a dump of its text contents:
+   hdfs dfs -rm -r hdfs://node2/user/vagrant/dump2
+   ./bin/nutch readdb -dump hdfs://node2/user/vagrant/dump2 -text -crawlId davidbHomePage
+
+* View the contents:
+hdfs dfs -cat hdfs://node2/user/vagrant/dump2/part-r-*
+
+
+* FIND OUT NUMBER OF URLS DOWNLOADED FOR THE SITE:
+vagrant@node2:~/apache-nutch-2.3.1/runtime/local$ ./bin/nutch readdb -stats -crawlId davidbHomePage
+WebTable statistics start
+Statistics for WebTable: 
+retry 0:	44
+status 5 (status_redir_perm):	4
+status 3 (status_gone):	1
+status 2 (status_fetched):	39
+jobs:	{[davidbHomePage]db_stats-job_local647846559_0001={jobName=[davidbHomePage]db_stats, jobID=job_local647846559_0001, counters={Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=135, REDUCE_INPUT_RECORDS=8, SPILLED_RECORDS=16, MERGED_MAP_OUTPUTS=1, VIRTUAL_MEMORY_BYTES=0, MAP_INPUT_RECORDS=44, SPLIT_RAW_BYTES=935, FAILED_SHUFFLE=0, MAP_OUTPUT_BYTES=2332, REDUCE_SHUFFLE_BYTES=135, PHYSICAL_MEMORY_BYTES=0, GC_TIME_MILLIS=0, REDUCE_INPUT_GROUPS=8, COMBINE_OUTPUT_RECORDS=8, SHUFFLED_MAPS=1, REDUCE_OUTPUT_RECORDS=8, MAP_OUTPUT_RECORDS=176, COMBINE_INPUT_RECORDS=176, CPU_MILLISECONDS=0, COMMITTED_HEAP_BYTES=595591168}, File Input Format Counters ={BYTES_READ=0}, File System Counters={FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, FILE_READ_OPS=0, FILE_BYTES_WRITTEN=1788140, FILE_BYTES_READ=1223290}, File Output Format Counters ={BYTES_WRITTEN=275}, Shuffle Errors={CONNECTION=0, WRONG_LENGTH=0, BAD_ID=0, WRONG_MAP=0, WRONG_REDUCE=0, IO_ERROR=0}}}}
+TOTAL urls:	44
+max score:	1.0
+avg score:	0.022727273
+min score:	0.0
+WebTable statistics: done
+
+------------------------------------
+STOPPING CONDITION
+Seems inbuilt
+* When I tell it to cycle 15 times, it stops after 6 cycles saying no more URLs to fetch:
+
+vagrant@node2:~/apache-nutch-2.3.1/runtime/local$ ./bin/crawl urls davidbHomePage2 15
+---
+No SOLRURL specified. Skipping indexing.
+Injecting seed URLs
+
+...
+
+Thu Oct 3 09:22:23 UTC 2019 : Iteration 6 of 15
+Generating batchId
+Generating a new fetchlist
+...
+Generating batchId
+Generating a new fetchlist
+/home/vagrant/apache-nutch-2.3.1/runtime/local/bin/nutch generate -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D mapred.reduce.tasks.speculative.execution=false -D mapred.map.tasks.speculative.execution=false -D mapred.compress.map.output=true -topN 50000 -noNorm -noFilter -adddays 0 -crawlId davidbHomePage2 -batchId 1570094569-27637
+GeneratorJob: starting at 2019-10-03 09:22:49
+GeneratorJob: Selecting best-scoring urls due for fetch.
+GeneratorJob: starting
+GeneratorJob: filtering: false
+GeneratorJob: normalizing: false
+GeneratorJob: topN: 50000
+GeneratorJob: finished at 2019-10-03 09:22:52, time elapsed: 00:00:02
+GeneratorJob: generated batch id: 1570094569-27637 containing 0 URLs
+Generate returned 1 (no new segments created)
+Escaping loop: no more URLs to fetch now
+vagrant@node2:~/apache-nutch-2.3.1/runtime/local$ 
+---
+
+* running readdb -stats show 44 URLs fetched, just as first time (when crawlId had been "davidbHomePage"):
+
+vagrant@node2:~/apache-nutch-2.3.1/runtime/local$ ./bin/nutch readdb -stats -crawlId davidbHomePage2
+---
+WebTable statistics start
+Statistics for WebTable: 
+retry 0:	44
+status 5 (status_redir_perm):	4
+status 3 (status_gone):	1
+status 2 (status_fetched):	39
+jobs:	{[davidbHomePage2]db_stats-job_local985519583_0001={jobName=[davidbHomePage2]db_stats, jobID=job_local985519583_0001, counters={Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=135, REDUCE_INPUT_RECORDS=8, SPILLED_RECORDS=16, MERGED_MAP_OUTPUTS=1, VIRTUAL_MEMORY_BYTES=0, MAP_INPUT_RECORDS=44, SPLIT_RAW_BYTES=935, FAILED_SHUFFLE=0, MAP_OUTPUT_BYTES=2332, REDUCE_SHUFFLE_BYTES=135, PHYSICAL_MEMORY_BYTES=0, GC_TIME_MILLIS=4, REDUCE_INPUT_GROUPS=8, COMBINE_OUTPUT_RECORDS=8, SHUFFLED_MAPS=1, REDUCE_OUTPUT_RECORDS=8, MAP_OUTPUT_RECORDS=176, COMBINE_INPUT_RECORDS=176, CPU_MILLISECONDS=0, COMMITTED_HEAP_BYTES=552599552}, File Input Format Counters ={BYTES_READ=0}, File System Counters={FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, FILE_READ_OPS=0, FILE_BYTES_WRITTEN=1788152, FILE_BYTES_READ=1223290}, File Output Format Counters ={BYTES_WRITTEN=275}, Shuffle Errors={CONNECTION=0, WRONG_LENGTH=0, BAD_ID=0, WRONG_MAP=0, WRONG_REDUCE=0, IO_ERROR=0}}}}
+TOTAL urls:	44
+---
+
+----------------------------------------------------------------------
+	Testing URLFilters: testing a URL to see if it's accepted
+----------------------------------------------------------------------
+Use the command
+	./bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined 
+(mentioned at https://lucene.472066.n3.nabble.com/Correct-syntax-for-regex-urlfilter-txt-trying-to-exclude-single-path-results-td3600376.html)
+
+Use as follows:
+
+	cd apache-nutch-2.3.1/runtime/local
+
+	./bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined 
+
+Then paste the URL you want to test, press Enter.
+	A + in front of response means accepted
+	A - in front of response means rejected.
+Can continue pasting URLs to test against filters until you send Ctrl-D to terminate input.
+
+
+
+
+-------------------
+Dr Nichols's suggestion: can store listing of potential product sites to inspect by checking url for /mi in combination with whether the domain's IP geolocates to OUTSIDE New Zealand (tld nz).
+* https://stackoverflow.com/questions/1415851/best-way-to-get-geo-location-in-java
+  - https://mvnrepository.com/artifact/com.maxmind.geoip/geoip-api/1.2.10
+  - older .dat.gz file is archived at https://web.archive.org/web/20180917084618/http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
+  - and newer geo country data at https://dev.maxmind.com/geoip/geoip2/geolite2/
+* https://dev.maxmind.com/geoip/geoip2/geolite2/
+* older GeoIp API (has LookupService): https://github.com/maxmind/geoip-api-java
+* Newer GeoIp2 API: https://dev.maxmind.com/geoip/geoip2/downloadable/#MaxMind_APIs
+	and https://maxmind.github.io/GeoIP2-java/doc/v2.12.0/
+* https://maxmind.github.io/GeoIP2-java/
+* https://github.com/AtlasOfLivingAustralia/ala-hub/issues/11
+
+
+---
+https://check-host.net/ip-info
+https://ipinfo.info/html/ip_checker.php
+
+
+
+----------
+MongoDB
+Installation:
+	https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
+		https://docs.mongodb.com/manual/administration/install-on-linux/
+	https://hevodata.com/blog/install-mongodb-on-ubuntu/
+	https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04
+	CENTOS (Analytics): https://tecadmin.net/install-mongodb-on-centos/
+	FROM SOURCE: https://github.com/mongodb/mongo/wiki/Build-Mongodb-From-Source
+GUI:
+	https://robomongo.org/
+	Robomongo is Robo 3T now
+
+https://www.tutorialspoint.com/mongodb/mongodb_java.htm
+JAR FILE:
+	http://central.maven.org/maven2/org/mongodb/mongo-java-driver/
+	https://mongodb.github.io/mongo-java-driver/
+
+
+INSTALLING THE MONGODB SERVER AND MONGO CLIENT ON LINUX
+Need to have sudo and root powers.
+
+https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
+http://www.programmersought.com/article/6500308940/
+
+   52  sudo apt-get install mongodb-clients
+   53  mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
+
+Failed with
+	Error: HostAndPort: host is empty at src/mongo/shell/mongo.js:148
+	exception: connect failed
+
+This is due to a version incompatibility between Client and mongodb Server.
+The solution is to follow instructions at http://www.programmersought.com/article/6500308940/
+and then https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
+as below:
+
+   54  sudo apt-get purge mongodb-clients
+   55  sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
+   56  echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
+   57  sudo apt-get update
+   58  sudo apt-get install mongodb-clients
+   59  mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
+(still doesn't work)
+   60  sudo apt-get install -y mongodb-org
+The above ensures an up to date mongo client but installs the mongodb server too. Maybe this is the only step that is needed to install up-to-date mongo client and mongodb server?
+   72  sudo service mongod status
+
+  103  sudo service mongod start
+"mongod" stands for mongo-daemon. This runs the mongo db server listening for client connections
+  104  sudo service mongod status
+   88  sudo service mongod stop
+
+
+RUNNING AND USING THE MONGO CLIENT SHELL:
+Among the many things you can do with the Mongo client shell, one can use it to find the mongo client version (which is the version of the shell) and the mongo db version.
+
+To run the mongo client shell WITHOUT loading a db:
+
+
+	wharariki:[880]/Scratch/ak19/gs3-extensions/maori-lang-detection>mongo --shell -nodb
+	MongoDB shell version: 2.6.10 <<<<<<<<<-------------------<<<< MONGO CLIENT VERSION
+
+	type "help" for help
+	> help
+		db.help()                    help on db methods
+		db.mycoll.help()             help on collection methods
+		sh.help()                    sharding helpers
+		rs.help()                    replica set helpers
+		help admin                   administrative help
+		help connect                 connecting to a db help
+		help keys                    key shortcuts
+		help misc                    misc things to know
+		help mr                      mapreduce
+
+		show dbs                     show database names
+		show collections             show collections in current database
+		show users                   show users in current database
+		show profile                 show most recent system.profile entries with time >= 1ms
+		show logs                    show the accessible logger names
+		show log [name]              prints out the last segment of log in memory, 'global' is default
+		use <db_name>                set current database
+		db.foo.find()                list objects in collection foo
+		db.foo.find( { a : 1 } )     list objects in foo where a == 1
+		it                           result of the last line evaluated; use to further iterate
+		DBQuery.shellBatchSize = x   set default number of items to display on shell
+		exit                         quit the mongo shell
+
+	> help connect
+
+	Normally one specifies the server on the mongo shell command line.  Run mongo --help to see those options.
+	Additional connections may be opened:
+
+	    var x = new Mongo('host[:port]');
+	    var mydb = x.getDB('mydb');
+	  or
+	    var mydb = connect('host[:port]/mydb');
+
+	Note: the REPL prompt only auto-reports getLastError() for the shell command line connection.
+
+	Getting help on connect options: 
+
+	> var x = new Mongo('mongodb.cms.waikato.ac.nz:27017');
+	> var mydb = x.getDB('anupama');
+
+	> mydb.connect.help()
+	DBCollection help
+		db.connect.find().help() - show DBCursor help
+		db.connect.count()
+		db.connect.copyTo(newColl) - duplicates collection by copying all documents to newColl; no indexes are copied.
+		db.connect.convertToCapped(maxBytes) - calls {convertToCapped:'connect', size:maxBytes}} command
+		db.connect.dataSize()
+		db.connect.distinct( key ) - e.g. db.connect.distinct( 'x' )
+		db.connect.drop() drop the collection
+		db.connect.dropIndex(index) - e.g. db.connect.dropIndex( "indexName" ) or db.connect.dropIndex( { "indexKey" : 1 } )
+		db.connect.dropIndexes()
+		db.connect.ensureIndex(keypattern[,options]) - options is an object with these possible fields: name, unique, dropDups
+		db.connect.reIndex()
+		db.connect.find([query],[fields]) - query is an optional query filter. fields is optional set of fields to return.
+			                                      e.g. db.connect.find( {x:77} , {name:1, x:1} )
+		db.connect.find(...).count()
+		db.connect.find(...).limit(n)
+		db.connect.find(...).skip(n)
+		db.connect.find(...).sort(...)
+		db.connect.findOne([query])
+		db.connect.findAndModify( { update : ... , remove : bool [, query: {}, sort: {}, 'new': false] } )
+		db.connect.getDB() get DB object associated with collection
+		db.connect.getPlanCache() get query plan cache associated with collection
+		db.connect.getIndexes()
+		db.connect.group( { key : ..., initial: ..., reduce : ...[, cond: ...] } )
+		db.connect.insert(obj)
+		db.connect.mapReduce( mapFunction , reduceFunction , <optional params> )
+		db.connect.aggregate( [pipeline], <optional params> ) - performs an aggregation on a collection; returns a cursor
+		db.connect.remove(query)
+		db.connect.renameCollection( newName , <dropTarget> ) renames the collection.
+		db.connect.runCommand( name , <options> ) runs a db command with the given name where the first param is the collection name
+		db.connect.save(obj)
+		db.connect.stats()
+		db.connect.storageSize() - includes free space allocated to this collection
+		db.connect.totalIndexSize() - size in bytes of all the indexes
+		db.connect.totalSize() - storage allocated for all data and indexes
+		db.connect.update(query, object[, upsert_bool, multi_bool]) - instead of two flags, you can pass an object with fields: upsert, multi
+		db.connect.validate( <full> ) - SLOW
+		db.connect.getShardVersion() - only for use with sharding
+		db.connect.getShardDistribution() - prints statistics about data distribution in the cluster
+		db.connect.getSplitKeysForChunks( <maxChunkSize> ) - calculates split points over all chunks and returns splitter function
+		db.connect.getWriteConcern() - returns the write concern used for any operations on this collection, inherited from server/db if set
+		db.connect.setWriteConcern( <write concern doc> ) - sets the write concern for writes to the collection
+		db.connect.unsetWriteConcern( <write concern doc> ) - unsets the write concern for writes to the collection
+	> mydb.version()
+	4.0.13 <<<<<<<<<-------------------<<<< MONGODB SERVER VERSION
+
+(Check Mongo server version: https://stackoverflow.com/questions/38160412/how-to-find-the-exact-version-of-installed-mongodb)
+
+Finally we now know the mongodb server version: 4.0.13
+This version didn't work with our mongo client (shell) version of 2.6.10. And that's we had to upgrade the client.
+
+
+INSTALLATION MONGO-DB AND CLIENT
+FROM: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
+	wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
+	echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
+	sudo apt-get update
+	sudo apt-get install -y mongodb-org
+
+UNINSTALLING
+	https://www.anintegratedworld.com/uninstall-mongodb-in-ubuntu-via-command-line-in-3-easy-steps/
+
+
+MONGO DB ROBO 3T
+1. Download "Double Pack" from https://robomongo.org/
+2. Untar its contents. Then untar the tarball in that.
+3. Run:
+	wharariki:[110]~/Downloads/robo3t-1.3.1-linux-x86_64-7419c406>./bin/robo3t
+
Index: /other-projects/maori-lang-detection/MoreReading/macrons_with_emacs.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/macrons_with_emacs.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/MoreReading/macrons_with_emacs.txt	(revision 33635)
@@ -0,0 +1,23 @@
+Adding Macrons using emacs:
+
+https://www.stat.auckland.ac.nz/~paul/Reports/maori/maori.html
+ The macron is the only accent required for written MÄori and the accent can only be applied to vowels, so the full set of accented characters are:
+lower case a, with macron 	Ä 		upper case A, with macron 	Ä
+lower case e, with macron 	Ä 		upper case E, with macron 	Ä
+lower case i, with macron 	Ä« 		upper case I, with macron 	Äª
+lower case o, with macron 	Å 		upper case O, with macron 	Å
+lower case u, with macron 	Å« 		upper case U, with macron 	Åª 
+
+http://emacs.1067599.n8.nabble.com/Entering-vowels-with-macrons-td72136.html
+	Ctrl+\
+	type rfc1345 (and enter)
+	type &a- to get a-macron
+	Then Ctrl+\ to toggle back to default input
+	(Can thereafter toggle with Ctrl+\ to get back to rfc1345 input method)
+
+
+OTHER:
+https://sachachua.com/blog/2011/04/writing-macrons-linux-latin-pronunciation/
+To add macrons: Ctrl-\ "latin-alt-postfix". But doesn't have all macronised vowels as used in Te Reo.
+Then Ctrl-\ to get default input editor back.
+https://www.gnu.org/software/emacs/manual/html_node/emacs/Select-Input-Method.html
Index: /other-projects/maori-lang-detection/README.txt
===================================================================
--- /other-projects/maori-lang-detection/README.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/README.txt	(revision 33635)
@@ -0,0 +1,261 @@
+-------------------------
+BASIC README: QUICK SETUP
+-------------------------
+0. The code and its necessary helper files and libraries, and this README, live at:
+    http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection
+
+You can checkout from svn with:
+    svn co http://svn.greenstone.org/gs3-extensions/maori-lang-detection
+
+This checkout contains:
+- tarball of the OpenNLP version that was current at the time of the commit, but you can also get it from its original site http://opennlp.apache.org/download.html
+- the folder "models-trainingdata-and-sampletxts", itself containing:
+	- langdetect-183.bin: The LanguageDetectionModel to be used by OpenNLP is also included (again, current at the time of commit), but you can get it from http://opennlp.apache.org/models.html (direct link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin)
+	- mri-sent_trained.bin: our own custom generated model for Maori Sentence Detection
+	- mri-sent.train: the training text file for generating the mri-sent_trained.bin Maori Sentence Detection model
+	- sample_mri_paragraphs.txt: contains some text to test the Maori Sentence Detection model on. Its content is from Niupepa collection page http://www.nzdl.org/cgi-bin/library?gg=text&e=p-00000-00---off-0niupepa--00-0----0-10-0---0---0direct-10---4-------0-1l--11-en-50---20-about---00-0-1-00-0-0-11-1-0utfZz-8-00&a=p&p=about&l=mi&nw=utf-8)
+	- sample_maori_shorttext.txt: to test the MaoriTextDetector.java with
+- "src" folder for Java classes, currently just MaoriTextDetector.java and its classfile. MaoriTextDetector.java uses the aforementioned LanguageDetectionModel, langdetect-183.bin, to detect whether input text from a file or stdin is in Maori or not
+- gen_SentenceDetection_model.sh, our custom script that generates both the mri-sent.train and model mri-sent_trained.bin files mentioned above
+	- the script works on opennlp's leipzig corpus of 100k Maori sentences from 2011 to get its sample sentences into the correct format in the mri-sent.train file
+	- from this file containing training sentences, it generates the Sentence Detector Model, mri-sent_trained.bin
+- mri-opennlp-corpus.tar.gz: a tarball containing the 100k Maori sentences opennlp corpus checked out with svn in its original directory structure from https://svn.apache.org/repos/bigdata/opennlp/trunk/mri_web_2011_100K-sentences.txt
+
+
+1. Once you've svn checked out the maori-lang-detection from gs3-extensions, create a folder called "models" and put the files langdetect-183.bin and mri-sent_trained.bin from the folder "models-trainingdata-and-sampletxts" into it.
+(These are just zip files, but have to remain with the .bin extension in order for OpenNLP to use them. If you ever wish to see the contents of such a .bin file, you can rename to .zip and use the Archive Manager or other Zip tool to inspect the contents.)
+
+
+2. Next extract the apache-opennlp-1.9.1-bin.tar.gz.
+This will create a folder called apache-opennlp-1.9.1. Move the "models" folder you created in step 1 into this folder.
+
+
+3. Before you can compile or run the MaoriTextDetector program, you always have to prepare a terminal by setting up the environment for OpenNLP as follows:
+   cd /type/here/path/to/your/extracted/apache-opennlp-1.9.1
+   export OPENNLP_HOME=`pwd`
+
+
+4. If you want to recompile, go up into the checked out maori-lang-detection folder's "src" subfolder. To compile, make sure you have the JDK7+ bin folder on your PATH environment variable.
+Still in the SAME terminal as where you set up the OPENNLP_HOME environment in step 3, you can now run:
+      maori-lang-detection/src$ javac -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/MaoriTextDetector.java
+
+5. To run the MaoriTextDetector program, you will need the JDK or JRE 7+ bin folder on your PATH and still in the SAME terminal as where you set up the OPENNLP_HOME environment in step 3,
+type one of the following:
+   maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" org.greenstone.atea.MaoriTextDetector --help
+   (prints the usage, including other options)  			    
+ 
+   maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" org.greenstone.atea.MaoriTextDetector --file &lt;full/path/to/textfile&gt;
+ 
+   maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" org.greenstone.atea.MaoriTextDetector -
+       which expects text to stream in from standard input.
+       If entering text manually, then remember to press Ctrl-D to indicate the end of StdIn as usual.
+
+
+For links to background reading materials, see the OLD README section further below.
+
+
+NOTE: The OpenNLP Language Detection Model can detect non-macronised MÄori text too,
+but as anticipated, the same text produces a lower confidence level for the language prediction. Compare:
+
+$maori-lang-detection/src>java -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" MaoriTextDetector -
+   Waiting to read text from STDIN... (press Ctrl-D when done entering text)>
+   Ko tenei te Whare Wananga o Waikato e whakatau nei i nga iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o nga maunga whakaruru e tau awhi nei.
+   Best language: mri
+   Best language confidence: 0.5959533972070814
+   Exitting program with returnVal 0...
+
+$maori-lang-detection/src>java -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" MaoriTextDetector -
+   Waiting to read text from STDIN... (press Ctrl-D when done entering text)>
+   Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei.
+   Best language: mri
+   Best language confidence: 0.6825737450092515
+   Exitting program with returnVal 0...
+
+
+-------------------------
+     OLD README
+-------------------------
+
+
+http://opennlp.apache.org/news/model-langdetect-183.html
+Language Detector Model for Apache OpenNLP released
+
+0. Can we make it run
+Can we detect it with with Eng, Fr, NL docs
+
+1. openNLP - is Maori included,
+if not learn how to teach it to recognise Maori
+how you run the training bit of their software and add in Maori language training set
+
+ANSWER: Yes, Maori is included, see https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+
+2. Add in a small program to detect particularly Maori
+
+3. Macron and non-macron input text language recognition support?
+
+-----------------------------------------
+
+OPENNLP LANGUAGE DETECTION READING:
+	General:
+	* https://stackoverflow.com/questions/7670427/how-does-language-detection-work
+	* https://github.com/andreasjansson/language-detection.el	
+	* https://en.wikipedia.org/wiki/ISO_639-3
+
+	Specific:
+	* openNLP download: http://opennlp.apache.org/download.html
+		"Models
+		The models for Apache OpenNLP are found here.
+		The models can be used for testing or getting started, please train your own models for all other use cases."
+
+	* On LanguageDetectionModel for OpenNLP:
+		http://opennlp.apache.org/news/model-langdetect-183.html
+		https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect.classifying
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.langdetect
+	* Good precision etc on detecting Maori: https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/langdetect-183.bin.report.txt
+	* Maybe useful: http://opennlp.sourceforge.net/models-1.5/
+		"Use the links in the table below to download the pre-trained models for the OpenNLP 1.5 series."
+
+
+
+1. Download OpenNLP from https://opennlp.apache.org/download.html.
+I got the binary version. I unzipped it.
+
+
+2. Get the model, langdetect-183.bin, from: http://opennlp.apache.org/models.html
+	Direct download link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin
+The page says:
+	"All models are zip compressed (like a jar file), they must not be uncompressed."
+- So when langdetect-183.bin is renamed to langdetect-183.zip, can inspect the contents.
+- But for using with openNLP, don't rename or unzip it.
+
+
+3. UNNECESSARY:
+I started by following the instructions at the botton of: 
+https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+	Note that the svn checkout from the leipzig data link is too huge. So I restricted it to just Maori, English, Dutch and French text, see further below.
+
+These instructions proved ultimately unnecessary, as the downloaded openNLP model for language detection, langdetect-183.bin, already contains all that
+
+
+Individual downloading of Leipzig corpora for just Maori or specific languages can be found via:
+	http://wortschatz.uni-leipzig.de/en/download/
+
+[svn co https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig]
+https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig/
+Here's the svn web view for the leipzig/data language files: https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig/data/
+
+[Here's the svn web view of apache.org: http://svn.eu.apache.org/viewvc
+Unfortunately this is other apache stuff, including openNLP, but not the leipzig language data files I was after]
+
+
+	svn co --depth immediates --trust-server-cert --non-interactive https://svn.apache.org/repos/bigdata/opennlp/trunk opennlp-corpus
+	cd opennlp-corpus
+	svn up --set-depth immediates --trust-server-cert --non-interactive
+	cd leipzig
+	svn up --set-depth immediates --trust-server-cert --non-interactive
+	cd resources/
+	wharariki:[197]/Scratch/ak19/openNLP-lang-detect/opennlp-corpus/leipzig/resources>svn up --set-depth infinity --trust-server-cert --non-interactive
+	cd ../data
+	svn up --trust-server-cert --non-interactive mri_web_2011_100K-sentences.txt
+	
+	(# UNNECESSARY TO DOWNLOAD:
+	svn up eng_wikipedia_2012_3M-sentences.txt
+	svn up nld_mixed_2012_1M-sentences.txt
+	svn up fra_mixed_2009_1M-sentences.txt
+	)
+
+	cd ..
+	# in opennlp-corpus/leipzig
+	chmod u+x create_langdetect_model.sh
+
+
+	cd /Scratch/ak19/openNLP-lang-detect/apache-opennlp-1.9.1
+	export OPENNLP_HOME=`pwd`
+	./create_langdetect_model.sh mri_web_2011_100K-sentences.txt
+	./create_langdetect_model.sh nld_mixed_2012_1M-sentences.txt
+
+
+4. Attempting to run OpenNLP + LanguageDetectorModel's command line tools for language prediction
+
+I couldn't get prediction to work from the command line, so I ended up writing that part as a Java class. But I could run some part of the command line tools with the instructions on the following pages. 
+
+
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect.classifying
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.langdetect
+	"Usage: opennlp LanguageDetector model < documents"
+
+# must have exported OPENNLP_HOME (maybe add its bin to PATH?)
+# Following the Usage instruction just above:
+	$ cd /Scratch/ak19/openNLP-lang-detect
+	$ ./apache-opennlp-1.9.1/bin/opennlp LanguageDetector langdetect-183.bin < opennlp-corpus-downloaded/mri-nz_web_2017_100K/mri-nz_web_2017_100K-words.txt
+
+	[# Sending all output into a file for inspection:
+	$ ./apache-opennlp-1.9.1/bin/opennlp LanguageDetector langdetect-183.bin < opennlp-corpus-downloaded/mri-nz_web_2017_100K/mri-nz_web_2017_100K-words.txt > bla.txt 2>&1
+	]
+
+
+
+	[# Didn't work, even though the first opennlp manual link above said that the "opennlp LangDetector model" command should take input from stdin:
+
+	./apache-opennlp-1.9.1/bin/opennlp LanguageDetector langdetect-183.bin < "Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei."
+	]
+
+
+UPDATE 24 July 2019: The following worked
+	>$OPENNLP_HOME/bin/opennlp LanguageDetector $OPENNLP_HOME/models/langdetect-183.bin < opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt 
+but it doesn't provide predictions. Not sure I understand what it did other than print the contents of the *sentences.txt file and end with:
+		...
+		Average: 0.1 doc/s 
+		Total: 1 doc
+		Runtime: 8.046s
+		Execution time: 8.719 seconds
+
+
+5. For writing Java code:
+To write the basic code, I followed the Java skeleton examples at
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect.classifying
+
+
+To get the imports correct, I searched for example/tutorial java code on using openNLP with the LanguageDetectorModel 
+* Java code: Import files:
+https://www.tutorialkart.com/opennlp/language-detector-example-in-apache-opennlp/
+* The tutorial link above also covers Java code to train detecting a particular language.
+
+
+apache-opennlp-1.9.1/lib/opennlp-tools-1.9.1.jar contains the imports we want, in particular 
+	import opennlp.tools.langdetect.*;
+	import opennlp.tools.util.*;
+
+6. Wrote the very basic form of MaoriDetector.java class.
+
+To compile and run:
+
+a. export OPENNLP_HOME=/Scratch/ak19/openNLP-lang-detect/apache-opennlp-1.9.1
+b. wharariki:[115]/Scratch/ak19/openNLP-lang-detect/src>javac -cp ".:$OPENNLP_HOME/lib/*" MaoriDetector.java
+c. wharariki:[116]/Scratch/ak19/openNLP-lang-detect/src>java -cp ".:$OPENNLP_HOME/lib/*" MaoriDetector
+
+
+(Though possibly the only jar file needed in $OPENNLP_HOME/lib is opennlp-tools-1.9.1.jar) 
+
+-------------------------------------------------
+READING: GENERAL LINKS FOR LANGUAGE DETECTION
+-------------------------------------------------
+
+https://www.slideshare.net/shuyo/short-text-language-detection-with-infinitygram-12949447
+http://alexott.blogspot.com/2017/10/evaluating-fasttexts-models-for.html
+
+--------------------------------------------------------------------------
+READING: LINKS FOR SENTENCE DETECTION AND TRAINING A SENTENCE DETECTOR
+--------------------------------------------------------------------------
+
+* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.sentdetect
+* SentenceDetectorTrainer java example:
+https://www.tutorialkart.com/opennlp/train-model-sentence-detection-java/
+* SentenceDetectorTrainer and SentenceDetector from commandline:
+https://stackoverflow.com/questions/36516363/sentence-detection-with-opennlp
+
+
Index: /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_index_csv.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_index_csv.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_index_csv.sh	(revision 33635)
@@ -0,0 +1,77 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="$2"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+#    --conf spark.hadoop.fs.s3a.access.key=AKIA2EVQBWSTBJ2M4BLM \
+#    --conf spark.hadoop.fs.s3a.secret.key=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 \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --conf spark.sql.parquet.mergeSchema=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length   
+            FROM ccindex                                                        
+            WHERE crawl = 'CC-MAIN-2019-30' AND subset = 'warc' AND content_languages LIKE '%mri%'" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    hdfs:///user/vagrant/cc-mri-csv
+
Index: /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_subset.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_subset.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_subset.sh	(revision 33635)
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="$2"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+# $SPARK_HOME/bin/spark-submit \
+#     $SPARK_ON_YARN \
+#     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+#     --conf spark.core.connection.ack.wait.timeout=600s \
+#     --conf spark.network.timeout=120s \
+#     --conf spark.task.maxFailures=20 \
+#     --conf spark.shuffle.io.maxRetries=20 \
+#     --conf spark.shuffle.io.retryWait=60s \
+#     --conf spark.driver.memory=$DRIVER_MEM \
+#     --conf spark.executor.memory=$EXECUTOR_MEM \
+#     $SPARK_EXTRA_OPTS \
+#     --num-executors $NUM_EXECUTORS \
+#     --executor-cores $EXECUTOR_CORES \
+#     --executor-memory $EXECUTOR_MEM \
+#     --conf spark.hadoop.parquet.enable.dictionary=true \
+#     --conf spark.sql.parquet.filterPushdown=true \
+#     --conf spark.sql.parquet.mergeSchema=false \
+#     --conf spark.sql.hive.metastorePartitionPruning=true \
+#     --conf spark.hadoop.parquet.enable.summary-metadata=false \
+#     --class org.commoncrawl.spark.CCIndex2Table $_APPJAR \
+#     --outputCompression=$COMPRS \
+#     --outputFormat=$FORMAT $NESTED \
+#     --partitionBy=$PARTITION_BY \
+#     "$DATA" "$OUTPUTDIR"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+   # --conf spark.hadoop.parquet.enable.dictionary=true \
+   # --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   # --conf spark.sql.hive.metastorePartitionPruning=true \
+   # --conf spark.sql.parquet.filterPushdown=true \
+   # --conf spark.sql.parquet.mergeSchema=true \
+
+
+
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --csv hdfs:///user/vagrant/cc-mri-unzipped-csv/cc-mri.csv \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-CC-2019-30 \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   hdfs:///user/vagrant/cc-mri-subset
+
Index: /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_subset_from_scratch.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_subset_from_scratch.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/hadoop-spark-scripts/export_maori_subset_from_scratch.sh	(revision 33635)
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="$2"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+# $SPARK_HOME/bin/spark-submit \
+#     $SPARK_ON_YARN \
+#     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+#     --conf spark.core.connection.ack.wait.timeout=600s \
+#     --conf spark.network.timeout=120s \
+#     --conf spark.task.maxFailures=20 \
+#     --conf spark.shuffle.io.maxRetries=20 \
+#     --conf spark.shuffle.io.retryWait=60s \
+#     --conf spark.driver.memory=$DRIVER_MEM \
+#     --conf spark.executor.memory=$EXECUTOR_MEM \
+#     $SPARK_EXTRA_OPTS \
+#     --num-executors $NUM_EXECUTORS \
+#     --executor-cores $EXECUTOR_CORES \
+#     --executor-memory $EXECUTOR_MEM \
+#     --conf spark.hadoop.parquet.enable.dictionary=true \
+#     --conf spark.sql.parquet.filterPushdown=true \
+#     --conf spark.sql.parquet.mergeSchema=false \
+#     --conf spark.sql.hive.metastorePartitionPruning=true \
+#     --conf spark.hadoop.parquet.enable.summary-metadata=false \
+#     --class org.commoncrawl.spark.CCIndex2Table $_APPJAR \
+#     --outputCompression=$COMPRS \
+#     --outputFormat=$FORMAT $NESTED \
+#     --partitionBy=$PARTITION_BY \
+#     "$DATA" "$OUTPUTDIR"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+	  
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --conf spark.hadoop.parquet.enable.dictionary=true \
+   --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   --conf spark.sql.hive.metastorePartitionPruning=true \
+   --conf spark.sql.parquet.filterPushdown=true \
+   --conf spark.sql.parquet.mergeSchema=true \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --query "SELECT url, warc_filename, warc_record_offset, warc_record_length
+            FROM ccindex
+            WHERE crawl = 'CC-MAIN-2019-30' AND subset = 'warc' AND content_languages LIKE '%mri%'" \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-CC-2019-30 \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   hdfs:///user/vagrant/cc-mri-subset
+#   s3://commoncrawl/cc-index/table/cc-main/warc/ \
+# Since s3:// deprecated, moved to using s3a:// now
Index: /other-projects/maori-lang-detection/bin/script/create-uniq-WET-urls-file.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/create-uniq-WET-urls-file.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/create-uniq-WET-urls-file.sh	(revision 33635)
@@ -0,0 +1,69 @@
+#!/bin/bash
+
+# Need to run this script from the top level folder of this extension
+# This script needs to be provided with the nz-only-TLDs-<date>.txt file produced by running get_commoncrawl_nz_urls.sh
+if [ -z $1 ]; then
+    echo ""
+    echo "Usage: $0 nz-only-TLDs-<date>.txt"
+    echo "   where the nz-only-TLDs-<date>.txt file is produced by running get_commoncrawl_nz_urls.sh."
+    echo ""
+    exit 1
+fi
+
+uniq_urls_file=uniq-tld-nz-WET-urls-`date +%F`.txt
+echo "Creating file $uniq_urls_file containing just the unique WET gz urls for the .nz TLD..."
+
+# cat is unnecessary: https://stackoverflow.com/questions/1915636/is-there-a-way-to-uniq-by-column/1915750
+#    "You can dump the cat! Rather than piping into <the subsequent process>, just let <the subsequent process> read the file using <. Piping through cat is a common unnecessary complication used by novices. For large amounts of data there's a performance effect to be had."
+
+#<$1 cut -d ' ' -f18 | sed -r 's@\.?"(,|})\s*$@@' | sed -r 's@^\s*"@@' > $uniq_urls_file
+#sort -u $uniq_urls_file > tmp.txt
+
+# The first cut grabs the filename field of the json, containing the relative path to the WET file
+# The sed operations remove any starting and terminating double quotes followed by , or }
+# Then we get the uniq urls out of all that.
+
+# The above is imperfect because space is not a proper field separator (some json property values
+# contain spaces)
+# https://unix.stackexchange.com/questions/155952/should-i-use-piping-or-redirection-for-input-to-sort
+tmpfile=tmp_WET_urls.txt
+
+<$1 sed -r 's@^.*"filename":\s*"@@' | sed -r 's@\.?"(,|}).*$@@' > $tmpfile
+sort -u $tmpfile > sorted_WET_urls.txt
+
+rm $tmpfile
+
+# remove all lines containing paths like /crawldiagnostics/ and /robotstxt/
+# https://stackoverflow.com/questions/17049962/delete-line-containing-one-of-multiple-strings
+
+# And convert links like
+#   https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/segments/1563195523840.34/warc/CC-MAIN-20190715175205-20190715200159-00000.warc.gz
+# to:
+#   https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/segments/1563195523840.34/wet/CC-MAIN-20190715175205-20190715200159-00000.warc.wet.gz
+
+awk '!/crawldiagnostics|robotstxt/' sorted_WET_urls.txt | sed -r 's@\/warc\/CC-MAIN-@/wet/CC-MAIN-@' | sed -r 's@\.warc\.gz@.warc.wet.gz@' > $uniq_urls_file
+
+rm sorted_WET_urls.txt
+
+# All urls need to be prefixed with:
+#ccprefix=https://commoncrawl.s3.amazonaws.com/
+
+
+# From each line of json of the input file, we want the filename value filed containing
+# the relative path to the gzipped WET file.
+# The first sed grabs everything after the filename field in each line/json
+# The second sed operation remove any terminating double quotes followed by , or }
+# Then we sort -u to get the uniq urls out of all that.
+
+echo "File $uniq_urls_file containing just the unique commoncrawl WET urls of the .nz TLD has now been created."
+num_uniq_urls=`wc -l $uniq_urls_file`
+total_urls=`wc -l $1`
+echo ""
+echo ""
+echo "Summary:"
+echo "There were $num_uniq_urls unique WET urls"
+echo "out of a total of $total_urls urls in $1."
+echo ""
+echo ""
+
+exit 0
Index: /other-projects/maori-lang-detection/bin/script/create-uniq-nz-urls-file.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/create-uniq-nz-urls-file.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/create-uniq-nz-urls-file.sh	(revision 33635)
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Need to run this script from the top level folder of this extension
+# This script needs to be provided with the nz-only-TLDs-<date>.txt file produced by running get_commoncrawl_nz_urls.sh
+if [ -z $1 ]; then
+    echo ""
+    echo "Usage: $0 nz-only-TLDs-<date>.txt"
+    echo "   where the nz-only-TLDs-<date>.txt file is produced by running get_commoncrawl_nz_urls.sh."
+    echo ""
+    exit 1
+fi
+
+uniq_urls_file=uniq-tld-nz-urls-`date +%F`.txt
+echo "Creating file $uniq_urls_file containing just the unique domains and subdomains..."
+
+# cat is unnecessary: https://stackoverflow.com/questions/1915636/is-there-a-way-to-uniq-by-column/1915750
+#    "You can dump the cat! Rather than piping into <the subsequent process>, just let <the subsequent process> read the file using <. Piping through cat is a common unnecessary complication used by novices. For large amounts of data there's a performance effect to be had."
+
+#cat $1 | cut -d ' ' -f4 | cut -d/ -f3 | sed -r 's@\.?",\s*$@@' | sed -r 's@^www\.@@' | uniq > $uniq_urls_file
+<$1 cut -d ' ' -f4 | cut -d/ -f3 | sed -r 's@\.?",\s*$@@' | sed -r 's@^www\.@@' | uniq > $uniq_urls_file
+
+
+# The first cut grabs the url field of the json.
+# The second cut grabs the domain name from the url (located between first // and immediately subsequent /).
+# The first sed process then removes any trailing . (e.g. "massey.ac.nz." becomes "massey.ac.nz") followed by ", and optional spaces before the end,
+# and the final sed removes any "www." prefix.
+# Then we get the uniq urls out of all that.
+
+echo "File $uniq_urls_file containing just the unique .nz sites (domains and subdomains) to be used as seed urls has now been created."
+num_uniq_urls=`wc -l $uniq_urls_file`
+total_urls=`wc -l $1`
+echo ""
+echo ""
+echo "Summary:"
+echo "There were $num_uniq_urls unique sites"
+echo "out of a total of $total_urls urls in $1."
+echo ""
+echo ""
+
+exit 0
Index: /other-projects/maori-lang-detection/bin/script/drop_nutch_solrcore.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/drop_nutch_solrcore.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/drop_nutch_solrcore.sh	(revision 33635)
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Number of fetch cycles 4. (6 takes 18 mins for ~davidb extended to ~ihw)
+# apache-nutch-1.15>./bin/crawl -s urls crawlFolder 4
+# apache-nutch-1.15>./bin/nutch index crawlFolder/crawldb/ -linkdb crawlFolder/linkdb/ crawlFolder/segments/20190918* -filter -normalize -deleteGone
+
+# This script drops and removes the "nutch" solr core from APACHE_SOLR_HOME
+# Then recreates and reinitializes it
+
+APACHE_SOLR_HOME=/Scratch/ak19/gs3-extensions/maori-lang-detection/apache-solr
+
+pushd $APACHE_SOLR_HOME
+./bin/solr start
+./bin/solr delete -c nutch
+./bin/solr stop
+
+pushd server/solr/configsets/
+rm -rf nutch
+
+mkdir nutch
+cp -r _default/* nutch/.
+rm nutch/conf/managed-schema
+
+# SCHEMA_DOWNLOADED.xml is saved from https://github.com/apache/nutch/blob/master/conf/schema.xml
+# If recreating this template file, make sure to visit that link and COPY the contents
+# and paste it into an .xml file, rather than just downloading the file.
+# Downloading saves the xml as html and introduces invalid xml.
+cp SCHEMA_DOWNLOADED.xml nutch/conf/schema.xml
+popd
+# we're back in apache_solr_home
+
+./bin/solr start
+./bin/solr create -c nutch -d $APACHE_SOLR_HOME/server/solr/configsets/nutch/conf/
+
+popd
+# we're back to where we launched this script from
+
Index: /other-projects/maori-lang-detection/bin/script/gen-all-dumps.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/gen-all-dumps.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/gen-all-dumps.sh	(revision 33635)
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# Put this script inside the apache nutch installation
+
+for segdir in crawl/segments/2* ; do
+    seg=${segdir##*/}
+
+    if [ ! -d "/tmp/dump-$seg" ] ; then
+	./bin/nutch readseg -dump $segdir "/tmp/dump-$seg" -nofetch -nogenerate -noparse -noparsedata -noparsetext
+    fi
+
+done
+
+
+echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
+echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
+
+for segdir in crawl/segments/2* ; do
+    seg=${segdir##*/}
+
+    echo "===================="
+    echo "$seg"
+    echo "===================="
+
+    
+    
+    urls=`egrep -a "^URL:" /tmp/dump-$seg/dump`
+
+    wc_urls=`echo $urls | wc`
+
+    echo "Number of urls: $wc_urls"
+
+    echo $urls | tr ' ' '\n' | egrep -v '^URL::'
+
+    # echo $urls
+done
Index: /other-projects/maori-lang-detection/bin/script/gen_SentenceDetection_model.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/gen_SentenceDetection_model.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/gen_SentenceDetection_model.sh	(revision 33635)
@@ -0,0 +1,140 @@
+#!/bin/bash
+
+#####################################################################################################
+# Following instructions at
+# https://stackoverflow.com/questions/36516363/sentence-detection-with-opennlp
+# But contrary to their example, don't insert empty newlines separating the sentences input file
+# Each sentence just needs to be on its own line. But no empty lines between each sentence.
+#####################################################################################################
+
+# 1. Create training file of Maori sentences in the correct format
+# Train this on the 2011 set (rather than 2017) set of 100k Maori language sentences
+# because the 2011 one appears to have fewer accidentally incorporated English sentences
+
+
+# Need to run this script from the top level folder of this extension
+
+if [ ! -z $1 ]; then
+    if [ "x$1" = "x--help" ]; then
+	echo ""
+	echo "Usage: $0 <mri sentences.txt>"
+	echo "   Try the one in ./opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt"
+	echo "   (Unzip mri-opennlp-corpus.tar.gz)"
+	echo "   It's better than the 100k sentences from 2017, as that contains a lot of English lines."
+	echo ""
+	exit
+    else
+	infile=$1
+    fi
+else
+    # use the file opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt as inputfile
+    # Check the file exists at the expected location. If not, check we have the tarball containing it and untar that.
+    # If the tarball doesn't exist, try to get what we want from svn if we have svn. If no svn either, bail.
+    if [ ! -f opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt ]; then
+	if [ -f mri-opennlp-corpus.tar.gz ]; then
+	    tar -xvzf mri-opennlp-corpus.tar.gz
+	else
+	    svn --help > /dev/null
+	    if [ "x$?" != "x0" ]; then
+		echo "Attempted to train the Sentence Detector on Maori sentences on missing mri_web_2011_100K-sentences.txt"
+		echo "The tarball mri-opennlp-corpus.tar.gz (containing mri_web_2011_100K-sentences.txt) didn't exist either."
+		echo "And couldn't get opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt using SVN: no SVN."
+		echo "Exitting..."
+		exit
+	    else
+		# Get just what we need from svn
+		# Based on https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+		# To auto-confirm repeated svn https check, see https://serverfault.com/questions/37929/how-do-you-accept-an-ssl-certificate-through-the-svn-command-line
+		svn co --depth immediates --trust-server-cert --non-interactive https://svn.apache.org/repos/bigdata/opennlp/trunk opennlp-corpus
+		cd opennlp-corpus
+		svn up --set-depth immediates --trust-server-cert --non-interactive
+		cd leipzig
+		svn up --set-depth immediates --trust-server-cert --non-interactive
+		cd resources/
+		svn up --set-depth infinity --trust-server-cert --non-interactive 
+		cd ../data
+		echo "About to download the sample Maori language 100k sentences file. This may take a while (<1 minute)."
+		svn up --trust-server-cert --non-interactive mri_web_2011_100K-sentences.txt
+		cd ..
+		# in opennlp-corpus/leipzig
+		chmod u+x create_langdetect_model.sh
+		cd ../..
+	    fi
+	fi	
+    fi
+    infile=./opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt
+fi
+
+echo $infile
+
+#tail -100 $infile
+
+# Ensure OPENNLP_HOME is set
+if [ "x$OPENNLP_HOME" = "x" ]; then
+    echo "OPENNLP_HOME not set, attempting to set it to the local apache-opennlp (v1.9.1). ENSURE THIS EXISTS OR SET OPENNLP_HOME YOURSELF!"
+    #if [ -d apache-opennlp-* ]; then
+    cd apache-opennlp-*
+    if [ "x$?" = "x0" ]; then
+	export OPENNLP_HOME=`pwd`
+	cd ..
+    else
+	echo "No OPENNLP_HOME set and could not find a subfolder 'apache-opennlp-...' to set it to."
+	echo "Set OPENNLP_HOME yourself before running this script. Exitting..."
+	exit
+    fi
+fi
+
+mkdir -p $OPENNLP_HOME/training_data
+sentences_train_file=$OPENNLP_HOME/training_data/mri-sent.train
+if [ -f "$sentences_train_file" ]; then
+    echo "Removing existing $sentences_train_file"
+    rm "$sentences_train_file"
+fi
+
+# Get just the sentences in the inputfile (remove the starting number followed by tab from each line)
+# see https://stackoverflow.com/questions/36516363/sentence-detection-with-opennlp
+# Don't add an extra newline at end of each sentence (don't insert an empty line between each sentence)
+
+
+# https://stackoverflow.com/questions/7619438/bash-read-a-file-line-by-line-and-process-each-segment-as-parameters-to-other-p
+# how come it removes the first number automatically?
+#while read file num sentence
+#do
+#    echo "$num $sentence" >> $sentences_train_file
+#done < "$infile"
+
+# Don't add the extra newline at the end, creating empty lines between sentences output,
+# or the SentenceDetector in step 3 will do a poor job.
+# (The example at the stackoverflow link may have included newlines separating sentences for clarity.)
+#cat $infile | awk -F '\t' '{ print $2 "\n" }' > $sentences_train_file
+cat $infile | awk -F '\t' '{ print $2 }' > $sentences_train_file
+
+# Why did my way below not work?
+#    openNLP-lang-detect/opennlp-corpus/leipzig/data>echo "100000\tYWCA Boarding house : ÄwhinaServices and support Kei te pÅ«manawa o TÄmaki Makaurau a YMCA." | awk -F "\t" '{ print $2 }'
+# $1 contains everything and $2 nothing. Why?
+# The problem appears to be that echo behaves differently from cat and less. Ask Dr Bainbridge what makes echo different.
+# Note that I tried manually inserting \t, after copying the original line with tabspacing had no effect. Still no difference.
+# Note 2: echo doesn't appear to preserve copied tab spaces.
+# Answer: echo doesn't treat \n as newline and \t as tab and so on, unless the -e flag is passed in:
+# echo -e "100000\tYWCA Boarding house : ÄwhinaServices and support Kei te pÅ«manawa o TÄmaki Makaurau a YMCA." | awk -F "\t" '{ print $2 }'
+
+# 2. Create mri sentences model from training sentences file
+#$OPENNLP_HOME/bin/opennlp SentenceDetectorTrainer -model mri-sent_trained.bin -lang en -data mri-sent.train -encoding UTF-8
+
+
+mkdir -p $OPENNLP_HOME/models
+
+$OPENNLP_HOME/bin/opennlp SentenceDetectorTrainer -model $OPENNLP_HOME/models/mri-sent_trained.bin -lang en -data $sentences_train_file -encoding UTF-8
+
+
+# 3. Let's try to split some Maori language text into sentences using our model created from training on sentences.
+# http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.sentdetect
+#    Usage: opennlp SentenceDetector model < sentences
+echo ""
+echo "Testing the model."
+echo "Running the SentenceDetector with the generated mri-sent_trained.bin model on sample_mri_paragraphs.txt..."
+echo "****************************"
+$OPENNLP_HOME/bin/opennlp SentenceDetector $OPENNLP_HOME/models/mri-sent_trained.bin < models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt
+echo "****************************"
+echo ""
+
Index: /other-projects/maori-lang-detection/bin/script/get_commoncrawl_nz_urls.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/get_commoncrawl_nz_urls.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/get_commoncrawl_nz_urls.sh	(revision 33635)
@@ -0,0 +1,187 @@
+#!/bin/bash
+
+#####################################################################################################
+# This script follows the instructions at https://groups.google.com/d/msg/common-crawl/3QmQjFA_3y4/vTbhGqIBBQAJ
+# to download all the .nz TLD URLs that CommonCrawl has crawled for a given date.
+# It needs to be provided with either a cc-index.paths.gz file for a particular date or
+# the URL for such a file, both obtainable from http://index.commoncrawl.org/
+#####################################################################################################
+
+
+function printUsage() {
+    echo ""
+    echo "Usage: $0 <cc-index.paths.gz | URL to a cc-index.paths.gz file, e.g. https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/cc-index.paths.gz>"
+    echo "   You can download such a file or copy its URL from http://index.commoncrawl.org/"
+    echo ""
+}
+
+
+
+# Need to run this script from the top level folder of this extension
+# This script needs to be provided with a cc-index.paths.gz downloaded from http://index.commoncrawl.org/
+# or a URL of the form https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/cc-index.paths.gz
+if [ -z $1 ]; then
+    printUsage
+    exit 1
+fi
+
+# now we have a $1
+if [[ "x$1" = "x--help" || "x$1" = "x-h" ]]; then
+    printUsage
+    exit 0
+fi	
+
+# We'll create a temporary directory to work in and tell the user to delete it after we're done
+mkdir -p tmp
+
+# some sanity checking
+infile=$1
+# if $1 contains the correct format for the URL to the cc-index.paths.gz, then wget it and make that file the 
+if [[ $infile == https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-*/cc-index.paths.gz ]]; then
+    wget $infile
+
+    if [ ! -f cc-index.paths.gz ]; then
+	echo "Downloading of $infile failed."
+	exit -1
+    fi
+    
+    mv cc-index.paths.gz tmp/.
+    infile=tmp/cc-index.paths.gz
+fi
+
+if [[ $infile != *cc-index.paths.gz ]]; then
+    echo "*** Invalid parameter: This script can only work with the cc-index.paths.gz file or URL to such a file"
+    echo ""
+    printUsage
+    exit -1
+fi
+
+# decompress. gunzip removes the .gz archive file and just leaves the decompressed version (same filename, without .gz)
+# So infile hereafter is without the .gz suffix, https://wiki.bash-hackers.org/syntax/pe#substring_removal
+gunzip $infile
+infile=${infile%.gz}
+#echo "Finished extracting $infile from the download"
+
+aws_prefix=https://commoncrawl.s3.amazonaws.com/
+cluster_url=
+# read in file line-by-line in reverse (cat-reversed tac instead of cat), and store the cluster.idx line
+# http://forums.codeguru.com/showthread.php?472039-RESOLVED-BASH-read-from-end-of-file-upwards
+
+#tac $infile | while read line; do
+#    if [[ $line = *cluster.idx ]]; then
+#	echo "Found cluster.idx at: $line"
+#	cluster_url=$aws_prefix$line
+#	echo "Will download cluster.idx file from $cluster_url"
+#    fi
+#done
+
+# Above way is bad because we can't BREAK out of the outer process (tac, reading each line in) when we've found the line we want.
+# Instead, do it the following way: to read until we found the line of interest so we can then BREAK out of the loop properly/cleanly:
+# https://unix.stackexchange.com/questions/166546/bash-cannot-break-out-of-piped-while-read-loop-process-substitution-works
+while read line; do
+    if [[ $line = *cluster.idx ]]; then
+	#echo "Found cluster.idx mentioned at: $line"
+	cluster_url=$aws_prefix$line
+	echo "Will download the cluster.idx file from $cluster_url"
+	echo ""
+	break
+    fi
+done < <(tac $infile) 
+
+# make sure we have a cluster.idx URL
+if [ -z $1 ]; then
+    echo "Could not find the cluster.idx listed in $infile"
+    echo ""
+    exit -1
+fi
+
+# Work in the tmp folder for a while
+pushd tmp
+
+# Now let's get cluster.idx
+wget $cluster_url
+
+if [ ! -f cluster.idx ]; then
+    echo "Downloading of $cluster_url failed."
+    echo ""
+    exit -1
+fi
+
+# Hereafter, the amazon prefix is the parent path of the full URL to cluster.idx
+aws_prefix=${cluster_url%cluster.idx}
+#echo "New prefix: $aws_prefix"
+
+outfile=nz-only-TLDs-`date +%F`.txt
+touch $outfile
+
+while read line; do
+    urls_file=$line
+    echo "Found .nz TLD mentioned in: $urls_file (located at in $aws_prefix). Will wget and extract."
+    echo ""
+    
+    # get the large file (under 1GB)
+    wget "$aws_prefix$urls_file"
+    # Unzip each, and we have all URLs with TLD .nz, but perhaps also .no for Norway in the first and .org in the last
+    # We just want the nz domains. 
+    
+    if [ ! -f $urls_file ]; then
+    	echo "ERROR: Downloading of $urls_file containing .nz specific URLs failed."
+    	echo ""
+    	break
+	# Let's not exit, but go around the loop trying to download any other gz files also mentioning the .nz TLD
+    else
+	# Extract the huge file
+    	echo "Extracting..."
+	echo "   (This may take a minute or so, please be patient.)"
+    	gunzip $urls_file
+    	echo ""
+
+	echo "Reducing the downloaded URLs to just those with toplevel domain .nz..."
+	echo ""
+    
+	# Extract just those URLs that start with "^nz," (since the SURT form of URLs, the form starting with the TLD) come first on each line
+	# and append to $outfile of all .nz TLDs for the specific crawl
+	# See https://www.unix.com/shell-programming-and-scripting/176608-how-copy-lines-starts-either-3-4-into-new-file.html
+	urls_file=${urls_file%.gz}
+	egrep "^nz," $urls_file >> $outfile
+    fi
+done < <(grep '^nz,' cluster.idx | cut -f2 | uniq)
+
+mv $outfile ../.
+# get out of the tmp dir
+popd
+
+echo ""
+echo "The file $outfile has now been created, containing all the .nz domains for the crawl of $1"
+echo ""
+
+# uniq_urls_file=uniq-tld-nz-urls-`date +%F`.txt
+# echo "Creating file $uniq_urls_file containing just the unique domains and subdomains..."
+# #cat $outfile | cut -d ' ' -f4 | cut -d/ -f3 | sed -r 's@\.?",\s*$@@' | sed -r 's@^www\.@@' | uniq > $uniq_urls_file
+# <$outfile cut -d ' ' -f4 | cut -d/ -f3 | sed -r 's@\.?",\s*$@@' | sed -r 's@^www\.@@' | uniq > $uniq_urls_file
+
+# # cat is unnecessary: https://stackoverflow.com/questions/1915636/is-there-a-way-to-uniq-by-column/1915750
+# #    "You can dump the cat! Rather than piping into <the subsequent process>, just let <the subsequent process> read the file using <. Piping through cat is a common unnecessary complication used by novices. For large amounts of data there's a performance effect to be had."
+
+# # The first cut grabs the url field of the json.
+# # The second cut grabs the domain name from the url (located between first // and immediately subsequent /).
+# # The first sed process then removes any trailing . (e.g. "massey.ac.nz." becomes "massey.ac.nz") followed by ", and optional spaces before the end,
+# # and the final sed removes any "www." prefix.
+# # Then we get the uniq urls out of all that.
+
+# echo "File $uniq_urls_file containing just the unique .nz sites (domains and subdomains) to be used as seed urls has now been created."
+# num_uniq_urls=`wc -l $uniq_urls_file`
+# total_urls=`wc -l $outfile`
+# echo ""
+# echo ""
+# echo "Summary:"
+# echo "There were $num_uniq_urls unique sites"
+# echo "out of a total of $total_urls urls in $outfile."
+# echo ""
+# echo ""
+
+echo ""
+echo "NEXT:"
+echo "Remember to delete the products in the tmp folder or the folder itself after inspecting its contents."
+echo ""
+exit 0
Index: /other-projects/maori-lang-detection/bin/script/unique_mri_domains_from_cc.sh
===================================================================
--- /other-projects/maori-lang-detection/bin/script/unique_mri_domains_from_cc.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/bin/script/unique_mri_domains_from_cc.sh	(revision 33635)
@@ -0,0 +1,82 @@
+#!/bin/bash
+
+# After running the WETProcessor on warc.wet(.gz) files
+# to create keepURLs.txt and discardURLs.txt, run this script on keepURLs.txt
+# RUN AS:
+# ./bin/script/unique_mri_domains_from_cc.sh tmp/processedWET/keepURLs.txt
+
+# https://stackoverflow.com/questions/4651437/how-do-i-set-a-variable-to-the-output-of-a-command-in-bash
+
+infile=$1
+prev_glued_together=
+
+# "The sort -u command removes duplicates"
+# see http://pages.di.unipi.it/corradini/Didattica/AP-17/DOCS/Scott-ch13.pdf
+<$1 sort -u > sorted_MRI_urls.txt
+
+
+while read url; do
+    line=$url
+
+    # Extract 3rd field of each url, where delimiter is /. So http:// counts as the first and second. Then third field follows up to next /.
+    line="$(echo $line | cut -d/ -f3)"
+
+    
+    #    line="$(echo $line | cut -d ' ' -f4)"
+
+    # remove optional www. at start
+    # and turn urls like "massey.ac.nz." into "massey.ac.nz"   
+    # https://stackoverflow.com/questions/16623835/remove-a-fixed-prefix-suffix-from-a-string-in-bash
+    line=${line#"www."}
+#    line=${line%"."}
+#    line=${line%"\","}
+    #echo $line
+    
+    # In order to test subdomains, first remove period marks
+    # https://stackoverflow.com/questions/13210880/replace-one-substring-for-another-string-in-shell-script
+    glued_together=${line//\./}
+    #echo $glued_together
+    # check if this line is a 'suffix' of the previous one
+    # https://stackoverflow.com/questions/2172352/in-bash-how-can-i-check-if-a-string-begins-with-some-value/2172365
+    
+    if [ -z $prev_glued_together ]; then
+	echo "$line"
+	# found the first domain, store as previous one
+	prev_glued_together=$glued_together
+	echo $url >> uniq_mri_urls.txt
+    else
+	#echo "> Previous: $prev_glued_together"
+	#echo "> Current: $glued_together"
+	
+	# if [[ $glued_together = *$prev_glued_together ]]; then
+	#     echo "X$line"
+	# elif [[ $prev_glued_together = *$glued_together ]]; then
+	#     # check if prev line is a 'suffix' of this one
+	#     echo "XX$line"
+	# else
+	#     echo "$line"
+	# fi
+#	if [[ $glued_together != *$prev_glued_together && $prev_glued_together != *$glued_together ]]; then
+	if [[ $glued_together != $prev_glued_together* && $prev_glued_together != $glued_together* ]]; then	    
+	    echo "Selecting: $line"
+	    # found a new domain, store as previous one
+	    prev_glued_together=$glued_together
+
+	    # write uniq url to file
+	    echo $url >> uniq_mri_urls.txt
+	fi
+    fi
+    
+    
+done < <(cat sorted_MRI_urls.txt)
+
+echo "**************************************"
+count=`wc -l $1`
+echo "Number of URLs in INPUT FILE: $count" # $1
+count=`wc -l sorted_MRI_urls.txt`
+echo "Number of URLs in intermediate unique sorted file: $count" #sorted_MRI_urls.txt
+count=`wc -l uniq_mri_urls.txt`
+echo "Number of URLs of unique domains in OUTPUT FILE: $count" #uniq_mri_urls.txt
+echo "**************************************"
+# remove intermediate file
+rm sorted_MRI_urls.txt
Index: /other-projects/maori-lang-detection/conf/config.properties
===================================================================
--- /other-projects/maori-lang-detection/conf/config.properties	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/config.properties	(revision 33635)
@@ -0,0 +1,31 @@
+# https://www.linuxjournal.com/content/downloading-entire-web-site-wget
+# https://linuxreviews.org/Wget:_download_whole_or_parts_of_websites_with_ease
+# https://www.webhostface.com/kb/knowledgebase/examples-using-wget/
+# "You can replicate the HTML content of a website with the âmirror option (or -m for short)
+#       wget -m http://domain.com"
+# https://www.linuxquestions.org/questions/linux-server-73/wget-how-to-download-more-than-one-file-at-once-instead-of-file-after-file-704693/
+wget.mirror.cmd=wget -Q10m -m %%BASE_URL%%
+
+# for downloading a single file
+wget.file.cmd=wget %%FILE_URL%%
+
+# Arbitrary cutoff values for WETProcessor.java
+WETprocessor.min.content.length=100
+WETprocessor.min.line.count=2
+WETprocessor.min.content.length.wrapped.line=500
+WETprocessor.min.spaces.per.wrapped.line=10
+
+# Arbitrary cutoff values for WETProcessor.java
+# for determining whether a WET record has sufficient and sensible content
+WETprocessor.max.word.length=15
+WETprocessor.min.num.words=20
+WETprocessor.max.words.camelcase=10
+
+
+mongodb.user=anupama
+mongodb.pwd=chang3m3
+# default mongodb port is 27017. Don't change the port unless you really have configured
+# your mongodb server to listen at some other port
+mongodb.port=27017
+mongodb.host=mongodb.cms.waikato.ac.nz
+mongodb.dbname=ateacrawldata
Index: /other-projects/maori-lang-detection/conf/keep-since-not-product-sites.txt
===================================================================
--- /other-projects/maori-lang-detection/conf/keep-since-not-product-sites.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/keep-since-not-product-sites.txt	(revision 33635)
@@ -0,0 +1,839 @@
+https://www.maoritelevision.com
+	https://www.maoritelevision.com/mi/tagged/all/Prison?page=8
+	https://www.maoritelevision.com/mi/tagged/news/Prison
+	https://www.maoritelevision.com/mi/tagged/all/Netball
+	https://www.maoritelevision.com/mi/tagged/all/Education?page=6
+	https://www.maoritelevision.com/mi/tagged/all/Police?rows=10
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini/groups/te-kaheru-matarau-a-haua
+	https://www.maoritelevision.com/mi/tagged/all/Women?page=2&rows=10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu
+	https://www.maoritelevision.com/mi/tagged/all/M%C4%81tauranga
+	https://www.maoritelevision.com/mi/page/news_article_overview/172762/purongo-hou
+	https://www.maoritelevision.com/mi/tagged/videos/M%C4%81ori
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81ti%20Awa
+	https://www.maoritelevision.com/mi/tagged/videos/Auckland%20City%20Mission
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-maori-news?available=0
+	https://www.maoritelevision.com/mi/tagged/news/Entertainment?page=2
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tao
+	https://www.maoritelevision.com/mi/tagged/all/History?page=6
+	https://www.maoritelevision.com/mi/kapahaka/tamaki-makaurau
+	https://www.maoritelevision.com/mi/tagged/all/Protest?page=6
+	https://www.maoritelevision.com/mi/tagged/all/Waikato?page=2
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-maori-news?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living?available=0
+	https://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/rereatea-news-bytes/purongo-hou
+	https://www.maoritelevision.com/mi/tagged/all/Police
+	https://www.maoritelevision.com/mi/kapahaka/asb-polyfest-2018/groups
+	https://www.maoritelevision.com/mi/tagged/all/Women?page=2&amp;rows=10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/nga-wahanga
+	https://www.maoritelevision.com/mi/tagged/all/Tamararo
+	https://www.maoritelevision.com/mi/tagged/news/M%C4%81ori
+	https://www.maoritelevision.com/mi/kiriata
+	https://www.maoritelevision.com/mi/kapahaka/tearawa
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu/groups/ngati-kahungunu-ki-heretaunga
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/my-country-song
+	https://www.maoritelevision.com/mi/tagged/all/Housing
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=6&amp;rows=10
+	https://www.maoritelevision.com/mi/tagged/all/Touch%20Rugby
+	https://www.maoritelevision.com/mi/tagged/all/Gisborne
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living?available=0
+	https://www.maoritelevision.com/mi/tagged/all/M%C4%81ori%20Broadcasting
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau
+http://www.maoritelevision.com
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa-new-zealand-documentaries/S06E001/mending-makotuku
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take/S03E003/media-take-3-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E009/te-tepu
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E028/opaki-1-upoko-28
+	http://www.maoritelevision.com/mi/tagged/all/Elections?page=3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kuia/S01E009/kuia-1-upoko-9
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S09E041/pukoro-9-upoko-41
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-taihoa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E002/nga-pari-karangaranga-nga-kapa-haka-o-te-taitokerau-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/paepae/S01E033/paepae-1-upoko-33
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kauta/S04E013/te-kauta
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/big-sing/S01E011/big-sing-1-upoko-11
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E067/toku-reo-1-upoko-67
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S09E027/pukoro-9-upoko-27
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E016/wairua-1-upoko-16
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S09E018/pukoro-9-upoko-18
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S01E023/nga-tangata-taumata-rau-nga-tamariki-o-te-kohu-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E079/toku-reo-1-upoko-79
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E023/opaki-1-upoko-23
+	http://www.maoritelevision.com/mi/tagged/all/History
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E003/nga-pari-karangaranga-tuwharetoa-4-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E078/toku-reo-1-upoko-78
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S10E010/pukoro-10-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E003/wairua-1-upoko-3-kupu-hauraro
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E005/nga-pari-karangaranga-tauranga-moana-5-upoko-5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E036/toku-reo-3-upoko-36
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living/S02E023/whanau-living-2-upoko-23
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take/S03E004/media-take-extended-3-upoko-4
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-o%20te-teko-nga-taiohi/184313
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E013/waiata-1-upoko-13
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-kaupapa-maori-o-bernard-fergusson%20/184425
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-kaupapa-maori-o-tuia-te-matangi%20/184414
+	http://www.maoritelevision.com/mi/tagged/all/Diabetes
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukuhohe/S01E005/pukuhohe-1-upoko-5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E033/opaki-1-upoko-33
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E008/waiata-1-upoko-8
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kuia/S01E011/kuia-1-upoko-11
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E007/wairua-1-upoko-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S06E013/ako-6-upoko-13
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E013/waiata-1-upoko-13
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/anzac-2018/S01E001/anzac-2015-awol-vignette
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take/S03E002/media-take-extended-3-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S02E046/opaki-2-upoko-46
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/koroua/S01E013/koroua-1-upoko-13
+	http://www.maoritelevision.com/mi/kapahaka/asb-polyfest-2017/groups/kaipara-college-nga-taniwha-o-kaipara/155826
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/matangi-rau/S06E004/matangi-rau-6-upoko-4
+http://community.nzdl.org
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-premier-women-w6-500-whiringa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E069/toku-reo-1-upoko-69
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S01E023/nga-tangata-taumata-rau-nga-tamariki-o-te-kohu-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E003/nga-pari-karangaranga-kuaka-wharau-5-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E004/te-tepu
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/paepae/S01E031/paepae-1-upoko-31
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/native-kitchen
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S02E050/opaki-series-2-episode-50-2-upoko-50
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living/S02E022/whanau-living-2-upoko-22
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E024/opaki-1-upoko-24
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-premier-men-w1-500-whiringa-toa
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	http://www.maoritelevision.com/mi/tagged/all/Waiariki
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-pataka-korero/S03E001/te-pataka-korero-3-upoko-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/reo-reka-heru
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whawhai/S01E030/whawhai-1-upoko-30
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E033/toku-reo-3-upoko-33
+	http://www.maoritelevision.com/mi/kapahaka/te-whenua-moemoea/groups/nga-rauika-a-maui/199663
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E100/toku-reo-1-upoko-100
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/matangi-rau/S06E004/matangi-rau-6-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E005/nga-pari-karangaranga-nga-kapa-haka-o-te-taitokerau-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E027/te-tepu
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S10E004/pukoro-10-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S02E070/kia-mau-2-upoko-70
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E003/opaki-1-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kauta/S04E009/te-kauta
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E010/wairua-1-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S06E022/ako-6-upoko-22
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S01E008/nga-tangata-taumata-rau-nga-tamariki-o-te-kohu-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E049/toku-reo-3-upoko-49
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S06E002/nga-pari-karangaranga-nga-korero-o-ngati-wai-6-upoko
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E025/waiata-1-upoko-25
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/matangi-rau/S06E011/matangi-rau-6-upoko-11
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E007/nga-pari-karangaranga-e-tu-maniapoto-5-upoko-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E005/nga-pari-karangaranga-he-wiwi-4-upoko-5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-master-men-w1-500-whiringa-to-0
+	http://www.maoritelevision.com/mi/tagged/all/Kaum%C4%81tua
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/anzac-2018/S01E001/anzac-2015-telecom-vignette
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kahui-nga-mata-raukura/184959
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E010/nga-pari-karangaranga-te-urewera-4-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E010/nga-pari-karangaranga-he-piko-he-taniwha-4-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E008/wairua-1-upoko-8-kupu-hauraro
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-wharekura-o-mauao
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whawhai/S01E030/whawhai-1-upoko-30
+	http://www.maoritelevision.com/mi/tagged/all/Taup%C5%8D
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E066/toku-reo-1-upoko-66
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kapa-hurutea-o-horouta-wananga/184913
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S03E036/ako-3-upoko-36
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E003/nga-pari-karangaranga-he-piko-he-taniwha-4-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take/S04E020/media-take-4-upoko-20
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/koroua/S01E008/koroua-1-upoko-8
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/big-sing/S01E010/big-sing-1-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whawhai/S01E009/whawhai-1-upoko-9
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-premier-men-w1-500-whiringa-toa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E006/nga-pari-karangaranga-he-piko-he-taniwha-5-upoko-6
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E083/toku-reo-1-upoko-83
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E065/toku-reo-1-upoko-65
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E098/toku-reo-1-upoko-98
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S02E048/opaki-2-upoko-48
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E001/opaki-ka-timata-hei-te-rahina-07-o-hongongoi-200-i-te-po
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/reo-reka-pukati
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whawhai/S01E029/whawhai-1-upoko-29-0
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kuia/S02E006/kuia-2-upoko-6
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E002/nga-pari-karangaranga-nga-reo-o-taranaki-5-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E004/nga-pari-karangaranga-te-reo-o-te-whanau-apanui-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa-new-zealand-documentaries/S01E001/lost-voyage-499
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E031/te-tepu
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E003/nga-pari-karangaranga-te-urewera-4-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S06E033/ako-6-upoko-33
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-mana-kuratahi/S01E013/te-mana-kuratahi-2017-te-kapa-puawai-tkkm-o-hoani-waitit-0
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whatukura-mareikura/S05E031/whatukura-mareikura-5-upoko-31
+	http://www.maoritelevision.com/mi/tagged/all/Lifestyle
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-iti-kahurangi-journey-success/S02E012/te-iti-kahurangi-2-upoko-12
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-inu
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/patapatai/S01E005/patapatai-1-upoko-5
+	http://www.maoritelevision.com/mi/tagged/all/Waikato
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E075/toku-reo-1-upoko-75
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E001/nga-pari-karangaranga-e-tu-maniapoto-5-upoko-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/building-block-kia-mau-ki-raro
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/tamaki-paenga-hira/S01E010/tamaki-paenga-hira-1-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/building-block-pukana
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-j16women-w6-1000-whiringa-toa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E027/opaki-1-upoko-27
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S10E011/pukoro-10-upoko-11
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-j16-women-w6-1000-whiringa-toa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waka-warriors/S01E004/waka-warriors-1-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-j16-women-w6-500-whiringa-toa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/iwi-anthems/S02E005/iwi-anthems-2-upoko-5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/gc/S01E007/gc-1-upoko-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E002/te-tepu
+	http://www.maoritelevision.com/mi/purongo/purongo-hou/pa-whutuporo
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S06E004/nga-pari-karangaranga-te-reo-irirangi-o-te-hiku-o-te
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/playlist/S01E002/playlist-1-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E002/opaki-1-upoko-2
+	http://www.maoritelevision.com/mi/tagged/all/Crime
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E022/waiata-1-upoko-22
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-kaupapa-maori-o-bernard-fergusson%20/184432
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E044/toku-reo-3-upoko-44
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E002/nga-pari-karangaranga-nga-reo-o-taranaki-4-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S06E003/nga-pari-karangaranga-he-tira-haka-6-upoko-3
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S01E045/ako-1-upoko-45
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/tamaki-paenga-hira/S01E011/tamaki-paenga-hira-1-upoko-11
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/waiata-2-ko-ahau
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/funny-whare/S01E012/funny-whare-gamesnight-1-upoko-12
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S01E021/nga-tangata-taumata-rau-nga-tamariki-o-te-kohu-1
+	http://www.maoritelevision.com/mi/kapahaka/te-tairawhiti
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S09E016/pukoro-9-upoko-16
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E035/toku-reo-3-upoko-35
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-whakamiharo-o-nia/S02E011/te-ao-whakamiharo-o-nia-2-upoko-11
+	http://www.maoritelevision.com/mi/tagged/all/Ng%C4%81i%20T%C5%ABhoe
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E004/opaki-1-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-aporo
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E008/wairua-1-upoko-8
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/karanga-first-voice/S02E005/karanga-first-voice-2-upoko-5
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kapa-hurutea-o-horouta-wananga/184919
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/koroua/S01E006/koroua-1-upoko-6
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-j16women-w6-1000-whiringa-toa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/swagger/S03E010/swagger-3-upoko-10
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E008/nga-pari-karangaranga-he-piko-he-taniwha-taki-korero
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-hamoa
+	http://www.maoritelevision.com/mi/tagged/all/Elections?page=21
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/anzac-2018/S01E001/anzac-2015-jam-tin-bomb-vignette
+	http://www.maoritelevision.com/mi/purongo/purongo-hou/koroneihana
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/reo-reka-poi
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E099/toku-reo-1-upoko-99
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E004/nga-pari-karangaranga-tauranga-moana-4-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-premier-women-dash-w1-250
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E003/wairua-1-upoko-3-kupu-hauraro
+	http://www.maoritelevision.com/mi/purongo/purongo-hou/te-ra-o-waitangi
+	http://www.maoritelevision.com/mi/kapahaka/asb-polyfest-2017/groups/kaipara-college-nga-taniwha-o-kaipara/168303
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living/S02E022/whanau-living-2-upoko-22
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-j19-men-dash-w1-250-whiringa
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kapahaka-o-te-kura-takawaenga-o-waitakere/183905
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-master-men-w6-500-whiringa-toa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S01E045/ako-1-upoko-45
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S06E016/ako-6-upoko-16
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S01E001/building-block-menemene
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-master-women-w12-500-whiringa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-whakamiharo-o-nia/S02E002/te-ao-whakamiharo-o-nia-2-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S09E040/pukoro-9-upoko-40
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E001/waiata-1-upoko-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-pahi
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-tohuto
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E008/wairua-1-upoko-8
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/paepae/S01E027/paepae-1-upoko-27
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S04E002/ako-4-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E008/waiata-1-upoko-8
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S10E005/pukoro-10-upoko-5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/iwi-anthems/S02E008/iwi-anthems-2-upoko-8
+	http://www.maoritelevision.com/mi/matakihia/ka-whakaaturia
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E037/toku-reo-3-upoko-37
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-premier-men-w6-1500-whiringa
+	https://www.maoritelevision.com/mi/nga-korero/korero-mo-whakaata-maori/wharangi-ipurangi
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/matangi-rau/S06E012/matangi-rau-6-upoko-12
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/wairua/S01E008/wairua-1-upoko-8-kupu-hauraro
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S01E059/toku-reo-1-upoko-59
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukoro/S09E048/pukoro-9-upoko-48
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E004/nga-pari-karangaranga-te-whare-tu-taua-o-aotearoa-5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take/S03E009/media-take-3-upoko-9
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E010/te-tepu
+	http://www.maoritelevision.com/mi/kapahaka/tainui
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E002/waiata-1-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/tamaki-paenga-hira/S01E007/tamaki-paenga-hira-1-upoko-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E005/nga-pari-karangaranga-he-piko-he-taniwha-taki-korero
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E026/opaki-1-upoko-26
+	https://www.maoritelevision.com/mi/kapahaka/waitaha/groups/te-pao-a-tahu/150805
+	https://www.maoritelevision.com/mi/tamarikiha/203465/episodes
+	https://www.maoritelevision.com/mi/tagged/news/King%20Tuheitia
+	https://www.maoritelevision.com/mi/tagged/all/Tangi?page=3
+	https://www.maoritelevision.com/mi/tamarikiha/214850/episodes
+	https://www.maoritelevision.com/mi/pouaka-whakaata/hongere-o-whakaata-maori?category=3128&subtitles=1&start=9&length=6&channel=1&tereo=5
+	https://www.maoritelevision.com/mi/tamarikiha/2865/episodes
+	https://www.maoritelevision.com/mi/tamarikiha/170920/episodes
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini/groups/ngati-kahungunu-ki-heretaunga
+	https://www.maoritelevision.com/mi/tagged/news/M%C4%81ori%20All%20Blacks?page=5&amp%3Brows=10
+	https://www.maoritelevision.com/mi/tagged/videos/Wellington?page=7
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=12&display_count=yes&length=6&sort=created_desc&start=9&view_mode=featured_catch_up
+	https://www.maoritelevision.com/mi/tagged/all/Mountain
+	https://www.maoritelevision.com/mi/tagged/all/Business
+	https://www.maoritelevision.com/mi/tagged/all/Police?page=5&amp%3Bamp%3Brows=10
+	https://www.maoritelevision.com/mi/tamarikiha/169279/episodes
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa?category=122&sort=latest&start=12&length=6
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=3125
+	https://www.maoritelevision.com/mi/tamarikiha/203528/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Budget
+	https://www.maoritelevision.com/mi/tagged/all/Rap
+	https://www.maoritelevision.com/mi/tamarikiha/169390/episodes
+	https://www.maoritelevision.com/mi/tagged/videos/Kids%20Kai%20Kart
+	https://www.maoritelevision.com/mi/nga-korero/momo-mahi/te-mahi-i-whakaata-maori
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Papa%20Museum?page=1
+	https://www.maoritelevision.com/mi/tagged/videos/Waka%20ama?page=1
+	https://www.maoritelevision.com/mi/tamarikiha/214848/episodes
+	https://www.maoritelevision.com/mi/kapahaka/waitaha/groups/te-pao-a-tahu/150799
+	https://www.maoritelevision.com/mi/tagged/all/Drama?page=6
+	https://www.maoritelevision.com/mi/tamarikiha/128658/episodes
+	https://www.maoritelevision.com/mi/tagged/videos/Waka%20ama?page=4
+	https://www.maoritelevision.com/mi/tamarikiha/255884/episodes
+	https://www.maoritelevision.com/mi/matakihia/movies
+	https://www.maoritelevision.com/mi/tagged/all/Police?page=5&amp%3Bamp%3Brows=10&rows=10
+	https://www.maoritelevision.com/mi/tagged/all/Hawke%27s%20Bay?page=2
+	https://www.maoritelevision.com/mi/tagged/all/Japanese
+	https://www.maoritelevision.com/mi/tagged/all/Maket%C5%AB
+	https://www.maoritelevision.com/mi/tagged/shows/Artefact
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=178&sort=popular
+	https://www.maoritelevision.com/mi/tagged/all/Snr%20Master%20Men
+	https://www.maoritelevision.com/mi/tamarikiha/2924/episodes
+	https://www.maoritelevision.com/mi/tamarikiha/170566/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Weather?page=3
+	https://www.maoritelevision.com/mi/tagged/all/T%C4%81maki%20Paenga%20Hira?page=1&rows=10
+	https://www.maoritelevision.com/mi/tamarikiha/170757/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Health?page=6
+	https://www.maoritelevision.com/mi/tamarikiha/176220/episodes
+	https://www.maoritelevision.com/mi/kapahaka/waitaha/groups/te-pao-a-tahu/153037
+	https://www.maoritelevision.com/mi/tamarikiha/180166/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Kapa%20haka?page=9
+	https://www.maoritelevision.com/mi/tamarikiha/72556/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Wharekura
+	https://www.maoritelevision.com/mi/tamarikiha/203469/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Kapa%20haka?page=9&rows=10
+	https://www.maoritelevision.com/mi/tamarikiha/248460/episodes
+	https://www.maoritelevision.com/mi/tamarikiha/86158/episodes
+	https://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kahui-nga-mata-raukura/184959
+	https://www.maoritelevision.com/mi/tagged/all/Vera%20Kirihau%20Morgan
+	https://www.maoritelevision.com/mi/tamarikiha/2928/episodes
+	https://www.maoritelevision.com/mi/tagged/news/Waikato?page=2
+	https://www.maoritelevision.com/mi/tamarikiha/214846/episodes
+	https://www.maoritelevision.com/mi/tagged/all/Fashion?page=3&rows=10
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81i%20Tahu?page=2
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?sort=expiring&category=3129
+	https://www.maoritelevision.com/mi/tagged/all/Tauranga?page=9
+	https://www.maoritelevision.com/mi/kapahaka/waitaha/groups/te-pao-a-tahu/150805
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81%20T%C4%81ngata%20Taumata%20Rau%20-%20Ng%C4%81%20Waru%20Pumanawa
+	https://www.maoritelevision.com/mi/tagged/all/Iwi?page=1
+	https://www.maoritelevision.com/mi/kapahaka
+	https://www.maoritelevision.com/mi/tagged/all/Nelson
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tupaias-endeavour/matakihia
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=114&sort=expiring
+	https://www.maoritelevision.com/mi/tagged/all/Highlanders%20%28rugby%29
+	https://www.maoritelevision.com/mi/tagged/all/Ru%C4%81toki
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/jc-tha-barber/matakihia
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini/groups/ruatahuna-kakahu-mauku
+	https://www.maoritelevision.com/mi/tagged/news/%23REOreasons
+	https://www.maoritelevision.com/mi/tagged/videos/Parihaka
+	https://www.maoritelevision.com/mi/tagged/videos/Darts
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?sort=expiring&category=114
+	https://www.maoritelevision.com/mi/tagged/videos/King%20Tuheitia
+	https://www.maoritelevision.com/mi/tagged/videos/Fashion
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Ministry%20of%20Fisheries
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?sort=expiring&category=11
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/poly-movement-kaikanikani/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Ururoa%20Flavell
+	https://www.maoritelevision.com/mi/tagged/all/Matau
+	https://www.maoritelevision.com/mi/tagged/news/Reggae
+	https://www.maoritelevision.com/mi/tagged/all/BMX
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Wiki%20o%20Te%20Reo%20M%C4%81ori
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-te-waipounamu
+	https://www.maoritelevision.com/mi/tagged/news/Police
+	https://www.maoritelevision.com/mi/kapahaka/tearawa
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=4&amp;rows=10
+	https://www.maoritelevision.com/mi/tagged/all/Rarotonga
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/te-wiki-o-te-reo-maori
+	https://www.maoritelevision.com/mi/kapahaka/te-pou-herenga-o-kia-aroha
+	https://www.maoritelevision.com/mi/tagged/all/T%C5%ABhoe?page=6
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/code/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Protest?page=11
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-snr-master-women-w1-500
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/only-aotearoa/S01E002/only-aotearoa-de-colonization
+	https://www.maoritelevision.com/mi/purongo/kairipoata/regan-paranihi
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kauta/S04E043/te-kauta
+	https://www.maoritelevision.com/mi/tagged/all/Housing
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S03E027/ako-3-upoko-27
+	https://www.maoritelevision.com/mi/purongo/purongo-hou/ratana-pa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E011/waiata-1-upoko-11
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E002/nga-pari-karangaranga-tauranga-moana-4-upoko-2
+	https://www.maoritelevision.com/mi/tagged/all/M%C4%81ori%20Party
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S02E016/opaki-2-upoko-16
+	https://www.maoritelevision.com/mi/matakihia/ka-whakaaturia
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S01E001/nga-pari-karangaranga-he-kuaka-marangaranga-1-upoko-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S04E006/ako-4-upoko-6
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kitchen-kura
+	https://www.maoritelevision.com/mi/tagged/all/Rotorua
+	https://www.maoritelevision.com/mi/kapahaka
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/autaia/S01E022/autaia-series-1-episode-22
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S03E043/ako-3-upoko-43
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-j16-women-w1-500-whiringa-toa
+	https://www.maoritelevision.com/mi/tagged/all/Gisborne
+	https://www.maoritelevision.com/mi/purongo/kairipoata
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S05E044/ako-5-upoko-44
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E005/opaki-1-upoko-5
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-master-70-men-w1-500-whiringa
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	https://www.maoritelevision.com/mi/kapahaka/te-ahurea-2018
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81i%20T%C5%ABhoe
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-master-women-w1-500-whiringa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/all-talk-anika-moa/S02E010/all-talk-anika-moa-2-upoko-10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E016/tautohetohe-on-road-5-upoko-16
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S06E005/ako-6-upoko-5
+	https://www.maoritelevision.com/mi/kapahaka/asb-polyfest-2018/groups/sir-edmund-hillary-collegiate-irimana-hirere/200218
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/autaia/S01E027/autaia-series-1-episode-27
+	https://www.maoritelevision.com/mi/tagged/all/Justice
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S05E022/ako-5-upoko-22
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/paepae/S01E001/paepae-1-upoko-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-j19-men-w6-500-whiringa-toa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S01E001/2015-waka-ama-sprints-int-men-w6-1000-whiringa-toa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S01E048/ako-1-upoko-48
+	https://www.maoritelevision.com/mi/matakihia/ka-whakaaturia
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E007/tautohetohe-on-road-5-upoko-7
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E004/nga-pari-karangaranga-he-wiwi-he-naati-he-whanoke
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E003/nga-pari-karangaranga-tauranga-moana-4-upoko-3
+	https://www.maoritelevision.com/mi/kapahaka/te-whenua-moemoea/groups/te-raranga-whanui
+	https://www.maoritelevision.com/mi/tagged/all/Taranaki
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/anzac-2018?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E007/nga-pari-karangaranga-tauranga-moana-5-upoko-7
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/head-start/S01E010/head-start-1-upoko-10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/autaia/S01E025/autaia-series-1-episode-25
+	https://www.maoritelevision.com/mi/purongo/kairipoata/raniera-harrison
+	https://www.maoritelevision.com/mi/tagged/all/King%20Tuheitia
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E010/nga-pari-karangaranga-he-piko-he-taniwha-taki-korero
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E001/opaki-1-upoko-1
+	https://www.maoritelevision.com/mi/tagged/all/Auckland
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S05E028/ako-5-upoko-28
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini/groups/te-kapa-haka-o-te-whanau-a-apanui
+	https://www.maoritelevision.com/mi/kapahaka/te-whanganui-tara
+	https://www.maoritelevision.com/mi/matakihia/ka-whakaaturia
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S03E026/ako-3-upoko-26
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/head-start/S01E008/head-start-1-upoko-8
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S04E049/ako-4-upoko-49
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S01E024/nga-tangata-taumata-rau-nga-tamariki-o-te-kohu-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S02E007/ako-2-upoko-7
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S04E011/nga-pari-karangaranga-he-piko-he-taniwha-4-upoko-11
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Arawa
+	https://www.maoritelevision.com/mi/purongo/purongo-hou/te-ra-o-waitangi
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=111&sort=alpha
+	https://www.maoritelevision.com/mi/tagged/all/Hockey
+	https://www.maoritelevision.com/mi/tagged/all/Wairoa
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini-2019
+	https://www.maoritelevision.com/mi/tagged/all/M%C4%81ori%20Seats
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini-2019
+	https://www.maoritelevision.com/mi/tagged/all/Huntly
+	https://www.maoritelevision.com/mi/tagged/all/Taup%C5%8D
+	https://www.maoritelevision.com/mi/tagged/all/Waka%20ama?page=5
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=8&rows=10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pukuhohe
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-maori-news
+	https://www.maoritelevision.com/mi/kapahaka/te-matatini/groups/te-kapa-haka-o-te-whanau-a-apanui
+	http://community.nzdl.org/macron-restoration/jsp/mi/about.jsp
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81%20Pakeke%20o%20Ratana
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81ruaw%C4%81hia
+	https://www.maoritelevision.com/mi/nga-korero/hokohoko-tuku-panui
+	https://www.maoritelevision.com/mi/tagged/all/Poi
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/haka-life
+	https://www.maoritelevision.com/mi/tagged/all/Basketball?page=4
+	http://community.nzdl.org/macron-restoration/jsp/mi/feedback.jsp
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa?category=1550&subtitles=1
+	https://www.maoritelevision.com/mi/tagged/news/Waikato?page=2
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=5&rows=10
+	https://www.maoritelevision.com/mi/kapahaka/te-tairawhiti
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	https://www.maoritelevision.com/mi/tagged/all/Joseph%20Parker
+	https://www.maoritelevision.com/mi/tagged/all/Ru%C4%81toki
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-te-waipounamu
+	https://www.maoritelevision.com/mi/purongo/kairipoata/election-aotearoa
+	https://www.maoritelevision.com/mi/purongo/purongo-hou/nga-whakataetae-waka-ama-a-motu
+	https://www.maoritelevision.com/mi/tagged/all/M%C4%81ori
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/all-talk-anika-moa/S01E001/all-talk-anika-moa-ludus
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/he-iwi-whakapono/S02E007/he-iwi-whakapono-2-upoko-7
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/all-talk-anika-moa/S01E001/all-talk-anika-moa-bloopers
+	https://www.maoritelevision.com/mi/tagged/all/Technology
+	https://www.maoritelevision.com/mi/tagged/all/Tikanga
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S06E004/nga-pari-karangaranga-he-piko-he-taniwha-6-upoko-4
+	https://www.maoritelevision.com/mi/tagged/all/M%C4%81ori
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S01E005/nga-pari-karangaranga-he-kuaka-marangaranga-1-upoko-5
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E022/toku-reo-3-upoko-22
+	https://www.maoritelevision.com/mi/kapahaka
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/celebrity-playlist/S01E006/celebrity-playlist-series-1-upoko-6
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tohu-taakaro-o-aotearoa-2018
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E009/tautohetohe-on-road-5-upoko-9
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E021/toku-reo-3-upoko-21
+	https://www.maoritelevision.com/mi/tagged/all/Education
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Ahik%C5%8Dmau%20o%20Hamo%20Te%20Rangi
+	https://www.maoritelevision.com/mi/purongo/kairipoata/mania-clarke
+	https://www.maoritelevision.com/mi/tagged/all/Education
+	https://www.maoritelevision.com/mi/tagged/all/Bay%20of%20Plenty
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/school-training
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E016/waiata-1-upoko-16
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Reo%20M%C4%81ori
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=13
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-j16-men-w6-1000-whiringa-toa
+	https://www.maoritelevision.com/mi/tagged/all/T%C4%81maki%20Makaurau
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/big-sing/nga-wahanga
+	https://www.maoritelevision.com/mi/tagged/all/Wh%C4%81nau%20Ora
+	https://www.maoritelevision.com/mi/kapahaka/rangitane
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2015-waka-ama-sprints/S03E012/2015-waka-ama-sprints-3-upoko-12
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E014/tautohetohe-on-road-5-upoko-14
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/iwi-anthems
+	https://www.maoritelevision.com/mi/tagged/all/Drilling
+	https://www.maoritelevision.com/mi/kapahaka/rangitane
+	https://www.maoritelevision.com/mi/tagged/all/Whanganui
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E008/tautohetohe-on-road-5-upoko-8
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E014/te-tepu
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-kupu-o-te-wiki/S01E001/te-kupu-o-te-wiki-ae
+	https://www.maoritelevision.com/mi/purongo/kairipoata/rahia-timutimu?page=1
+	https://www.maoritelevision.com/mi/tagged/all/Online%20News
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kawekorero/S02E003/kawe-korero-reporters-2-upoko-3
+	https://www.maoritelevision.com/mi/tagged/all/Whanganui
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E010/tautohetohe-on-road-5-upoko-10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S02E021/opaki-2-upoko-21
+	https://www.maoritelevision.com/mi/tagged/all/K%C5%8Dhanga%20Reo
+	https://www.maoritelevision.com/mi/purongo/kairipoata/regan-paranihi
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pukuhohe
+	https://www.maoritelevision.com/mi/purongo/kairipoata/tema-hemi
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S05E004/nga-pari-karangaranga-kuaka-wharau-5-upoko-4
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Reo%20M%C4%81ori
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/matangi-rau/S06E009/matangi-rau-6-upoko-9
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kapa
+	https://www.maoritelevision.com/mi/tagged/all/Online%20News
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tradition-on-plate/S01E001/tradition-on-plate-1-upoko-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tautohetohe-on-road/S05E011/tautohetohe-on-road-5-upoko-11
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/whawhai-fight-night/S01E012/whawhai-fight-night-1-upoko-12
+	https://www.maoritelevision.com/mi/tagged/all/Health
+	https://www.maoritelevision.com/mi/tagged/all/Disability
+	https://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/nga-mahuri/183502
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kitchen-kura
+	https://www.maoritelevision.com/mi/tagged/all/Netball
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-tepu/S10E026/te-tepu
+	https://www.maoritelevision.com/mi/tagged/all/Waikato?page=2
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/funny-whare
+	https://www.maoritelevision.com/mi/kapahaka/tearawa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/hunting-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/sidewalk-karaoke/S02E012/sidewalk-karaoke-2-upoko-12
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu/groups/nga-pa-haka-o-te-wairoa/206049
+	https://www.maoritelevision.com/mi/tagged/all/Police
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/tagged/all/Motiti
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-bake/S02E001/whanau-bake-2-upoko-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/hunting-aotearoa?available=0
+	https://www.maoritelevision.com/mi/kapahaka/te-whanganui-tara
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81ti%20Porou
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau/S03E010/aotearoa-series-3-episode-10
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/ahorangi-next-generation/S01E001/ahorangi-next-generation-1-upoko-1
+	https://www.maoritelevision.com/mi/tagged/all/Netball
+	https://www.maoritelevision.com/mi/kapahaka/te-tau-ihu
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-premier-men-w1-500-whiringa-toa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/tagged/all/Dr.%20Pita%20Sharples
+	https://www.maoritelevision.com/mi/tagged/all/Obesity
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/hunting-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S08E007/tauranga-moana-tauranga-tangata-8-upoko-7
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/autaia/S01E023/autaia-series-1-episode-23
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kai-ora?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/matangi-rau/S06E003/matangi-rau-6-upoko-3
+	https://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/nga-mahuri/183488
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pete-and-pio/S01E004/pete-and-pio-1-upoko-4
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pete-and-pio/S01E007/pete-and-pio-1-upoko-7
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kawekorero/S02E138/kawe-korero-reporters-2-upoko-138
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/hunting-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tradition-on-plate/S01E001/tradition-on-plate-1-upoko-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/hunting-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau/S02E068/kia-mau-2-upoko-68
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/native-kitchen
+	https://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/nga-mahuri/183494
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/stage-haka-fusion?available=0
+	https://www.maoritelevision.com/mi/tagged/all/Manukura
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu/groups/nga-pa-haka-o-te-wairoa/206053
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81ti%20Awa
+	https://www.maoritelevision.com/mi/nga-korero/korero-mo-whakaata-maori/kowaiwai
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa?available=0
+	https://www.maoritelevision.com/mi/tagged/all/Film
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/paepae
+	https://www.maoritelevision.com/mi/tagged/all/NZ%20Land%20Wars
+	https://www.maoritelevision.com/mi/tagged/all/History?page=2
+	https://www.maoritelevision.com/mi/tagged/all/Howick%20Hornets
+	https://www.maoritelevision.com/mi/tagged/all/History?page=9
+	https://www.maoritelevision.com/mi/tagged/videos/Poverty
+	https://www.maoritelevision.com/mi/tagged/all/Boarding%20Schools
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?sort=expiring&amp;category=12
+	https://www.maoritelevision.com/mi/tagged/all/Poi
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81i%20T%C5%ABhoe
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/tamaki-paenga-hira
+	https://www.maoritelevision.com/mi/tagged/all/Operation%208
+	https://www.maoritelevision.com/mi/tagged/news/Waka%20ama
+	https://www.maoritelevision.com/mi/tagged/all/Funding
+	https://www.maoritelevision.com/mi/tagged/news/Poverty
+	https://www.maoritelevision.com/mi/tagged/all/Police?amp%3Brows=10
+	https://www.maoritelevision.com/mi/tagged/all/Marae
+	https://www.maoritelevision.com/mi/tagged/all/Maraenui
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Whare%20W%C4%81nanga%20o%20Awanui%C4%81rangi
+	https://www.maoritelevision.com/mi/tagged/videos/Waitangi
+	https://www.maoritelevision.com/mi/kapahaka/te-pou-herenga-o-kia-aroha
+	https://www.maoritelevision.com/mi/tagged/all/Education?page=1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-maori-news?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/stage-haka-fusion
+	https://www.maoritelevision.com/mi/tagged/all/Arts
+	https://www.maoritelevision.com/mi/tagged/all/Auckland%20Girls%20Grammar
+	https://www.maoritelevision.com/mi/tagged/all/Crime?page=10
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=156&amp;sort=expiring
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/road-nats
+	https://www.maoritelevision.com/mi/tagged/all/Tikanga
+	https://www.maoritelevision.com/mi/tagged/all/Protest?page=10
+	https://www.maoritelevision.com/mi/tagged/all/Tour
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=3124&amp;sort=alpha&amp;display_count=yes&amp;view_mode=featured_catch_up&amp;start=9&amp;length=6
+	https://www.maoritelevision.com/mi/tagged/news/Prison?page=6
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/te-ao-maori-news?available=0
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=111
+	http://community.nzdl.org/macron-restoration/jsp/mi/feedback.jsp
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=1550&amp;sort=expiring
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/pakipumeka-aotearoa
+	http://community.nzdl.org/macron-restoration/jsp/mi/about.jsp
+	https://www.maoritelevision.com/mi/tagged/all/Australia
+	https://www.maoritelevision.com/mi/tagged/news/Touch%20Rugby
+	https://www.maoritelevision.com/mi/tagged/all/Ikaroa-R%C4%81whiti
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?category=13&sort=alpha
+	https://www.maoritelevision.com/mi/tagged/all/Regionals%202018%20Te%20Whanganui-a-tara
+	https://www.maoritelevision.com/mi/tagged/all/Fashion?page=3
+	https://www.maoritelevision.com/mi/tagged/all/Speech
+	https://www.maoritelevision.com/mi/tagged/news/Waka%20ama
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/tagged/all/Wharekura
+https://teaomaori.news
+	https://teaomaori.news/mi/tagged/all/Ru%C4%81toki
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/lions-tour--powhiri-2017
+	https://www.maoritelevision.com/mi/nga-korero/korero-mo-whakaata-maori/kowaiwai
+	https://www.maoritelevision.com/mi/tagged/all/K%C4%AB-o-rahi
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/school-training/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/John%20Minto
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/tagged/news/Benji%20Marshall
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/kapahaka/waitaha/groups/te-pao-a-tahu
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Tau%20Ihu
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=7&rows=10
+	https://www.maoritelevision.com/mi/tagged/videos/NPK%20Series%208
+	https://www.maoritelevision.com/mi/tagged/all/New%20Zealand%20Army%20Museum
+	https://www.maoritelevision.com/mi/nga-korero/korero-mo-whakaata-maori/kowaiwai
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/tagged/all/Tennis
+	https://www.maoritelevision.com/mi/tagged/news/T%C5%ABhoe
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/swagger/S02E001/swagger-2-upoko-1
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/sisters/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Kabaddi
+	https://www.maoritelevision.com/mi/tagged/news/Poverty
+	https://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/uk-super-league/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Competition
+	https://www.maoritelevision.com/mi/tagged/shows/Hip%20hop
+	https://www.maoritelevision.com/mi/tagged/news/Te%20Ohu%20Kaimoana
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa?category=3129&tereo=5&sort=alpha
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa?category=1550&start=12&length=6&tereo=5
+	https://www.maoritelevision.com/mi/tagged/all/Gambling
+	https://www.maoritelevision.com/mi/tagged/all/News
+	https://www.maoritelevision.com/mi/tagged/videos/Education?page=2&amp;rows=10
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Whare%20W%C4%81nanga%20o%20Awanui%C4%81rangi?page=1&rows=10
+	https://www.maoritelevision.com/mi/tagged/videos/Food
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/2018-waka-ama-sprint-nationals/matakihia
+	https://www.maoritelevision.com/mi/tagged/news/Drugs?page=3
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu/groups/nga-pa-haka-o-te-wairoa
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=4&amp%3Brows=10
+	https://teaomaori.news/mi/tagged/all/Education
+	https://www.maoritelevision.com/mi/kapahaka/waitaha/groups/te-pao-a-tahu/150805
+	https://www.maoritelevision.com/mi/tagged/videos/Ng%C4%81ti%20Paoa
+	https://www.maoritelevision.com/mi/nga-korero/korero-mo-whakaata-maori/poari-me-nga-kaiwhakahaere/glenn-hawkins
+	https://teaomaori.news/mi/tagged/all/Tennis
+	https://www.maoritelevision.com/mi/tagged/news/Drugs?page=2
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/poly-movement-kaikanikani/matakihia
+	https://www.maoritelevision.com/mi/pouaka-whakaata/matakihia?sort=popular&category=156
+	https://www.maoritelevision.com/mi/kapahaka/kahungunu/groups/te-rangiura-o-wairarapa
+	https://www.maoritelevision.com/mi/tagged/all/James%20Cook%20High%20School%20-%20Te%20Pou%20Herenga%20Waka
+	https://www.maoritelevision.com/mi/tagged/all/Justice
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/code/matakihia
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pakipumeka-te-rahina?available=0
+	https://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/uk-super-league/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Wellington?page=7&amp%3Brows=10
+	https://www.maoritelevision.com/mi/tagged/all/Police?page=5&amp;rows=10
+	https://www.maoritelevision.com/mi/tagged/news/Ahuwhenua?page=1&rows=10
+	https://www.maoritelevision.com/mi/tagged/videos/Education?page=2&amp%3Brows=10
+	https://www.maoritelevision.com/mi/tagged/news/Ng%C4%81ti%20Awa
+	https://www.maoritelevision.com/mi/tagged/all/Fitness?page=8
+	https://www.maoritelevision.com/mi/kowhiri/nga-kaitono/emma-jane-mihaere-kingi
+	https://www.maoritelevision.com/mi/tagged/all/Brazil
+	https://www.maoritelevision.com/mi/tagged/all/Ng%C4%81ti%20P%C5%8Dneke%20Young%20M%C4%81ori%20Club?page=1&rows=10
+	https://www.maoritelevision.com/mi/tagged/all/Pamapuria
+	https://www.maoritelevision.com/mi/tagged/all/Snr%20Master%20Men
+	https://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/brian-jacques-redwall/matakihia
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/witbn-programme-exchange/matakihia
+	https://www.maoritelevision.com/mi/tagged/all/Te%20Wh%C4%81nau-%C4%81-Apanui?page=2
+	https://www.maoritelevision.com/mi/tagged/news/King%20Tuheitia
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2017-waka-ama-sprints/S01E001/2017-waka-ama-sprints-premier-men-w12-500-final
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/he-iwi-whakapono/S02E007/he-iwi-whakapono-2-upoko-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E015/opaki-1-upoko-15
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whats-tumoanas/S02E001/whats-tumoanas-2-upoko-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whawhai-fight-night/S13E312/whawhai-fight-night-1-upoko-9
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/all-talk-anika-moa/S02E002/all-talk-anika-moa-2-upoko-2
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/pukuhohe
+	http://www.maoritelevision.com/mi/kapahaka/te-taumata/groups/nga-tumanako/169148
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E021/toku-reo-3-upoko-21
+	http://www.maoritelevision.com/mi/tagged/all/Bay%20of%20Plenty
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kete-korero/S01E010/kete-korero
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/my-country-song/S02E007/my-country-song-2-upoko-7
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/find-me-maori-bride
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E003/nga-pari-karangaranga-nga-kapa-haka-o-te-tai-tokerau
+	http://www.maoritelevision.com/mi/tagged/all/K%C5%8Dwhiri%20-%20Election%202017
+	http://www.maoritelevision.com/mi/tagged/all/Technology
+	http://www.maoritelevision.com/mi/tagged/all/Wellington?page=5
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whats-tumoanas/S01E004/whats-tumoanas-1-upoko-4
+	http://www.maoritelevision.com/mi/tagged/all/Ikaroa-R%C4%81whiti
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-gld-master-men-w6-1000-whiringa
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-mid-men-w12-250-whiringa-toa
+	http://www.maoritelevision.com/mi/purongo/kairipoata/aroha-treacher
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/2016-waka-ama-sprints/S01E001/2016-waka-ama-sprints-j19-women-w6-1000-whiringa-toa
+	https://www.maoritelevision.com/mi/nga-hotaka-katoa/native-affairs
+	http://www.maoritelevision.com/mi/kapahaka
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E021/waiata-1-upoko-21
+	https://www.maoritelevision.com/mi/pouaka-whakaata/rarangi-hotaka
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/native-affairs?available=0
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	http://www.maoritelevision.com/mi/tagged/all/Reo
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/homai-te-pakipaki/S09E005/homai-te-pakipaki-wahanga-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S03E044/ako-3-upoko-44
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/whanau-living/S02E016/whanau-living-2-upoko-16
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E019/waiata-1-upoko-19
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/waiata/S01E016/waiata-1-upoko-16
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/big-sing/S01E006/big-sing-1-upoko-6
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/big-sing/S01E001/big-sing-1-upoko-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/opaki/S01E013/opaki-1-upoko-13
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/koroua/S01E016/koroua-1-upoko-16
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kia-mau
+	http://www.maoritelevision.com/mi/tagged/all/Housing%20New%20Zealand
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S01E018/nga-tangata-taumata-rau-nga-tamariki-o-te-kohu-1
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E001/nga-pari-karangaranga-nga-korero-o-ngati-wai-7-upoko
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kete-korero/S01E004/kete-korero-0
+	http://www.maoritelevision.com/mi/tagged/all/Obesity
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/ako/S03E021/ako-3-upoko-21
+	http://www.maoritelevision.com/mi/tagged/all/M%C4%81ngere
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/kuia/nga-kaiwhakataki
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/media-take/S03E004/media-take-extended-series-3-episode-6
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/only-aotearoa?available=0
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/toku-reo/S03E022/toku-reo-3-upoko-22
+	https://www.maoritelevision.com/mi/purongo/purongo-hou/te-reo-maori
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/playlist/S01E004/playlist-1-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/big-sing/S01E004/big-sing-1-upoko-4
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/stage-haka-fusion
+	http://www.maoritelevision.com/mi/nga-hotaka-katoa/nga-pari-karangaranga/S07E007/nga-pari-karangaranga-nga-rakau-tupuna-7-upoko-7
+	http://www.maoritelevision.com/mi/node/101110/S01E021/te-mana-kuratahi-2017-nga-pou-o-tararua-whakaeke
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/2018-waka-ama-sprint-nationals/S01E001/2018-waka-ama-sprints-j19-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/anzac-2018
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/2017-waka-ama-sprints/S01E001/2017-waka-ama-sprints-premier-women
+	http://www.maoritelevision.com/mi/kapahaka/tearawa/groups/ngati-tarawhai/198332
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/school-training/S01E010/school-training-series-1-upoko-10
+	http://www.maoritelevision.com/mi/node/101110/S01E027/te-mana-kuratahi-2017-tkkm-o-nga-mokopuna-waiata-ringa
+	http://www.maoritelevision.com/mi/purongo/purongo-hou/nga-whakataetae-kapa-haka
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S08E002/nga-pari-karangaranga-8-upoko-2
+	http://www.maoritelevision.com/mi/node/101110/S01E025/te-mana-kuratahi-2017-te-ra-o-te-uru-tkkm-o-ngati-ruanui-me-te-kura-o-nga-3
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S08E009/ngati-whakaue-whakanuia-8-upoko-9
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/hunting-aotearoa/S10E001/hunting-aotearoa-10-upoko-1
+	http://www.maoritelevision.com/mi/node/101110/S01E025/te-mana-kuratahi-2017-te-ra-o-te-uru-tkkm-o-ngati-ruanui-me-te-kura-o-nga-6
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/hunting-aotearoa?available=0
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/cube/S01E001/cube-kakahu-huatau-spain-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/cube/S01E001/cube-mahi-ngahau-diwali-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/2017-waka-ama-sprints/S01E001/2017-waka-ama-sprints-j16-men-w12-500
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/pakipumeka-aotearoa-new-zealand-documentaries/S06E001/freezing
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawekorero/S02E144/kawe-korero-reporters-2-upoko-144
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/on-ladder-0/S01E002/on-ladder-series-1-upoko-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/cube/S01E001/cube-kakahu-huatau-kmart-summer-preview-sydney-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/game-bros/S02E002/game-bros-2-upoko-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/2018-waka-ama-sprint-nationals/S06E003/waka-ama-sprints-2018-6
+	http://www.maoritelevision.com/mi/kapahaka/mataatua/groups/opotiki-mai-tawhiti/196458
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/whats-tumoanas/S02E007/whats-tumoanas-2-upoko-7
+	http://www.maoritelevision.com/mi/node/101110/S01E026/te-mana-kuratahi-2017-te-roopu-atawhai-waiata-ringa
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S08E007/te-whare-korero-o-rongowhakaata-8
+	http://community.nzdl.org/macron-restoration/jsp/mi/about.jsp
+	http://www.maoritelevision.com/mi/tagged/all/social%20media
+	http://www.maoritelevision.com/mi/kapahaka/te-whenua-moemoea/groups/nga-rauika-a-maui/199662
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/opaki/S01E054/opaki-1-upoko-54
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/only-aotearoa/S01E002/only-aotearoa-de-colonization
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E083/kawe-korero-reporters-2-upoko-83
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/whats-tumoanas/S02E006/whats-tumoanas-2-upoko-6
+	http://www.maoritelevision.com/mi/purongo/purongo-hou/nga-whakataetae-waka-ama-a-motu
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/only-aotearoa/S01E006/only-aotearoa-agree-disagree
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/mahinga-kai/S01E004/mahinga-kai-1-upoko-4-0
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/haka-life
+	http://www.maoritelevision.com/mi/kapahaka/tearawa/groups/ngati-tarawhai/198326
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/whanau-bake/S02E008/whanau-bake-2-upoko-8
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E100/kawe-korero-reporters-2-upoko-100
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/game-bros/S02E007/game-bros-2-upoko-7
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/only-aotearoa/S01E004/only-aotearoa-series-1-upoko-4
+	http://www.maoritelevision.com/mi/kapahaka/tearawa/groups/ngati-tarawhai/198330
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/te-po-mekemeke/S06E025/te-po-mekemeke-6-upoko-25
+	http://www.maoritelevision.com/mi/kapahaka/tamaki-makaurau
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawekorero/S02E180/kawekorero-reporters-2-upoko-180
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S08E005/ngati-hine-a-hine-a-maru-8-upoko-5
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S04E001/nga-pari-karangaranga-ina-te-mahi-4
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/pukeroa-oruawhata/184619
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/te-po-mekemeke/S06E020/te-po-mekemeke-6-upoko-20
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawekorero/S02E188/kawekorero-reporters-2-upoko-188
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/hunting-aotearoa/S10E002/hunting-aotearoa-10-upoko-2
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/whanau-bake
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/2018-waka-ama-sprint-nationals/S01E001/2018-waka-ama-sprints-3
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/stage-haka-fusion
+	http://www.maoritelevision.com/mi/node/101110/S01E002/te-mana-kuratahi-2017-te-rangi-aniwaniwa-nga-mahuri-whakawatea
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawekorero/S02E137/kawe-korero-reporters-2-upoko-137
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S07E002/nga-pari-karangaranga-nga-rakau
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-pari-karangaranga/S07E003/nga-pari-karangaranga-nga-rakau
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E077/kawe-korero-reporters-2-upoko-77
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E123/kawe-korero-reporters-2-upoko-123
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E091/kawe-korero-2-upoko-91
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/all-talk-anika-moa/S01E007/all-talk-anika-moa-1-upoko-7
+	http://www.maoritelevision.com/mi/node/101110/S01E022/te-mana-kuratahi-2017-te-kura-o-te-paroa-poi
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/hunting-aotearoa?available=0
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawekorero/S02E132/kawe-korero-reporters-2-upoko-132
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E099/kawe-korero-reporters-2-upoko-99
+	http://www.maoritelevision.com/mi/tagged/all/Nanaia%20Mahuta
+	http://community.nzdl.org/macron-restoration/jsp/mi/feedback.jsp
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/te-kauta/S04E043/te-kauta
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-maori-o-porirua/183800
+	http://community.nzdl.org/macron-restoration/jsp/mi/main.jsp
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/nga-tangata-taumata-rau-0/S02E005/nga-tangata-taumata-rau-rawiri
+	http://www.maoritelevision.com/mi/node/101110/S01E027/te-mana-kuratahi-2017-tkkm-o-nga-mokopuna-waiata-tira
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-o-te-koutu%20/183798
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/swagger/S04E010/swagger-4-upoko-10
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/media-take/S04E019/media-take-4-upoko-19
+	http://www.maoritelevision.com/mi/node/101110/S01E002/te-mana-kuratahi-2017-te-rangi-aniwaniwa-nga-mahuri-waiata-ringa
+	https://www.maoritelevision.com/mi/kapahaka/tainui
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/ahikaroa/S01E004/ahikaroa-ep-4-face-time
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/only-aotearoa/S01E006/only-aotearoa-whanau-feud-part-1
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/kawe-korero/S02E102/kawe-korero-reporters-2-upoko-102
+	http://www.maoritelevision.com/mi/kapahaka/te-mana-kuratahi/groups/te-kura-maori-o-nga-tapuwae/184977
+	http://www.maoritelevision.com/mi/tagged/all/Kiingitanga
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/only-aotearoa/S01E008/only-aotearoa-basketball-scholarship
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/cube/S01E001/cube-kakahu-huatau-spain-0
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/all-talk-anika-moa/S01E001/all-talk-anika-moa-episode-2-stan-sings
+	http://www.maoritelevision.com/mi/tagged/all/Te%20Reo%20M%C4%81ori
+	http://www.maoritelevision.com/mi/node/101110/S01E002/te-mana-kuratahi-2017-te-rangi-aniwaniwa-nga-mahuri-haka
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/opaki/S01E052/opaki-1-upoko-52
+	http://www.maoritelevision.com/mi/tagged/all/Tauranga
+	https://www.maoritelevision.com/mi/purongo/purongo-hou/nga-whakataetae-waka-ama-a-motu
+	http://www.maoritelevision.com/mi/kapahaka
+	http://www.maoritelevision.com/mi/pouaka-whakaata/nga-hotaka-katoa/playlist/S01E041/playlist-1-upoko-41
Index: /other-projects/maori-lang-detection/conf/log4j.properties
===================================================================
--- /other-projects/maori-lang-detection/conf/log4j.properties	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/log4j.properties	(revision 33635)
@@ -0,0 +1,48 @@
+# See for basic example https://stackoverflow.com/questions/125 32339/no-appenders-could-be-found-for-loggerlog4j
+# For better examples, see /greenstone/greenstone3/web/WEB-INF/classes/log4j.properties
+# Manual (examples use XML versions of log4j config files): https://logging.apache.org/log4j/2.x/manual/configuration.html
+# https://logging.apache.org/log4j/2.x/manual/configuration.html#Properties
+# HH (0-24 hrs) vs hh (0-12 hrs, am/pm): https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
+# https://logging.apache.org/log4j/1.2/manual.html
+####################### 
+
+# define some reusable properties
+maori-lang-detector.home=/Scratch/ak19/gs3-extensions/maori-lang-detection
+maori-lang-detector.logpath=${maori-lang-detector.home}/logs
+
+# Let's log everything from the most basic DEBUG msgs on (so INFO, WARN, ERROR too)
+# Also want to log to the console, see
+# https://stackoverflow.com/questions/3382985/how-to-make-log4j-to-write-to-the-console-as-well
+
+log4j.rootCategory=DEBUG, console, mldlog
+
+# Define Console Appender
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+# Define the layout for console appender. If you do not define it, you will get an error
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.Target=System.err
+
+# We're saying our log "mldlog" is some file that gets appended to,
+# and it will write it out to "maori-lang-detector.log" located in whatever we defined the "logpath" to be
+
+# Old way of logging:
+#log4j.appender.mldlog=org.apache.log4j.RollingFileAppender
+#log4j.appender.mldlog.File=${maori-lang-detector.logpath}/maori-lang-detector.log
+## https://howtodoinjava.com/log4j/log4j-rolling-file-appender/
+##log4j.appender.mldlog.MaxBackupIndex=10
+
+# We want a DailyRollingFileAppender.
+# Its behaviour is explained at
+# https://examples.javacodegeeks.com/enterprise-java/log4j/log4j-rolling-daily-file-example/
+log4j.appender.mldlog=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.mldlog.File=${maori-lang-detector.logpath}/maori-lang-detector.log
+log4j.appender.mldlog.DatePattern='.'yyyy-MM-dd
+	# set to roll over every day, can change this to weekly/every min etc, see https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
+
+# if you have more logs, define category, additivity, appender and appender.File for it as above
+# for each log also set the format of the output as below, after setting the layout to use PatternLayout class to parse the format
+
+log4j.appender.mldlog.layout=org.apache.log4j.PatternLayout
+log4j.appender.mldlog.layout.ConversionPattern=%-5p: [%d{yyyy-MM-dd HH:mm:ss}] %C{1}.%M()%n %m%n
+
+#######################
Index: /other-projects/maori-lang-detection/conf/log4j.properties.in
===================================================================
--- /other-projects/maori-lang-detection/conf/log4j.properties.in	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/log4j.properties.in	(revision 33635)
@@ -0,0 +1,45 @@
+# See for basic example https://stackoverflow.com/questions/125 32339/no-appenders-could-be-found-for-loggerlog4j
+# For better examples, see /greenstone/greenstone3/web/WEB-INF/classes/log4j.properties
+# Manual (examples use XML versions of log4j config files): https://logging.apache.org/log4j/2.x/manual/configuration.html
+# https://logging.apache.org/log4j/2.x/manual/configuration.html#Properties
+# HH (0-24 hrs) vs hh (0-12 hrs, am/pm): https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
+# https://logging.apache.org/log4j/1.2/manual.html
+####################### 
+
+# define some reusable properties
+maori-lang-detector.home=@maori-lang-detector.home@
+maori-lang-detector.logpath=${maori-lang-detector.home}/web/logs
+
+# Let's log everything from the most basic DEBUG msgs on (so INFO, WARN, ERROR too)
+log4j.rootCategory=DEBUG, mldlog
+
+# copied from GS3
+log4j.category.web.servlets.DirectInput=INFO, mldlog
+# Prevent logging duplicating its messages into rootLogger:
+# https://stackoverflow.com/questions/9208710/java-log4j-log-to-both-category-and-root-logger
+log4j.additivity.web.servlets.DirectInput=false
+
+# We're saying our log "mldlog" is some file that gets appended to,
+# and it will write it out to "maori-lang-detector.log" located in whatever we defined the "logpath" to be
+
+# Old way of logging:
+#log4j.appender.mldlog=org.apache.log4j.RollingFileAppender
+#log4j.appender.mldlog.File=${maori-lang-detector.logpath}/maori-lang-detector.log
+## https://howtodoinjava.com/log4j/log4j-rolling-file-appender/
+##log4j.appender.mldlog.MaxBackupIndex=10
+
+# We want a DailyRollingFileAppender.
+# Its behaviour is explained at
+# https://examples.javacodegeeks.com/enterprise-java/log4j/log4j-rolling-daily-file-example/
+log4j.appender.mldlog=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.mldlog.File=${maori-lang-detector.logpath}/maori-lang-detector.log
+log4j.appender.mldlog.DatePattern='.'yyyy-MM-dd
+	# set to roll over every day, can change this to weekly/every min etc, see https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
+
+# if you have more logs, define category, additivity, appender and appender.File for it as above
+# for each log also set the format of the output as below, after setting the layout to use PatternLayout class to parse the format
+
+log4j.appender.mldlog.layout=org.apache.log4j.PatternLayout
+log4j.appender.mldlog.layout.ConversionPattern=%-5p: [%d{yyyy-MM-dd HH:mm:ss}] %C{1}.%M()%n %m%n
+
+#######################
Index: /other-projects/maori-lang-detection/conf/possible-product-sites.txt
===================================================================
--- /other-projects/maori-lang-detection/conf/possible-product-sites.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/possible-product-sites.txt	(revision 33635)
@@ -0,0 +1,1184 @@
+DE : http://hargasamsungandroid.info
+	http://hargasamsungandroid.info/mi/kinemaster-pro-video-editor-android
+US : https://www.datemypet.com
+	https://www.datemypet.com/mi/7-simple-ways-to-make-him-fall-in-love
+CN : https://www.outstandingdm.com
+	https://www.outstandingdm.com/mi/hanging-metal-essential-oil-diffuser-for-car-612.html
+CN : http://www.wedacdisplays.com
+	http://www.wedacdisplays.com/mi/small-cosmetic-display-unit.html
+CN : http://www.bestshelly.com
+	http://www.bestshelly.com/mi/about-us
+ : http://snau.com.ua
+	http://snau.com.ua/mi/apply-online/
+CZ : https://www.nelzu.cz
+	https://www.nelzu.cz/mi/
+US : http://www.chuanqiangmotor.com
+	http://www.chuanqiangmotor.com/mi/contact-us/
+US : https://www.td-casting.com
+	https://www.td-casting.com/mi/die-casting/
+US : https://www.conele-mixer.com
+	https://www.conele-mixer.com/mi/case/
+CN : http://www.world-starter.com
+	http://www.world-starter.com/mi/contact-us/
+US : https://www.dshprecision.com
+	https://www.dshprecision.com/mi/
+US : https://soccerbetshoot.com
+	https://soccerbetshoot.com/mi/ticket-football-predictions/
+NL : https://www.martinvrijland.nl
+	https://www.martinvrijland.nl/mi/te-mana-hinengaro/Ko-te-nuinga-ake-o-nga-tangata-kei-te-timata-ki-te-kite-kei-te-noho-tatou-i-roto-i-te-whakaata-ko-te-aha-tenei/
+US : https://www.csunplugged.org
+	https://www.csunplugged.org/mi/principles/
+CN : https://www.bofanpc.com
+	https://www.bofanpc.com/mi/
+US : https://www.zjyongqi.com
+	https://www.zjyongqi.com/mi/
+US : https://way2enjoy.com
+	https://way2enjoy.com/matt-gillanders/mi/?matt-gillanders
+US : http://www.gmmdjx.com
+	http://www.gmmdjx.com/mi/
+US : https://www.everfineplastics.com
+	https://www.everfineplastics.com/mi/about-us/
+US : https://www.aquark.com.cn
+	https://www.aquark.com.cn/mi/contact-us
+US : http://www.nbbvc.com
+	http://www.nbbvc.com/mi/accessories/
+CN : https://www.nfyo.com
+	https://www.nfyo.com/mi/
+US : https://www.sdspraybooth.com
+	https://www.sdspraybooth.com/mi/about-us/
+US : https://www.nickel-alloy.net
+	https://www.nickel-alloy.net/mi/
+US : https://www.valve-pipe-fitting.com
+	https://www.valve-pipe-fitting.com/mi/
+US : https://www.airpullfilter.com
+	https://www.airpullfilter.com/mi/mann-screw-air-compressor-replacement-parts-air-oil-separator.html
+US : http://www.carlhenryglobal.com
+	http://www.carlhenryglobal.com/mi/sitemap
+US : https://www.fctele.com
+	https://www.fctele.com/mi/about-us/
+US : http://www.zhonghe222.com
+	http://www.zhonghe222.com/mi/tower-spring-conical-spring.html
+US : http://www.meihua-wm.com
+	http://www.meihua-wm.com/mi/photo/staff-photo/
+CA : https://cloudsfeed.com
+	https://cloudsfeed.com/mi/entertainment/fifa-world-cup/
+RU : http://bopt.org
+	http://bopt.org/mi/
+US : https://vimeo.roseconverter.com
+	https://vimeo.roseconverter.com/mi/
+UA : http://ukraine.admission.center
+	http://ukraine.admission.center/mi/about-ukraine/
+US : http://www.hzzjair.com
+	http://www.hzzjair.com/mi/
+US : http://www.cnfreda.com
+	http://www.cnfreda.com/mi/
+CN : http://www.jsbotanics.com
+	http://www.jsbotanics.com/mi/bilberry-extract.html
+ : http://wintoflash.com
+	http://wintoflash.com/overview/mi/
+US : http://www.newbaoquan.com
+	http://www.newbaoquan.com/mi/
+JP : https://forexmania.org
+	https://forexmania.org/mi/forex-brokers-ranking/
+US : http://www.ledecofr.com
+	http://www.ledecofr.com/mi/news/
+US : https://www.sinodryair.com
+	https://www.sinodryair.com/mi/
+US : http://carlhenryglobal.com
+	http://carlhenryglobal.com/mi/sitemap
+US : https://escolhasegura.com
+	https://escolhasegura.com/mi/
+US : https://www.wbuybuy.com
+	https://www.wbuybuy.com/3004541678@qq.com/item/detail/811v5359/mi/
+US : http://www.fangtongpharm.com
+	http://www.fangtongpharm.com/mi/
+US : http://www.nantaidiesel.com
+	http://www.nantaidiesel.com/mi/
+DK : http://www.holmbygden.se
+	http://www.holmbygden.se/mi/foreningar/holms-sportklubb/stod-holms-sk-gratis/
+US : http://www.sokenswitch.com
+	http://www.sokenswitch.com/mi/red-dot-round-rocker-switch-small-switches-10a-250vac.html
+US : http://www.omnicnc.com
+	http://www.omnicnc.com/mi/door-cnc-router.html
+US : http://www.tongyujiaju.com
+	http://www.tongyujiaju.com/mi/
+US : https://www.bestforexeas.com
+	https://www.bestforexeas.com/mi/
+US : http://www.bigrollscloth.com
+	http://www.bigrollscloth.com/mi/
+US : https://www.risenltd.com
+	https://www.risenltd.com/mi/sales-services/
+RU : http://www.treningmozga.com
+	http://www.treningmozga.com/mi/iq/iq-test-1/iq-test.php
+US : https://www.gigalight.com
+	https://www.gigalight.com/mi/
+CN : https://www.xinyuesteel.com
+	https://www.xinyuesteel.com/mi/
+CN : https://www.aootan.com
+	https://www.aootan.com/mi/
+US : http://www.cnsongben.com
+	http://www.cnsongben.com/mi/news/the-new-official-web-site-of-qingdao-songben-packaging-machinery-co-ltd-has-put-into-running-formally
+US : http://www.haiwensteroids.com
+	http://www.haiwensteroids.com/mi/about-us/
+US : http://www.litbright-candles.com
+	http://www.litbright-candles.com/mi/about-us/
+US : http://www.chinachairtable.com
+	http://www.chinachairtable.com/mi/
+US : http://www.xfinsulation.com
+	http://www.xfinsulation.com/mi/about-us/
+US : http://www.fumeiseating.com
+	http://www.fumeiseating.com/mi/fm-315.html
+US : http://www.gecko-kalimba.com
+	http://www.gecko-kalimba.com/mi/contact-us/
+CN : https://www.xianglin-plastics.com
+	https://www.xianglin-plastics.com/mi/
+US : http://www.chinabosun.com
+	http://www.chinabosun.com/mi/
+US : http://www.jpslurrypump.com
+	http://www.jpslurrypump.com/mi/abroad-market/
+HK : https://www.nbhalltools.com
+	https://www.nbhalltools.com/mi/
+US : https://www.goethe.de
+	https://www.goethe.de/ins/nz/mi/kul/ser/uef.html
+US : http://www.bst-elecs.com
+	http://www.bst-elecs.com/mi/125w-quad-output-switching-power-supply-itq-125w.html
+CN : https://www.richest-group.com
+	https://www.richest-group.com/mi/
+FR : https://www.planetkeyboard.com
+	https://www.planetkeyboard.com/mi/
+US : http://www.qjfiberglass.com
+	http://www.qjfiberglass.com/mi/self-adhesive-fiberglass-mesh-tape-2.html
+US : http://www.aiweidylight.com
+	http://www.aiweidylight.com/mi/trade-capacity/
+US : http://www.secondhormone.com
+	http://www.secondhormone.com/mi/news/customers-visited-our-facility
+US : http://www.hanshang-hydraulic.com
+	http://www.hanshang-hydraulic.com/mi/svk10-flow-diverters.html
+US : http://www.siwoc.com
+	http://www.siwoc.com/mi/about-us/
+CN : https://www.bescatray.com
+	https://www.bescatray.com/mi/
+US : http://www.ttyzfilter.com
+	http://www.ttyzfilter.com/mi/
+CN : https://www.tianseoffice.com
+	https://www.tianseoffice.com/mi/
+US : http://www.ncpcvet.com
+	http://www.ncpcvet.com/mi/download/
+US : http://www.sdtzgloves.com
+	http://www.sdtzgloves.com/mi/
+US : http://www.lbeerequipment.com
+	http://www.lbeerequipment.com/mi/fermentation-tank%EF%BC%88xuzhou.html
+US : http://pixelhelper.org
+	http://pixelhelper.org/mi
+US : http://www.sdcncrouter.com
+	http://www.sdcncrouter.com/mi/case/samples/
+US : https://www.fenghuo-led.com
+	https://www.fenghuo-led.com/mi/
+US : http://topbitcoincard.com
+	http://topbitcoincard.com/mi/bitcoin-debit-card-faq/
+ : http://www.porousfabric.com
+	http://www.porousfabric.com/mi/about-us/
+US : http://www.zjnbzy.com
+	http://www.zjnbzy.com/mi/news/the-absolute-beginners-guide-to-google-analytics
+US : https://www.yourcloudlibrary.com
+	https://www.yourcloudlibrary.com/mi/fire-tikiake/
+US : http://www.qypaperbox.com
+	http://www.qypaperbox.com/mi/cinderella-paper-box-with-ribbon.html
+US : https://www.nbkeming.com
+	https://www.nbkeming.com/mi/
+US : https://wbuybuy.com
+	https://wbuybuy.com/category/el8/mi/
+US : https://specialfinds.com
+	https://specialfinds.com/mi/
+US : https://www.pldyes.com
+	https://www.pldyes.com/mi/
+US : http://www.sinemagnetic.com
+	http://www.sinemagnetic.com/mi/about-us/
+US : http://www.gemnice.com
+	http://www.gemnice.com/mi/silver-midi-hoop-earring.html
+US : http://www.jlgrating.com
+	http://www.jlgrating.com/mi/railing-stanchion.html
+ : https://wintoflash.com
+	https://wintoflash.com/home/mi/
+US : http://www.cannapresso.com
+	http://www.cannapresso.com/mi/warranty-terms/
+US : http://www.zichengpacking.net
+	http://www.zichengpacking.net/mi/
+CN : https://www.sdtoplit.com
+	https://www.sdtoplit.com/mi/
+US : http://www.windsolarchina.com
+	http://www.windsolarchina.com/mi/news/wind-solar-and-grid-hybrid-power-supply-system
+CN : http://www.eburn-burner.com
+	http://www.eburn-burner.com/mi/
+US : http://www.teda-hydraulic.com
+	http://www.teda-hydraulic.com/mi/honor/
+US : https://www.czzhit.com
+	https://www.czzhit.com/mi/about-us/
+US : https://csunplugged.org
+	https://csunplugged.org/mi/principles/
+RU : http://swiss-apteka.com
+	http://swiss-apteka.com/mi/
+US : http://www.aluminum-profiles-supplier.com
+	http://www.aluminum-profiles-supplier.com/mi/factory-picture/
+US : http://binaryoptionsindicators.com
+	http://binaryoptionsindicators.com/mi/
+US : http://www.eternal-friendship.com
+	http://www.eternal-friendship.com/mi/perfect-teflon-tape.html
+FR : https://blog.sexy-rugby.com
+	https://blog.sexy-rugby.com/mi/
+US : https://blog.os-store.com
+	https://blog.os-store.com/mi/
+US : https://www.tkthvac.com
+	https://www.tkthvac.com/mi/about-us/
+US : https://tattoosartideas.com
+	https://tattoosartideas.com/mi/mo-te-kotiro-mo-kotiro/
+US : http://www.hc-machine.net
+	http://www.hc-machine.net/mi/
+US : http://www.mytrickstips.com
+	http://www.mytrickstips.com/mi/free-xbox-homai-k%C4%81ri/
+US : http://www.qikangcn.com
+	http://www.qikangcn.com/mi/
+US : http://www.quickcncmachine.com
+	http://www.quickcncmachine.com/mi/about-us/trustpass-profile/
+US : http://www.roserealdoll.com
+	http://www.roserealdoll.com/mi/video/
+US : http://www.longs-motor.com
+	http://www.longs-motor.com/mi/hybrid-stepper-motor-nema24.html
+US : http://www.pressurelantern.com
+	http://www.pressurelantern.com/mi/news/1188
+US : https://www.roodergroup.com
+	https://www.roodergroup.com/mi/folding-electric-scooter-r803t-with-10-inch-wheel-front-rear-led-light-500w-brushless-motor-40kmh.html
+US : https://betcsgo.eu
+	https://betcsgo.eu/mi/
+US : http://tanzaniacrusher.org
+	http://tanzaniacrusher.org/mi/jaw-crusher/
+US : http://www.nbhltrade.com
+	http://www.nbhltrade.com/mi/
+US : http://jobdescriptionsample.org
+	http://jobdescriptionsample.org/mi/aircraft-launch-and-recovery-specialists-job-description-responsibility-template-and-assignments/1100/
+US : https://livecasino.ie
+	https://livecasino.ie/mi/%C4%81raia/
+CA : https://goldenwebawards.com
+	https://goldenwebawards.com/mi/
+US : http://www.autosunsoul.com
+	http://www.autosunsoul.com/mi/about-us/company-profile/
+US : http://mytrickstips.com
+	http://mytrickstips.com/mi/
+US : http://indigenousblogs.com
+	http://indigenousblogs.com/mi/
+US : https://portal.smart-project.info
+	https://portal.smart-project.info/smart/lang/mi
+US : http://forexwelcomebonus.com
+	http://forexwelcomebonus.com/mi/about-us/
+US : http://www.10turntables.com
+	http://www.10turntables.com/mi/contact-us-2/
+DE : https://transposh.org
+	https://transposh.org/mi/version-0-6-2-lots-of-small-candies/
+ : http://www.lovphonegoogle.com
+	http://www.lovphonegoogle.com/mi/lovphone-products/sporting-goods/mobile-phone-armband/armband-for-samsung/
+CA : http://newsrule.com
+	http://newsrule.com/mi/tag/samuel-gibbs/
+DE : http://transposh.org
+	http://transposh.org/mi/category/general/
+CN : https://www.phhydraulic.com
+	https://www.phhydraulic.com/mi/
+US : http://blicanada.net
+	http://blicanada.net/mi/whakam%C4%81tautau-whakawhiti-rau/
+US : https://www.coinfalls.com
+	https://www.coinfalls.com/mi/huira/huira-phi-coinfalls-k%C4%93mu/
+US : http://mobhax.com
+	http://mobhax.com/mi/clash-royale-hack-link/
+DE : https://traynews.com
+	https://traynews.com/mi/news/how-will-the-ico-market-change-in-2018/
+UA : http://umsa.admission.center
+	http://umsa.admission.center/mi/apply-online/
+US : http://www.szhaiwang.com
+	http://www.szhaiwang.com/mi/
+CA : http://omg-solutions.com
+	http://omg-solutions.com/mi/gps-tracker/
+HK : http://www.wtftools.com
+	http://www.wtftools.com/mi/
+US : http://www.rofvape.com
+	http://www.rofvape.com/mi/
+CN : https://www.herbal-ingredients.com
+	https://www.herbal-ingredients.com/mi/
+US : http://www.cnsdmetal.com
+	http://www.cnsdmetal.com/mi/
+CN : https://www.kinrom.com
+	https://www.kinrom.com/mi/hydro-forming-pipe.html
+US : http://www.damiser.com
+	http://www.damiser.com/mi/news/new-showroom-coming
+GB : https://itstechschool.com
+	https://itstechschool.com/mi/decrease-bounce-rate-on-your-wordpress-site/
+US : http://www.tactile-indicator.com
+	http://www.tactile-indicator.com/mi/stamping-parts-cnc-stamped-parts-sheet-metal-stamping-punched-products-13.html
+FR : https://www.casino.uk.com
+	https://www.casino.uk.com/mi/k%C4%93mu/p%C5%8Dtekiteki-rakau-pai-free-i'ai-ipurangi/
+GB : https://centrallanguageschool.com
+	https://centrallanguageschool.com/mi/enrol/enrol-online
+US : https://www.livehoster.com
+	https://www.livehoster.com/mi/blog
+US : http://www.ningboware.com
+	http://www.ningboware.com/mi/on-off-shower-hands-zh2262w.html
+JP : http://www.surfkayakaccessories.com
+	http://www.surfkayakaccessories.com/mi/
+US : http://www.chuamotor.com
+	http://www.chuamotor.com/mi/
+GB : http://luxurycastlehire.co.uk
+	http://luxurycastlehire.co.uk/mi/
+US : http://kratomonline.org
+	http://kratomonline.org/mi/maeng-da-kratom/
+US : http://www.bestwaytowhitenteethguide.org
+	http://www.bestwaytowhitenteethguide.org/mi/tag/vitamin-c/
+DE : http://catalog-moto.com
+	http://catalog-moto.com/mi/aprilia_2/aprilia-na-850-mana-and-honda-nc-700-s-dct.html
+ : http://freesport-tv.com
+	http://freesport-tv.com/mi/privacy-policy/
+US : https://www.jw.org
+	https://www.jw.org/mi/ng%C4%81-whakaputanga-whakahirahira/Convention-Releases/
+DE : https://malvorlagen-seite.de
+	https://malvorlagen-seite.de/mi/optische-illusion-drei-oder-vier/
+CN : https://www.abdindustrial.com
+	https://www.abdindustrial.com/mi/pump-air-compressor-portable-12v-60w-car-wet-dry-vacuum-cleaner-tire-inflator.html
+NL : https://www.bitbybitbook.com
+	https://www.bitbybitbook.com/mi/1st-ed/observing-behavior/strategies/
+US : http://www.liweimetal.com
+	http://www.liweimetal.com/mi/contact-us/
+CN : https://www.safesworld.com
+	https://www.safesworld.com/mi/
+US : http://www.whties.com
+	http://www.whties.com/mi/contact-us/
+CN : https://www.goldenlaser.cc
+	https://www.goldenlaser.cc/mi/laser-solutions/sublimation-sportswear/
+FR : http://donau-grundel-schiff.de
+	http://donau-grundel-schiff.de/mi/series/openplotter-dokumentation
+US : http://www.jointcontrols.net
+	http://www.jointcontrols.net/mi/jt701.html
+US : http://www.yixinhetrade.com
+	http://www.yixinhetrade.com/mi/about-us
+US : https://follow3rs.com
+	https://follow3rs.com/mi
+GB : https://www.highlands4you.co.uk
+	https://www.highlands4you.co.uk/mi/introduction/management.html
+US : http://www.ruifeng-leather.com
+	http://www.ruifeng-leather.com/mi/about-us/
+CN : http://www.cntiescarf.com
+	http://www.cntiescarf.com/mi/
+US : https://www.nfcchem.com
+	https://www.nfcchem.com/mi/calcium-2-hydroxy-4-methylpentanoate.html
+US : http://www.chinapipemills.com
+	http://www.chinapipemills.com/mi/
+US : https://www.kiwiproperty.com
+	https://www.kiwiproperty.com/westgate-lifestyle/mi/kia-moohio-noa/
+US : http://www.kingoneoffroad.com
+	http://www.kingoneoffroad.com/mi/news/
+US : http://www.steel-in-china.com
+	http://www.steel-in-china.com/mi/engineering-standard-activity-room-type-k-room.html
+CN : http://www.wigglewires.com
+	http://www.wigglewires.com/mi/contact-us/
+FR : http://www.gpedia.com
+	http://www.gpedia.com/mi/gpedia/Reo_M%C4%81ori
+CN : https://www.rykay.com
+	https://www.rykay.com/mi/
+US : http://www.jiejingfactory.com
+	http://www.jiejingfactory.com/mi/contact-us/
+US : https://www.magicstonegarden.com
+	https://www.magicstonegarden.com/mi/
+US : http://www.dayangfty.com
+	http://www.dayangfty.com/mi/download/
+US : https://www.kubbamachine.com
+	https://www.kubbamachine.com/mi/faqs/
+US : http://www.ncpcpharma.com
+	http://www.ncpcpharma.com/mi/contact-us-2/
+US : https://usahello.org
+	https://usahello.org/mi/resources/apply-for-asylum-in-us/
+US : http://www.steelprotectionpack.com
+	http://www.steelprotectionpack.com/mi/vci-stretch-film.html
+US : https://www.zhongxinlighting.com
+	https://www.zhongxinlighting.com/mi/
+US : http://www.shshenyong.com
+	http://www.shshenyong.com/mi/news/prewinch
+US : http://www.tubemillcn.com
+	http://www.tubemillcn.com/mi/end-facing-chamfering-machine.html
+US : http://www.lishin.cc
+	http://www.lishin.cc/mi/mop-refill-ls-2808-2.html
+US : http://www.hs-stationery.com
+	http://www.hs-stationery.com/mi/contact-us/
+CN : http://www.yulong-cellulose-cmc.com
+	http://www.yulong-cellulose-cmc.com/mi/honor/
+US : http://www.sunnymaycn.com
+	http://www.sunnymaycn.com/mi/
+US : https://companyhub.com
+	https://companyhub.com/mi/email-automation-software?ref=Exit_Bar
+CN : https://www.szradiant.com
+	https://www.szradiant.com/mi/support/faqs/
+US : http://www.sdzthl.com
+	http://www.sdzthl.com/mi/ppgl-prepainted-galvalume-aluzinc-steel-coils-as-per-jis3321az40-150.html
+US : http://www.gracebestbuy.com
+	http://www.gracebestbuy.com/mi/
+US : https://dwsolo.com
+	https://dwsolo.com/mi/
+US : https://estamerica.org
+	https://estamerica.org/mi/
+GB : https://www.centrallanguageschool.com
+	https://www.centrallanguageschool.com/mi/enrol/fees
+US : http://www.samewe.net
+	http://www.samewe.net/mi/
+US : http://www.wavesspring.com
+	http://www.wavesspring.com/mi/linear-wave-springs.html
+US : https://www.inpnurseryproducts.com
+	https://www.inpnurseryproducts.com/mi/news/high-quality-flower-sales
+US : http://www.shanghailangzhiweld.com
+	http://www.shanghailangzhiweld.com/mi/300a-plasma-cutting.html
+US : http://www.tantalum-caps.com
+	http://www.tantalum-caps.com/mi/product-tag/CL32B474KAFNNNE-datasheet-catalogue-price-samsung-electro-mechanics-authorized-distributor-smd-capacitor-mulilayer-ceramic-chip-capacitor/
+US : http://www.linphos.com
+	http://www.linphos.com/mi/soil-test-oem/
+HK : http://www.chaoyucabinet.com
+	http://www.chaoyucabinet.com/mi/news/network-cabinet
+UA : http://educationbro.com
+	http://educationbro.com/mi/universities/australia/australian-catholic-university/
+FR : https://www.slotsltd.com
+	https://www.slotsltd.com/mi/games/mobile-slots-mobile-billing-taco-brothers/
+US : https://www.anatod.com
+	https://www.anatod.com/mi/
+US : https://www.hello4x4.com
+	https://www.hello4x4.com/mi/alloy-genuine-beadlock-128d.html
+US : http://www.tjcylr.com
+	http://www.tjcylr.com/mi/
+US : https://www.tymexnetting.com
+	https://www.tymexnetting.com/mi/blue--shade-sail-maxwidth-m--new-hdpe-for-beachyardparking-area-35.html
+US : https://www.huadongmedical.com
+	https://www.huadongmedical.com/mi/about-us/company-video/
+US : https://www.junschem.com
+	https://www.junschem.com/mi/
+CN : https://www.pacopower.com
+	https://www.pacopower.com/mi/
+US : http://www.htwindsolarpower.com
+	http://www.htwindsolarpower.com/mi/
+CN : https://www.special-metal.com
+	https://www.special-metal.com/mi/about-us/
+CA : https://omg-solutions.com
+	https://omg-solutions.com/mi/emergency-panic-alarm/
+CN : http://www.eudemonbaby.com
+	http://www.eudemonbaby.com/mi/about-us/about-uscompany-profile
+US : http://www.qjqdvalve.com
+	http://www.qjqdvalve.com/mi/type-2252-servo-assisted-22way-diaphragm-valve.html
+US : http://www.fjxfxglass.com
+	http://www.fjxfxglass.com/mi/partners/
+US : http://www.pxbaisheng.com
+	http://www.pxbaisheng.com/mi/metallic-structured-packing.html
+CN : https://www.sjzhgw.com
+	https://www.sjzhgw.com/mi/
+CN : https://www.dm-compressor.com
+	https://www.dm-compressor.com/mi/semi-hermetic-workshop/
+US : http://www.livepro-beauty.com
+	http://www.livepro-beauty.com/mi/about-us/
+US : http://www.gmk-valve.com
+	http://www.gmk-valve.com/mi/basket-strainer.html
+US : http://www.gormeet.com
+	http://www.gormeet.com/mi/injection-molding-products/
+US : http://www.zhengmaoelec.com
+	http://www.zhengmaoelec.com/mi/about-us/
+US : https://carlhenryglobal.com
+	https://carlhenryglobal.com/mi/free-5-minute-sales-funnel/
+US : http://www.amcbox.com
+	http://www.amcbox.com/mi/mobile-shops.html
+US : http://www.wzdongyi.com
+	http://www.wzdongyi.com/mi/
+US : http://www.qitai-adhesive.com
+	http://www.qitai-adhesive.com/mi/thread-locking-glue/instant-glue/loctite-414/
+US : http://www.richina-tools.com
+	http://www.richina-tools.com/mi/
+US : https://www.samsungwiremesh.com
+	https://www.samsungwiremesh.com/mi/
+US : http://www.jhc-nonwoven.com
+	http://www.jhc-nonwoven.com/mi/faqs/
+US : http://www.linbaymachinery.com
+	http://www.linbaymachinery.com/mi/news/arabia-corrugated-roll-forming-machine-and-curving-machine
+US : http://www.chinagxmy.com
+	http://www.chinagxmy.com/mi/single-bagged-black-garlic.html
+US : http://www.ldsolarpv.com
+	http://www.ldsolarpv.com/mi/ld2410c-10a-pwm-solar-controller.html
+ : https://www.elitedeluxe.com.tr
+	https://www.elitedeluxe.com.tr/mi/
+US : http://www.ainuogas.com
+	http://www.ainuogas.com/mi/
+CN : https://www.nbjiatong.com
+	https://www.nbjiatong.com/mi/
+US : https://www.bestpvcfence.com
+	https://www.bestpvcfence.com/mi/
+CN : https://www.simphoenix.com
+	https://www.simphoenix.com/mi/download/manual/
+US : http://www.chinacombinerbox.com
+	http://www.chinacombinerbox.com/mi/four-in-one-rib-out-of-the-exchange-of-ac-convergence-integrated-components-amj2-4003h1003hx4.html
+NL : https://nonlinearsystem.eu
+	https://nonlinearsystem.eu/mi/
+US : http://www.ladybagcn.com
+	http://www.ladybagcn.com/mi/hot-sale-2018-newly-designed-6pcs-set-shoulder-messenger-bag-backpack-leather-lady-handbag.html
+CN : https://www.study-mandarin.com
+	https://www.study-mandarin.com/mi/news/
+CN : http://www.koowheel.com
+	http://www.koowheel.com/mi/koowheel-patent-8-5inch-big-wheels-bluetooth-hoverboard-off-road.html
+US : http://www.szebo.com
+	http://www.szebo.com/mi/download
+US : https://www.glorystarlaser.com
+	https://www.glorystarlaser.com/mi/fiber-laser-marking-machine.html
+RU : http://www.swiss-apteka.com
+	http://www.swiss-apteka.com/mi/information/voprosy-i-otvety/
+CN : https://www.szhtpmart.com
+	https://www.szhtpmart.com/mi/
+US : https://jobdescriptionsample.org
+	https://jobdescriptionsample.org/mi/animal-control-workers-job-description-obligation-template-and-functions/1092
+US : http://www.allutertech.com
+	http://www.allutertech.com/mi/contact-us/
+US : https://www.csmaterialstech.com
+	https://www.csmaterialstech.com/mi/
+US : http://www.mksmartcard.com
+	http://www.mksmartcard.com/mi/factory-tour/
+US : https://www.tianjia-lock.com
+	https://www.tianjia-lock.com/mi/
+US : https://www.aquagem.com.cn
+	https://www.aquagem.com.cn/mi/isaver-patent-design-frequency-inverter.html
+CA : https://www.wikiplanet.click
+	https://www.wikiplanet.click/enciclopedia/mi/Tiamana
+US : http://www.envicool.net
+	http://www.envicool.net/mi/news_catalog/telecom-site-cooling/
+US : http://www.cnrgxy.com
+	http://www.cnrgxy.com/mi/
+US : https://maxspeedtest.com
+	https://maxspeedtest.com/mi/
+CN : https://www.jy-glass.com
+	https://www.jy-glass.com/mi/
+US : http://www.chinaocan.com
+	http://www.chinaocan.com/mi/customized-color-rigid-pvc-sheet-0-2-6mm-thickness.html
+CN : https://www.imposalight.com
+	https://www.imposalight.com/mi/users-manual-for-chainzones-led-high-bay-eos-3030-a6.html
+US : https://www.egbadges.com
+	https://www.egbadges.com/mi/news/talking-about-the-role-of-the-base-for-the-embroidery-badge
+US : https://www.the-tailoress.com
+	https://www.the-tailoress.com/mi/home/
+US : http://www.xida-electronics.com
+	http://www.xida-electronics.com/mi/products
+CN : https://www.aoxinhvacr.com
+	https://www.aoxinhvacr.com/mi/brand/daming/y-4cylinders
+CN : https://www.medke.com
+	https://www.medke.com/mi/about-us/
+US : http://www.albertnovosino.com
+	http://www.albertnovosino.com/mi/
+US : http://www.ksdoing.com
+	http://www.ksdoing.com/mi/factory-tour/
+US : https://www.fxcc.com
+	https://www.fxcc.com/mi/currencies-forecast-poll
+CN : https://www.wecare-life.com
+	https://www.wecare-life.com/mi/
+CN : https://www.ngyc.com
+	https://www.ngyc.com/mi/quality-system/
+US : http://www.btmeac.com
+	http://www.btmeac.com/mi/contact-us/
+CN : https://www.nbulboy.com
+	https://www.nbulboy.com/mi/
+US : http://www.jbpcba.com.cn
+	http://www.jbpcba.com.cn/mi/contact-us/
+CN : https://www.slagremoving.com
+	https://www.slagremoving.com/mi/
+US : http://www.sog-pump.com
+	http://www.sog-pump.com/mi/chl-horizontal-multistage-stainless-steel-centrifugal-pump.html
+CN : https://www.charmingmetal.com
+	https://www.charmingmetal.com/mi/
+US : http://www.tjcywires.com
+	http://www.tjcywires.com/mi/
+US : http://www.luluae.com
+	http://www.luluae.com/mi/enterprise-album/
+US : http://www.sxceramic.com
+	http://www.sxceramic.com/mi/
+US : https://www.aedisduct.com
+	https://www.aedisduct.com/mi/
+CN : https://www.ez-walk.com
+	https://www.ez-walk.com/mi/
+US : http://www.lanlinprintech.com
+	http://www.lanlinprintech.com/mi/services/technical-support/
+US : https://www.njkeyuda.com
+	https://www.njkeyuda.com/mi/kyd-crawling-mat-3.html
+US : https://www.drickinstruments.com
+	https://www.drickinstruments.com/mi/
+US : http://www.nicehut-window.com
+	http://www.nicehut-window.com/mi/company-profile/business-performance
+US : https://www.prostepper.com
+	https://www.prostepper.com/mi/
+US : https://worldstarhiphop.roseconverter.com
+	https://worldstarhiphop.roseconverter.com/mi/
+US : https://www.tjshenzhoutong.com
+	https://www.tjshenzhoutong.com/mi/
+US : https://www.judinwire.net
+	https://www.judinwire.net/mi/
+CN : https://www.chinarfidcard.com
+	https://www.chinarfidcard.com/mi/contact-us/
+US : https://www.hongxiangmould.com
+	https://www.hongxiangmould.com/mi/mazda-clutch-housing.html
+DE : http://www.almancax.com
+	http://www.almancax.com/mi/
+US : http://www.gfh-electric.com
+	http://www.gfh-electric.com/mi/
+US : http://www.huamachinery.com
+	http://www.huamachinery.com/mi/news/before-operate-to-the-mini-round-baler-you-must-to-do-what%EF%BC%9F%EF%BC%9F
+US : https://www.artiegarden.com
+	https://www.artiegarden.com/mi/
+US : http://www.shengrunqiche.com
+	http://www.shengrunqiche.com/mi/video/cement-tank-trailer/
+US : https://cycletraderpro.com
+	https://cycletraderpro.com/mi/
+US : http://www.homey-tec.com
+	http://www.homey-tec.com/mi/
+CN : https://www.3drambery.com
+	https://www.3drambery.com/mi/news/know-the-pla-material-of-3d-printed-materials
+CN : http://www.focuslasersystems.com
+	http://www.focuslasersystems.com/mi/videos/
+US : https://www.rikoooo.com
+	https://www.rikoooo.com/mi/downloads/category/7
+US : http://www.forever-moving.com
+	http://www.forever-moving.com/mi/contact-us/
+NL : https://www.emergency-live.com
+	https://www.emergency-live.com/mi/
+CN : https://www.desunpump.com
+	https://www.desunpump.com/mi/
+CN : https://www.pakite.com
+	https://www.pakite.com/mi/
+US : https://www.acenlmn.com
+	https://www.acenlmn.com/mi/
+US : http://www.cnyaonan.com
+	http://www.cnyaonan.com/mi/faqs/
+US : https://www.weld-automation.com
+	https://www.weld-automation.com/mi/news/italy-order-for-3-tons-3-axis-hydraulic-welding-positioners
+CN : http://www.coneleqd.com
+	http://www.coneleqd.com/mi/boom-pump-truck-howo-44m.html
+US : https://www.nbwinwinea.com
+	https://www.nbwinwinea.com/mi/air-conditioning-remote-control.html
+CN : http://www.medicohongkong.com
+	http://www.medicohongkong.com/mi/
+ : https://www.sokrh.com
+	https://www.sokrh.com/mi/
+US : https://vk.roseconverter.com
+	https://vk.roseconverter.com/mi/
+US : http://www.zhenchengscrew.com
+	http://www.zhenchengscrew.com/mi/philip-flat-head-self-tapping-and-self-drilling-screw-2.html
+US : http://www.hqftex.com
+	http://www.hqftex.com/mi/about-us/
+US : http://www.hebeipackaging.com
+	http://www.hebeipackaging.com/mi/about-us/
+US : http://www.wf-fastener.com
+	http://www.wf-fastener.com/mi/
+US : http://www.b-packaging.com
+	http://www.b-packaging.com/mi/
+US : https://twitter.roseconverter.com
+	https://twitter.roseconverter.com/mi/
+US : http://www.armati.biz
+	http://www.armati.biz/mi/luxury-bathroom-faucet-manufacturers-high-end-bath-and-plumbing-supplies.html
+US : http://www.qbd-group.com
+	http://www.qbd-group.com/mi/
+CN : http://www.likvchina.com
+	http://www.likvchina.com/mi/news/copy-the-absolute-beginners-guide-to-google-analytics
+US : http://www.ntvigourbrush.com
+	http://www.ntvigourbrush.com/mi/
+US : https://therefugeecenter.org
+	https://therefugeecenter.org/mi/resources/citizenship-and-rights/
+US : https://www.liveolight.com
+	https://www.liveolight.com/mi/
+US : http://www.czzhit.com
+	http://www.czzhit.com/mi/about-us/service/
+US : https://kratomonline.org
+	https://kratomonline.org/mi/category/kratom/
+US : http://www.jeftrade.com
+	http://www.jeftrade.com/mi/
+US : http://www.viairdoormat.com
+	http://www.viairdoormat.com/mi/
+US : http://www.chinatopcnc.com
+	http://www.chinatopcnc.com/mi/
+US : https://www.fivestar-wire.com
+	https://www.fivestar-wire.com/mi/
+US : http://www.conele-mixer.com
+	http://www.conele-mixer.com/mi/case/
+US : http://www.joyseaplywood.com
+	http://www.joyseaplywood.com/mi/about-us/
+PA : http://www.joyo-filter.com
+	http://www.joyo-filter.com/mi/about-us/
+US : http://www.vigor-industry.com
+	http://www.vigor-industry.com/mi/
+US : http://www.ruk-tech.com
+	http://www.ruk-tech.com/mi/news/25th-shanghai-apppexpo
+US : http://www.cnxh-electric.com
+	http://www.cnxh-electric.com/mi/cnc1-r-series-special-forwater-boiler-1232r.html
+US : http://www.bdknitting.com
+	http://www.bdknitting.com/mi/
+CN : https://www.cntfsolar.com
+	https://www.cntfsolar.com/mi/
+US : https://templatica.net
+	https://templatica.net/mi/
+US : https://www.chinawelken.com
+	https://www.chinawelken.com/mi/
+US : https://facebook.roseconverter.com
+	https://facebook.roseconverter.com/mi/
+US : http://www.twtvalvecn.com
+	http://www.twtvalvecn.com/mi/about-us/achievement/
+US : http://www.weddingfurniture.com
+	http://www.weddingfurniture.com/mi/
+US : http://www.arjextrailerparts.com
+	http://www.arjextrailerparts.com/mi/about-us/
+US : http://www.cz-juteng.com
+	http://www.cz-juteng.com/mi/
+US : http://www.soontruepackaging.com
+	http://www.soontruepackaging.com/mi/yl400-vertical-packing-machine.html
+US : https://www.hengweihoseclamp.com
+	https://www.hengweihoseclamp.com/mi/air-condition-hose-clamps.html
+FR : https://www.88c.co.uk
+	https://www.88c.co.uk/mi/producttag/88c-co-uk/
+US : http://www.shenhe-bearing.com
+	http://www.shenhe-bearing.com/mi/about-us/
+NL : https://www.spectrumschool.be
+	https://www.spectrumschool.be/mi/page/2/
+US : http://www.xr-ovenlamp.com
+	http://www.xr-ovenlamp.com/mi/company-profile/
+US : http://www.risepipe.com
+	http://www.risepipe.com/mi/
+CN : http://www.abdindustrial.com
+	http://www.abdindustrial.com/mi/d006-triangle-reflector.html
+DE : https://www.saper-link-news.com
+	https://www.saper-link-news.com/mi/
+US : https://www.yuriebsf.com
+	https://www.yuriebsf.com/mi/
+US : http://www.wrdtubemill.com
+	http://www.wrdtubemill.com/mi/contact-us/
+US : http://www.nbyobo.com
+	http://www.nbyobo.com/mi/car-seat-rear-view-mirror-yobo-510.html
+US : http://www.xzc9.com
+	http://www.xzc9.com/mi/news/shandong-xingzhicheng-create-brand-new-market-model
+US : http://www.mao-shuo.com
+	http://www.mao-shuo.com/mi/about-us/
+US : https://techadminlab.com
+	https://techadminlab.com/mi/
+US : http://www.qdmiracleshelf.com
+	http://www.qdmiracleshelf.com/mi/many-patterns-hand-trolley-solid-rubber-powder-wheel.html
+US : http://www.pvcroofingtile.com
+	http://www.pvcroofingtile.com/mi/high-temperatuercorrosion-resistance-roof-sheet.html
+US : https://www.sakysteel.com
+	https://www.sakysteel.com/mi/
+CN : http://www.longtopmining.com
+	http://www.longtopmining.com/mi/about-us/
+US : http://www.swedishcurrency.net
+	http://www.swedishcurrency.net/mi/canadian-currency-exchange/choosing-the-right-canadian-currency-exchange-broker/
+DE : https://herocity.de
+	https://herocity.de/mi/
+AU : https://www.vincenzosplate.com
+	https://www.vincenzosplate.com/mi/
+US : https://ispyoo.com
+	https://ispyoo.com/mi/
+US : https://snaggys.com
+	https://snaggys.com/mi/
+CN : https://www.zengrit.com
+	https://www.zengrit.com/mi/
+US : http://www.suoxuehuwai.com
+	http://www.suoxuehuwai.com/mi/contact-us/
+US : http://www.huaxinfurnace.com
+	http://www.huaxinfurnace.com/mi/
+US : http://www.armost.cn
+	http://www.armost.cn/mi/
+US : http://www.hv-caps.com
+	http://www.hv-caps.com/mi/
+US : http://www.jindunlaobao.com
+	http://www.jindunlaobao.com/mi/
+US : http://www.jiajiebathmirror.com
+	http://www.jiajiebathmirror.com/mi/about-us/
+US : http://www.gtpcba.com
+	http://www.gtpcba.com/mi/products-services/
+US : https://www.fxpremiere.com
+	https://www.fxpremiere.com/mi/
+US : http://www.mtpak.com
+	http://www.mtpak.com/mi/
+US : http://www.unibest.cc
+	http://www.unibest.cc/mi/news/fda-approves-vosevi-for-hepatitis-c
+US : https://www.king-pcb.com
+	https://www.king-pcb.com/mi/
+CN : http://www.prius-automatic.com
+	http://www.prius-automatic.com/mi/
+CN : https://www.foocles.com
+	https://www.foocles.com/mi/
+US : http://www.chinacarbonfibre.com
+	http://www.chinacarbonfibre.com/mi/news/
+US : http://www.hengshengcold.com
+	http://www.hengshengcold.com/mi/
+US : http://www.wellformpacking.com
+	http://www.wellformpacking.com/mi/
+US : http://www.longxin-global.com
+	http://www.longxin-global.com/mi/application/
+US : https://www.yogemcasting.com
+	https://www.yogemcasting.com/mi/
+US : http://www.sindadisplay.com
+	http://www.sindadisplay.com/mi/
+US : http://www.homewin88.com
+	http://www.homewin88.com/mi/
+US : http://www.fanhaopets.com
+	http://www.fanhaopets.com/mi/
+EU : https://www.the-good-stuff-factory.be
+	https://www.the-good-stuff-factory.be/mi/
+CN : https://www.china-hdpepipe.com
+	https://www.china-hdpepipe.com/mi/hdpe-dredging-pipe.html
+US : https://www.hushhush.com
+	https://www.hushhush.com/mi/
+US : http://portal.smart-project.info
+	http://portal.smart-project.info/smart/lang/mi
+US : http://www.lsseals.com
+	http://www.lsseals.com/mi/
+US : http://www.silicone-odm.com
+	http://www.silicone-odm.com/mi/
+US : http://www.csmaterialstech.com
+	http://www.csmaterialstech.com/mi/cornerstone-cs110.html
+US : http://www.iceboxchina.com
+	http://www.iceboxchina.com/mi/faqs/
+US : http://www.pamaens.com
+	http://www.pamaens.com/mi/about-us/
+US : http://www.sdxhhd.com
+	http://www.sdxhhd.com/mi/about-us/
+US : http://www.hanortex.com
+	http://www.hanortex.com/mi/about-us/
+US : http://www.czldfloor.com
+	http://www.czldfloor.com/mi/about-us/
+US : http://www.evergrowingcage.com
+	http://www.evergrowingcage.com/mi/news/looks-at-prospects-for-fisheries-and-aquaculture
+US : http://www.julongjewelry.cn
+	http://www.julongjewelry.cn/mi/contact-us/
+US : http://www.weldpipemill.com
+	http://www.weldpipemill.com/mi/ends-beveling-machine.html
+US : http://www.brushcutterjusen.com
+	http://www.brushcutterjusen.com/mi/news/business-department-jiufeng-hill-play
+US : http://www.china-brewhouse.com
+	http://www.china-brewhouse.com/mi/about-us/
+US : http://www.shengxinsport.com
+	http://www.shengxinsport.com/mi/news_catalog/industry-news/
+US : http://www.americasportsfloor.com
+	http://www.americasportsfloor.com/mi/
+US : http://www.ycautoc.com
+	http://www.ycautoc.com/mi/professional-spirit/
+CN : http://www.wenwencf.com
+	http://www.wenwencf.com/mi/about-us/company/
+US : https://www.jxsten.com
+	https://www.jxsten.com/mi/faqs/
+US : http://www.hlinsectscreen.com
+	http://www.hlinsectscreen.com/mi/news/polyester-pleated-mesh-for-plisse
+US : http://www.gxaluminium.com
+	http://www.gxaluminium.com/mi/about-us/
+US : http://www.fancyco.com
+	http://www.fancyco.com/mi/
+US : http://www.focusway-casting.com
+	http://www.focusway-casting.com/mi/convenient-one-stop-service/
+US : http://www.inpnurseryproducts.com
+	http://www.inpnurseryproducts.com/mi/news/fertilizer-for-herb-gardens
+US : http://www.yrkseal.com
+	http://www.yrkseal.com/mi/about-us/
+US : http://www.honglu-mining.com
+	http://www.honglu-mining.com/mi/single-row-vehicle-brake-flag.html
+US : https://www.webhostingsecretrevealed.net
+	https://www.webhostingsecretrevealed.net/mi/
+US : http://www.zypackag.com
+	http://www.zypackag.com/mi/about-us/
+US : http://www.tangres100.com
+	http://www.tangres100.com/mi/contact-us/
+US : http://www.bmaxmachine.com
+	http://www.bmaxmachine.com/mi/
+US : http://www.wellfit-sportswear.com
+	http://www.wellfit-sportswear.com/mi/
+US : http://www.tailikids.com
+	http://www.tailikids.com/mi/why-taili-kids/
+US : http://www.ictctruss.com
+	http://www.ictctruss.com/mi/news/ictc-aluminum-truss-create-chen-chusheng-s-concert-in-beijing
+US : http://www.kd-physicalrehab.com
+	http://www.kd-physicalrehab.com/mi/
+CN : http://www.cnchangfa.cn
+	http://www.cnchangfa.cn/mi/our-products/copperaluminum-conponents/
+US : http://www.fxctool.com
+	http://www.fxctool.com/mi/contact-us/
+US : http://www.bluekin.com
+	http://www.bluekin.com/mi/
+CN : http://www.3drambery.com
+	http://www.3drambery.com/mi/news/is-3d-printed-material-the-soul-take-you-to-know-the-hottest-3d-printing-materials
+US : http://www.vango-tech.com
+	http://www.vango-tech.com/mi/vmt-series-mechanical-thermostat.html
+US : http://www.wosaicabinet.com
+	http://www.wosaicabinet.com/mi/custom-made-mode.html
+US : http://www.artmetalcn.com
+	http://www.artmetalcn.com/mi/faqs/
+US : http://www.gxyfoods.com
+	http://www.gxyfoods.com/mi/snow-white-pumpkin-seeds.html
+US : http://www.supplyfurniture.com
+	http://www.supplyfurniture.com/mi/stainless-steel-chair.html
+US : http://www.sps-squeegee.com
+	http://www.sps-squeegee.com/mi/about-us/
+US : http://www.sunshinebelt.com
+	http://www.sunshinebelt.com/mi/factory-tour/
+US : https://blockchains.io
+	https://blockchains.io/mi/
+US : https://www.niceterminal.com
+	https://www.niceterminal.com/mi/customer-case/
+US : http://www.dmdryer.com
+	http://www.dmdryer.com/mi/about-us/
+DE : https://universaldocumentation.com
+	https://universaldocumentation.com/mi/
+CN : http://www.estarspareparts.com
+	http://www.estarspareparts.com/mi/cfb09-series-bronze-rolling-bearings-12.html
+US : http://www.qymachines.com
+	http://www.qymachines.com/mi/
+US : http://www.hbxg-bulldozer.com
+	http://www.hbxg-bulldozer.com/mi/service/
+US : http://www.strongsaw.com
+	http://www.strongsaw.com/mi/
+CN : http://www.comfortebicycle.com
+	http://www.comfortebicycle.com/mi/news/
+US : http://www.guliindustry.com
+	http://www.guliindustry.com/mi/
+CN : http://www.ankaicnc.com
+	http://www.ankaicnc.com/mi/about-us/
+US : http://www.wanmaroto.com
+	http://www.wanmaroto.com/mi/45l-white-picnic-ice-box.html
+US : http://www.rosin-kings.com
+	http://www.rosin-kings.com/mi/
+US : http://www.philibuilding.com
+	http://www.philibuilding.com/mi/interior-wall-lining-project/
+US : http://www.nide-industry.com
+	http://www.nide-industry.com/mi/about-us/culture/
+DE : https://afrikhepri.org
+	https://afrikhepri.org/mi/
+US : http://www.toption-ingredients.com
+	http://www.toption-ingredients.com/mi/ascorbic-acid-vitamin-c.html
+US : https://www.water-filterhousing.com
+	https://www.water-filterhousing.com/mi/
+CN : http://www.refinehotelsupply.com
+	http://www.refinehotelsupply.com/mi/download/
+CN : http://www.kavounautoparts.com
+	http://www.kavounautoparts.com/mi/raw-edge-v-belt.html
+US : http://www.prostepper.com
+	http://www.prostepper.com/mi/
+US : http://www.slidingdoorchina.com
+	http://www.slidingdoorchina.com/mi/
+US : https://lingeriefc.com
+	https://lingeriefc.com/mi/
+CN : http://www.czhengfa.com
+	http://www.czhengfa.com/mi/
+US : http://www.cnfeinade.com
+	http://www.cnfeinade.com/mi/current-monitoring-relay-jfy-701.html
+US : https://www.hjfoodmachinery.com
+	https://www.hjfoodmachinery.com/mi/desktop-automatic-small-kubba-kibbe-encrusting-machine-for-home.html
+CN : https://www.painting-machine.com
+	https://www.painting-machine.com/mi/
+CN : https://www.km-medicine.com
+	https://www.km-medicine.com/mi/about-us/
+GB : https://www.solasolv.com
+	https://www.solasolv.com/mi/
+CN : http://www.hermdecosteel.com
+	http://www.hermdecosteel.com/mi/about-us/
+US : https://mamaclub.info
+	https://mamaclub.info/mi/
+US : https://www.et-magnet.com
+	https://www.et-magnet.com/mi/
+VG : https://best-immunotherapy.com
+	https://best-immunotherapy.com/mi/
+CN : https://www.diamante-tech.com
+	https://www.diamante-tech.com/mi/
+US : http://www.cdebyte.net
+	http://www.cdebyte.net/mi/e43%EF%BC%88433t13s%EF%BC%89.html
+US : https://bit-calculator.com
+	https://bit-calculator.com/mi/
+US : http://www.tkfanen.com
+	http://www.tkfanen.com/mi/
+US : https://www.jtaluminumcircle.com
+	https://www.jtaluminumcircle.com/mi/
+US : http://www.wpcline.com
+	http://www.wpcline.com/mi/
+US : https://drugsinc.eu
+	https://drugsinc.eu/mi/
+CN : https://www.jsbotanics.com
+	https://www.jsbotanics.com/mi/
+CN : https://www.qlart.com
+	https://www.qlart.com/mi/
+US : http://www.carlhenryonlinesuccess.com
+	http://www.carlhenryonlinesuccess.com/mi/
+US : http://www.houshenshoes.com
+	http://www.houshenshoes.com/mi/
+CN : https://www.aerial-display.com
+	https://www.aerial-display.com/mi/
+US : http://carlhenryonlinesuccess.com
+	http://carlhenryonlinesuccess.com/mi/
+FR : https://www.expresscasino.com
+	https://www.expresscasino.com/mi/k%C4%93mu/online-slots-real-money-no-deposit-pixie-gold/
+CN : https://www.coffbrewing.com
+	https://www.coffbrewing.com/mi/
+IE : http://netkiosk.co.uk
+	http://netkiosk.co.uk/mi/
+NL : http://www.cbdolievoordelen.nl
+	http://www.cbdolievoordelen.nl/mi/aratohu-M%C3%A4kaurau/
+US : https://pixelhelper.org
+	https://pixelhelper.org/mi/tag/ruestung/
+US : https://church-of-christ.org
+	https://church-of-christ.org/mi/cb-forgot-login.html
+CN : https://www.bailixin.com
+	https://www.bailixin.com/mi/
+US : https://www.jlextract.com
+	https://www.jlextract.com/mi/
+CN : http://www.jetwayamenities.com
+	http://www.jetwayamenities.com/mi/eye-mask-002.html
+CN : https://www.goldnard.com
+	https://www.goldnard.com/mi/
+CN : https://www.c-superun.com
+	https://www.c-superun.com/mi/
+CN : https://www.sitzonechair.com
+	https://www.sitzonechair.com/mi/
+US : https://biblia.gospelprime.com.br
+	https://biblia.gospelprime.com.br/mi/deuteronomio/4.44/28.68/
+CN : https://www.sehenda-en.com
+	https://www.sehenda-en.com/mi/
+US : http://loginmail.online
+	http://loginmail.online/mi/skynet-mail-login/
+US : https://www.church-of-christ.org
+	https://www.church-of-christ.org/mi/pulpit-preview.html
+CN : https://www.n2o2gas.com
+	https://www.n2o2gas.com/mi/
+CN : https://www.fibereye2.com
+	https://www.fibereye2.com/mi/
+CA : https://dehaut.com
+	https://dehaut.com/mi/
+US : https://www.judipak.com
+	https://www.judipak.com/mi/privacy-policy/
+US : http://www.seasum.cn
+	http://www.seasum.cn/mi/news_catalog/news/
+US : https://www.chinapresses.com
+	https://www.chinapresses.com/mi/about-us/honor/
+CN : https://www.qdruidetai.com
+	https://www.qdruidetai.com/mi/chain-accessories/
+CN : https://www.jsjlmachinery.com
+	https://www.jsjlmachinery.com/mi/complete-equipment/interior-door-equipment/
+CN : https://www.fc-med.com
+	https://www.fc-med.com/mi/
+CN : https://www.insharevape.com
+	https://www.insharevape.com/mi/
+US : https://www.acouplefortheroad.com
+	https://www.acouplefortheroad.com/mi/
+CA : https://www.chinanbdb.com
+	https://www.chinanbdb.com/mi/
+US : https://www.axnewdisplay.com
+	https://www.axnewdisplay.com/mi/
+CN : https://www.sino-masterbatch.com
+	https://www.sino-masterbatch.com/mi/
+CA : https://www.dghk-buffer.com
+	https://www.dghk-buffer.com/mi/
+CN : https://www.rswires.com
+	https://www.rswires.com/mi/
+CN : https://english.taiergroup.com
+	https://english.taiergroup.com/mi/
+CN : https://www.jinhejinshu.com
+	https://www.jinhejinshu.com/mi/
+CN : https://www.lasonparts.com
+	https://www.lasonparts.com/mi/
+US : http://www.touchdisplays-tech.com
+	http://www.touchdisplays-tech.com/mi/solutions/interactive-digital-signage/
+US : https://www.jewellrylove.com
+	https://www.jewellrylove.com/mi/
+US : http://www.restart-industry.com
+	http://www.restart-industry.com/mi/contact-us/
+US : http://www.interryfrp.com
+	http://www.interryfrp.com/mi/news/interry-double-layer-oil-tank-fill-in-gaps-built-the-nationwide-largest-sf-double-layer-oil-tank-base
+CN : https://www.tjtgsteel.com
+	https://www.tjtgsteel.com/mi/
+US : https://2fish.co
+	https://2fish.co/mi/bible/old-testament/isaiah/
+CN : https://www.capacitorfilmhaoda.com
+	https://www.capacitorfilmhaoda.com/mi/
+DK : https://www.sugarlove.one
+	https://www.sugarlove.one/mi/
+US : http://www.accotech.net
+	http://www.accotech.net/mi/magic-music-and-story-player-with-learning-tools-green.html
+US : http://www-hotmail-com.email
+	http://www-hotmail-com.email/mi/facebook-login-facebook-sign-in/
+US : https://www.hobbycarbon.com
+	https://www.hobbycarbon.com/mi/sitemap.html
+CN : https://www.yangrutingtrade.com
+	https://www.yangrutingtrade.com/mi/yrt-t10-medical-stainless-steel-two-layers-hospital-treatment-trolley.html
+US : https://www.inductorchina.com
+	https://www.inductorchina.com/mi/
+CN : https://www.nbhengchen.com
+	https://www.nbhengchen.com/mi/
+CN : https://www.micropreparedslides.com
+	https://www.micropreparedslides.com/mi/
+CN : https://www.changjia-machinery.com
+	https://www.changjia-machinery.com/mi/
+US : https://policies.oclc.org
+	https://policies.oclc.org/mi/privacy/privacy-statement.html
+US : https://www.szzhsbag.com
+	https://www.szzhsbag.com/mi/
+US : https://www.peptidejymed.com
+	https://www.peptidejymed.com/mi/
+CN : https://www.bestardoors.com
+	https://www.bestardoors.com/mi/nylon-handle.html
+HK : https://www.dktestp.com
+	https://www.dktestp.com/mi/
+US : https://www.hytokstech.com
+	https://www.hytokstech.com/mi/
+US : http://www.qxmic.com
+	http://www.qxmic.com/mi/1-14150-lbs-malleable-cast-iron-pipe-fitting-thread-flange.html
+CN : https://www.haitungchem.com
+	https://www.haitungchem.com/mi/
+CN : https://www.band-ss.com
+	https://www.band-ss.com/mi/
+CN : https://www.grandstarcn.com
+	https://www.grandstarcn.com/mi/
+US : https://www.cz-juteng.com
+	https://www.cz-juteng.com/mi/factory-view/
+US : https://www.worldquitsmoking.com
+	https://www.worldquitsmoking.com/mi/
+CN : https://www.gaofeng-petro.com
+	https://www.gaofeng-petro.com/mi/
+US : http://www.nyforgedwheels.com
+	http://www.nyforgedwheels.com/mi/contact-us/
+CN : http://www.nbwellrun.com
+	http://www.nbwellrun.com/mi/
+US : https://guidebooq.com
+	https://guidebooq.com/mi/
+US : http://www.niceterminal.com
+	http://www.niceterminal.com/mi/
+US : http://www.chinahyperauto.com
+	http://www.chinahyperauto.com/mi/
+US : http://www.weld-automation.com
+	http://www.weld-automation.com/mi/3000kg-3-axis-hydraulic-positioner.html
+US : http://www.kehengmixing.com
+	http://www.kehengmixing.com/mi/
+VG : http://www.overlandtent.com
+	http://www.overlandtent.com/mi/
+US : http://www.hzhinew.com
+	http://www.hzhinew.com/mi/contact-us/
+ES : https://www.torresbus.es
+	https://www.torresbus.es/mi/
+CN : https://www.radiant-lighting.com
+	https://www.radiant-lighting.com/mi/
+US : http://www.nbwegro.com
+	http://www.nbwegro.com/mi/
+US : http://www.nicerelay.com
+	http://www.nicerelay.com/mi/
+FR : https://www.topslotsite.com
+	https://www.topslotsite.com/mi/play/
+US : http://indigenoustweets.com
+	http://indigenoustweets.com/blogs/mi/
+ZA : http://www.szsinotec.com
+	http://www.szsinotec.com/mi/
+US : https://mega-winning.com
+	https://mega-winning.com/mi/
+HK : https://www.mouldpet.com
+	https://www.mouldpet.com/mi/
+US : http://www.keyluo.com
+	http://www.keyluo.com/mi/
+US : http://www.longda-inc.com
+	http://www.longda-inc.com/mi/
+US : https://www.waterproof-factory.com
+	https://www.waterproof-factory.com/mi/
+VG : http://www.hacksdirect.com
+	http://www.hacksdirect.com/mi/how-to-get-free-cash-on-8-ball-pool/
+US : https://idmcrackdownload.net
+	https://idmcrackdownload.net/mi/
+ : http://libertex.win
+	http://libertex.win/mi/
Index: /other-projects/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt
===================================================================
--- /other-projects/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt	(revision 33635)
@@ -0,0 +1,600 @@
+# Mapping of top sites in base url forms to value
+
+# This file contains sites that are too large to crawl exhaustively.
+# The domains are from Alexa top sites (where only the first 50 were visible)
+# Added further top sites from https://en.wikipedia.org/wiki/List_of_most_popular_websites
+# Finally also added https://moz.com/top500 by downloading its CSV file and
+# adding its URLs to the existing listing here from alexa/wiki.
+# Then used LibreOffice's Calc spreadsheet software to sort alphabetically and remove duplicates.
+# Then in Gedit, used regex search and replace to remove <subdomain>.<site>.ext variants, keeping
+# just <site>.ext
+# And finally, re-sorted the reduced list alphabetically and pasted into here.
+
+# FORMAT OF THIS FILE'S CONTENTS:
+#    <topsite-base-url>,<value>
+# where <value> can or is one of
+#    empty, SUBDOMAIN-COPY, FOLLOW-LINKS-WITHIN-TOPSITE, SINGLEPAGE, <url-form-without-protocol>
+#
+#   - if value is left empty: if seedurl contains topsite-base-url, the seedurl will go into the
+#     file unprocessed-topsite-matches.txt and the site/page won't be crawled.
+#     The user will be notified to inspect the file unprocessed-topsite-matches.txt.
+#   - SINGLEPAGE: if seedurl matches topsite-base-url, then only download the page at that seedurl.
+#     For example, if the seedurl is http://docs.google.com/some-long-suffix-in-base64, then it
+#     matches the topsite-base-url of docs.google.com and its value of SINGLEPAGE will add the
+#     seedurl itself as the regex url-filter, to restrict the crawl to just the specified page.
+#   - SUBDOMAIN-COPY: if seedurl CONTAINS topsite-base-url, then whatever the seedurl's subdomain
+#     or else domain is, will make up the urlfilter, so we don't leak out into a larger domain.
+#     Use SUBDOMAIN-COPY to restrict to a domain prefix/subdomain. For example, if seedurl is
+#     pinky.blogspot.com, it will match the topsite-base-url of blogspot.com, but SUBDOMAIN-COPY
+#     will ensure we restrict crawling to pages on pinky.blogspot.com.
+#     However, if the seedurl's domain is an exact match on topsite-base-url, the seedurl will go
+#     into the file unprocessed-topsite-matches.txt and the site/page won't be crawled.
+#   - FOLLOW-LINKS-WITHIN-TOPSITE: if pages linked from the seedURL page can be followed and
+#     downloaded, as long as it's within the same subdomain matching the topsite-base-url.
+#     This is different from SUBDOMAIN-COPY, as that can download all of a specific subdomain but
+#     restricts against downloading the entire domain (e.g. all pinky.blogspot.com and not anything
+#     else within blogspot.com). FOLLOW-LINKS-WITHIN-TOPSITE can download all linked pages (at
+#     depth specified for the nutch crawl) as long as they're within the topsite-base-url.
+#     e.g. seedURLs on docs.google.com containing links will have those linked pages and any
+#     they link to etc. downloaded as long as they're on docs.google.com.
+#   - <url-form-without-protocol>: if a seedurl contains topsite-base-url, then the provided
+#     url-form-without-protocol will make up the urlfilter, again preventing leaking into a
+#     larger part of the domain. For example, if the seedurl is mi.wikipedia.org/SomePage, it will
+#     match the topsite-base-url of wikipedia.org for which the <url-form-without-protocol>
+#     value is mi.wikipedia.org, which should be all that's accepted for wikipedia.org. The
+#     <url-form-without-protocol> ends up in the regex urlfilter file, thereby restricting the
+#     crawl to just mi.wikipedia.org.
+#     Remember to leave out any protocol <from url-form-without-protocol>.
+#
+# TODO If useful:
+#   column 3: whether nutch should do fetch all or not
+#   column 4: number of crawl iterations
+
+
+# NOT TOP SITES, BUT SITES WE INSPECTED AND WANT TO CONTROL SIMILARLY TO TOP SITES
+00.gs,SINGLEPAGE
+# May be a large site with only seedURLs of real relevance
+topographic-map.com,SINGLEPAGE
+ami-media.net,SINGLEPAGE
+# 2 pages of declarations of human rights in Maori, rest in other languages
+anitra.net,SINGLEPAGE
+# special case
+mi.centr-zashity.ru,SINGLEPAGE
+
+martinvrijland.nl,martinvrijland.nl/mi/
+csunplugged.org,csunplugged.org/mi/
+gpedia.com,gpedia.com/mi/
+
+# TOP SITE BUT NOT TOP 500
+www.tumblr.com,SINGLEPAGE
+
+
+# TOP SITES
+
+# docs.google.com is a special case: not all pages are public and any interlinking is likely to
+# be intentional. Grab all linked pages, for link depth set with nutch's crawl, as long as the
+# links are within the given topsite-base-url
+docs.google.com,FOLLOW-LINKS-WITHIN-TOPSITE
+
+# Just crawl a single page for these:
+drive.google.com,SINGLEPAGE
+forms.office.com,SINGLEPAGE
+player.vimeo.com,SINGLEPAGE
+static-promote.weebly.com,SINGLEPAGE
+
+# Special case of yale.edu: its Rapa-Nui pages are on blacklist, but we want this page + its photos
+# The page's containing folder is whitelisted in case the photos are there.
+korora.econ.yale.edu,SINGLEPAGE
+
+
+000webhost.com
+360.cn
+4shared.com
+a8.net
+abc.es
+abc.net.au
+abcnews.go.com
+about.com
+about.me
+aboutads.info
+abril.com.br
+academia.edu
+accuweather.com
+addthis.com
+addtoany.com
+adobe.com
+adweek.com
+airbnb.com
+akamaihd.net
+alexa.com
+alibaba.com
+aliexpress.com
+alipay.com
+aljazeera.com
+allaboutcookies.org
+allrecipes.com
+amazon.ca
+amazon.co.jp
+amazon.co.uk
+amazon.com
+amazon.de
+amazon.es
+amazon.fr
+amazon.in
+ameblo.jp
+ampproject.org
+android.com
+aol.com
+ap.org
+apache.org
+apachefriends.org
+apple.com
+archive.org
+archives.gov
+arstechnica.com
+arxiv.org
+asahi.com
+ask.fm
+asus.com
+axs.com
+babytree.com
+baidu.com
+bandcamp.com
+bbc.co.uk
+bbc.com
+behance.net
+berkeley.edu
+biblegateway.com
+biglobe.ne.jp
+billboard.com
+bing.com
+bit.ly
+bitly.com
+blackberry.com
+blogger.com
+blogspot.com,SUBDOMAIN-COPY
+bloomberg.com
+booking.com
+boston.com
+box.com
+britannica.com
+bt.com
+bund.de
+businessinsider.com
+businesswire.com
+buydomains.com
+buzzfeed.com
+ca.gov
+cambridge.org
+canalblog.com
+cbc.ca
+cbslocal.com
+cbsnews.com
+cdc.gov
+change.org
+channel4.com
+chicagotribune.com
+chinadaily.com.cn
+cisco.com
+clickbank.net
+cloudflare.com
+cmu.edu
+cnbc.com
+cnet.com
+cnn.com
+cocolog-nifty.com
+columbia.edu
+connect.over-blog.com
+cornell.edu
+corriere.it
+cpanel.com
+cpanel.net
+creativecommons.org
+csdn.net
+csmonitor.com
+dailymail.co.uk
+dailymotion.com
+dan.com
+daum.net
+debian.org
+dell.com
+depositfiles.com
+detik.com
+digg.com
+discovery.com
+disney.com
+disney.go.com
+disqus.com
+doubleclick.net
+dreniq.com
+dribbble.com
+dropbox.com,SINGLEPAGE
+dropboxusercontent.com
+dw.com
+e-recht24.de
+ea.com
+ebay.co.uk
+ebay.com
+economist.com
+eff.org
+ehow.com
+elmundo.es
+elpais.com
+engadget.com
+entrepreneur.com
+eonline.com
+espn.com
+espn.go.com
+etsy.com
+europa.eu
+eventbrite.com
+example.com
+excite.co.jp
+express.co.uk
+facebook.com
+fandom.com
+fastcompany.com
+fb.com
+fb.me
+fda.gov
+fedoraproject.org
+feedburner.com
+fifa.com
+files.wordpress.com
+flickr.com
+forbes.com
+fortune.com
+foursquare.com
+foxnews.com
+ft.com
+ftc.gov
+gen.xyz
+geocities.jp
+gesetze-im-internet.de
+ggpht.com
+github.com
+gizmodo.com
+globo.com
+gmail.com
+gnu.org
+godaddy.com
+gofundme.com
+goo.gl
+goo.ne.jp
+goodreads.com
+google.ca
+google.co.id
+google.co.in
+google.co.jp
+google.co.uk
+google.com
+google.com.br
+google.com.hk
+google.com.tr
+google.de
+google.es
+google.fr
+google.it
+google.nl
+google.pl
+google.ru
+googleapis.com
+googleblog.com
+googleusercontent.com
+gooyaabitemplates.com
+gov.uk
+gravatar.com
+greenpeace.org
+gstatic.com
+guardian.co.uk
+harvard.edu
+hatena.ne.jp
+histats.com
+hm.com
+hollywoodreporter.com
+home.pl
+house.gov
+howstuffworks.com
+hp.com
+huffingtonpost.com
+huffpost.com
+hugedomains.com
+ibm.com
+ibtimes.com
+icann.org
+ieee.org
+ietf.org
+ig.com.br
+ign.com
+ikea.com
+imageshack.us
+imdb.com
+imgur.com
+inc.com
+independent.co.uk
+indiatimes.com
+indiegogo.com
+instagram.com
+instructables.com
+intel.com
+interia.pl
+issuu.com
+istockphoto.com
+iubenda.com
+jd.com
+joomla.org
+jquery.com
+jstor.org
+kickstarter.com
+kinja.com
+last.fm
+latimes.com
+lefigaro.fr
+lemonde.fr
+line.me
+linkedin.com
+list-manage.com
+live.com
+livejournal.com
+livescience.com
+loc.gov
+lonelyplanet.com
+lycos.com
+m.wikipedia.org,mi.m.wikipedia.org
+mail.ru
+marketwatch.com
+marriott.com
+mashable.com
+mediafire.com
+medium.com
+mega.nz
+megaupload.com
+mercurynews.com
+merriam-webster.com
+metro.co.uk
+microsoft.com,microsoft.com/mi-nz/
+microsoftonline.com
+mirror.co.uk
+mit.edu
+mixcloud.com
+mlb.com
+mozilla.com
+mozilla.org
+msn.com
+myspace.com
+mysql.com
+namecheap.com
+narod.ru
+nasa.gov
+nationalgeographic.com
+nature.com
+naver.com
+naver.jp
+nba.com
+nbcnews.com
+ndtv.com
+netflix.com
+netsons.com
+netvibes.com
+networkadvertising.org
+news.com.au
+newscientist.com
+newsweek.com
+newyorker.com
+nginx.com
+nginx.org
+nhk.or.jp
+nicovideo.jp
+nifty.com
+nih.gov
+nikkei.com
+noaa.gov
+nokia.com
+npr.org
+nvidia.com
+nydailynews.com
+nypost.com
+nytimes.com
+nyu.edu
+odnoklassniki.ru
+office.com
+offset.com
+ok.ru
+okezone.com
+opera.com
+oracle.com
+orange.fr
+oreilly.com
+oup.com
+over-blog.com
+ovh.co.uk
+ovh.com
+ovh.net
+ox.ac.uk
+parallels.com
+pastebin.com
+paypal.com
+pbs.org
+pcmag.com
+people.com
+photobucket.com
+php.net
+pinterest.com,SINGLEPAGE
+pixabay.com
+playstation.com
+plesk.com
+plos.org
+politico.com
+prestashop.com
+prezi.com
+princeton.edu
+privacyshield.gov
+prnewswire.com
+psychologytoday.com
+qq.com
+quantcast.com
+quora.com
+rakuten.co.jp
+rambler.ru
+rapidshare.com
+reddit.com
+repubblica.it
+researchgate.net
+reuters.com
+ria.ru
+rottentomatoes.com
+rt.com
+rtve.es
+sakura.ne.jp
+samsung.com
+sapo.pt
+scholastic.com
+sciencedaily.com
+sciencedirect.com
+sciencemag.org
+scientificamerican.com
+scribd.com
+seattletimes.com
+secureserver.net
+sedo.com
+seesaa.net
+sendspace.com
+sfgate.com
+shopify.com
+shutterstock.com
+siemens.com
+sina.com.cn
+sky.com
+skype.com
+skyrock.com
+slate.com
+slideshare.net
+sm.cn
+smh.com.au
+so-net.ne.jp
+softonic.com
+sogou.com
+sohu.com
+soratemplates.com
+soso.com
+soundcloud.com
+spiegel.de
+spotify.com
+springer.com
+sputniknews.com
+ssl-images-amazon.com
+stackoverflow.com
+standard.co.uk
+stanford.edu
+state.gov
+steamcommunity.com
+steampowered.com
+storage.canalblog.com
+storage.googleapis.com
+stores.jp
+storify.com
+stuff.co.nz,SINGLEPAGE
+surveymonkey.com
+symantec.com
+t-online.de
+t.co
+t.me
+tabelog.com
+taobao.com
+target.com
+teamviewer.com
+techcrunch.com
+ted.com
+telegram.me
+telegraph.co.uk
+terra.com.br
+theatlantic.com
+thefreedictionary.com
+theglobeandmail.com
+theguardian.com
+themeforest.net
+thenextweb.com
+thestar.com
+thesun.co.uk
+thetimes.co.uk
+theverge.com
+thoughtco.com
+tianya.cn
+time.com
+tinyurl.com
+tmall.com
+tmz.com
+tribunnews.com
+tripadvisor.com
+trustpilot.com
+twitch.tv
+twitter.com
+ucoz.ru
+uiuc.edu
+umich.edu
+un.org
+undeveloped.com
+unesco.org
+uol.com.br
+urbandictionary.com
+usa.gov
+usatoday.com
+usgs.gov
+usnews.com
+uspto.gov
+ustream.tv
+utexas.edu
+variety.com
+venturebeat.com
+vice.com
+viglink.com
+vimeo.com
+vk.com
+vkontakte.ru
+vox.com
+w3.org
+w3schools.com
+wa.me
+walmart.com
+washington.edu
+washingtonpost.com
+wattpad.com
+weather.com
+web.fc2.com
+webmd.com
+weebly.com
+weibo.com
+welt.de
+whatsapp.com
+whitehouse.gov
+who.int
+wikia.com
+wikihow.com
+wikimedia.org
+wikipedia.org,mi.wikipedia.org
+wiktionary.org,mi.wiktionary.org
+wiley.com
+windowsphone.com
+wired.com
+wix.com
+wordpress.org,SUBDOMAIN-COPY
+worldbank.org
+wp.com
+wsj.com
+xbox.com
+xinhuanet.com
+yadi.sk
+yahoo.co.jp
+yahoo.com
+yale.edu
+yandex.ru
+yelp.com
+youku.com
+youronlinechoices.com
+youtu.be
+youtube.com
+ytimg.com
+zdnet.com
+zend.com
+zendesk.com
+zippyshare.com
Index: /other-projects/maori-lang-detection/conf/url-blacklist-filter.txt
===================================================================
--- /other-projects/maori-lang-detection/conf/url-blacklist-filter.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/url-blacklist-filter.txt	(revision 33635)
@@ -0,0 +1,122 @@
+# URL blacklist
+# FORMAT:
+# precede URL by ^ to blacklist urls that match the given prefix
+# succeed URL by $ to blacklist urls that match the given suffix
+# ^url$ will blacklist urls that match the given url completely
+# Without either ^ or $ symbol, urls containing the given url will get blacklisted
+
+
+# manually adjusting for irrelevant topsite hits
+# Rapa-Nui is related to Easter Island
+^http://codex.cs.yale.edu/avi/silberschatz/gallery/trips-photos/South-America/Rapa-Nui/
+
+# We will blacklist this yale.edu domain except for the subportion that gets whitelisted
+# then in the sites-too-big-to-exhaustively-crawl.txt, we have a mapping for an allowed url
+# pattern in case elements on the page are stored elsewhere
+^http://korora.econ.yale.edu/
+
+# wikipedia pages in
+# ksh (a German dialect), ilo (Filippino), ty Tahitian, wa for Walons/Walloon,
+# io (Ido version of Esperanto) and zh-min-nan (Min-Nan-Chinese) are not in the Maori language
+# Not sure why Commoncrawl had found them for language code MRI
+ksh.wikipedia.org
+ilo.wikipedia.org
+wa.wikipedia.org
+ty.m.wikipedia.org
+io.m.wikipedia.org
+zh-min-nan.wikipedia.org
+zh-min-nan.wiktionary.org
+
+######
+# unwanted domains
+.video-chat.
+.videochat.
+3chat.ru
+livevideochatting.org
+lovewebcam.net
+
+cherrybabe.biz
+dreamsbabes.com
+adultfantasyboutique.com
+adultterra.com
+
+leatherdyke.porn
+hornyteenharlots.com
+adultviewsex.com
+adultsexualvideo.com
+ctbererotica.sexe-traque.com
+cybererotia.porn234.com
+cybereroticz.adultsupermart.com
+freegaywebcams.info
+lesbiansinmysoup.com
+videopornoxx.online
+sexandplay.com
+sexynakedselfies.info
+barebabez.com
+britnudes.net
+camaporno.com
+webxvideo.com
+gayspornosex.com
+jasminreviews.com
+sexchatlines4u.com
+sexybabeworld.org
+sexyleaks.info
+uniqueporno.com
+wildsexsluts.com
+xxxblacknudes.com
+bigsexymelons.com
+
+# more adult sites
+acba.osb-land.com
+
+
+# just get rid of any URL containing "livejasmin"
+## livejasmin
+# Actually: do that in the code (CCWETProcessor) with a log message,
+# since we actually need to get rid of any sites in entirety that contain
+# any url with the string "livejasmin"
+# So run the program once, check the log for messages mentioning "additional"
+# adult sites found and add their domains in here.
+anigma-beauty.com
+adultfeet.com
+atopian.org
+bellydancingvideo.net
+bmmodelsagency.com
+brucknergallery.com
+fuckvidz.org
+photobattle.net
+votekat.info
+
+# Similar to above, the following contained the string "jasmin" in the URL
+teenycuties.com
+a.tiles.mapbox.com
+blazingteens.net
+redtubeporn.info
+osb-land.com
+totallyhotmales.com
+babeevents.com
+talkserver.de
+hehechat.org
+fetish-nights.com
+lesslove.com
+hebertsvideo.com
+
+# sounds like some pirating site
+^http://pirateguides.com/
+fastmp3.ru
+
+# from alexa topsites at https://www.alexa.com/topsites
+livejasmin.com
+pornhub.com
+# listed as a similar topsite at https://en.wikipedia.org/wiki/List_of_most_popular_websites
+redtube.com
+xvideos.com
+xhamster.com
+xnxx.com
+
+
+# not sure about the domain name and/or full url seems like it belongs here
+abcutie.com
+
+# only had a single seedURL and it quickly redirected to an adult site
+apparactes.gq
Index: /other-projects/maori-lang-detection/conf/url-greylist-filter.txt
===================================================================
--- /other-projects/maori-lang-detection/conf/url-greylist-filter.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/url-greylist-filter.txt	(revision 33635)
@@ -0,0 +1,65 @@
+# URL 'greylist': save matching urls to one side, to eyeball later and decide if
+# they should be included after all or whether it was okay to have skipped them
+# FORMAT:
+# precede URL by ^ to greylist urls that match the given prefix
+# succeed URL by $ to greylist urls that match the given suffix
+# ^url$ will greylist urls that match the given url completely
+# Without either ^ or $ symbol, urls containing the given url will get greylisted
+
+
+# Product sites: unwanted auto-translation pages of online product stores and other websites
+/product/
+/products/
+/product-page/
+/product-category/
+ledlamp.china-led-lighting.com
+ledpar64.china-led-lighting.com
+ledwallwasher.china-led-lighting.com
+abacre.com
+cn-huafu.net
+apteka.social
+
+
+# not product stores but autotranslated?
+192-168-1-1l.com
+19216811login.club
+19216811login.club
+1videosmusica.com
+256file.com
+# already in greylisting of all .ru 
+#7773033.ru
+#abali.ru
+#allbeautyone.ru
+aqualuz.org
+
+# if page doesn't load and can't be tested
+1videosmusica.com
+www.kiterewa.pl
+
+
+
+# MANUALLY INSPECTED URLS AND ADDED TO GREYLIST
+
+# license plate site? - already in greylisting of all .ru
+#eba.com.ru
+
+# As per archive.org, there's just a photo on the defunct page at this site
+# And the picture label and filename is probably Japanese
+agri.mine.utsunomiya-u.ac.jp
+
+# seems to be Indonesian or Malaysian Bible rather than in Maori or any Polynesian language
+alkitab.life:2022
+
+# appears defunct
+alixira.com
+
+# single seedURL was not a page in Maori, but global languages.
+# And the rest of the domain appears to be in English
+anglican.org
+
+
+### TLDs that we greylist - any exceptions will be in the whitelist
+# Our list of .ru and .pl domains were not relevant
+.ru/
+.pl/
+.tk/
Index: /other-projects/maori-lang-detection/conf/url-whitelist-filter.txt
===================================================================
--- /other-projects/maori-lang-detection/conf/url-whitelist-filter.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/conf/url-whitelist-filter.txt	(revision 33635)
@@ -0,0 +1,31 @@
+# URL 'whitelist': urls of these forms go into the keep pile.
+# whitelist overrides blacklist and greylist.
+# FORMAT:
+# precede URL by ^ to whitelist urls that match the given prefix
+# succeed URL by $ to whitelist urls that match the given suffix
+# ^url$ will whitelist urls that match the given url completely
+# Without either ^ or $ symbol, urls containing the given url will get whitelisted
+
+# Special exception for this url on yale.edu, since we needed to blacklist
+# some particular other urls on yale.edu
+http://korora.econ.yale.edu/phillips/archive/hauraki.htm
+
+# We've added .ru$ sites to the blacklist, but the following 
+# Russian website contains actual Maori language content
+http://www.krassotkin.ru/sites/prayer.su/maori/
+https://mi.centr-zashity.ru/
+
+
+
+# WHITELIST WEBSITES THAT HAVE NON-AUTOMATED /mi/ SUBSECTIONS
+# WE CONTROL WHAT PART OF THEM WILL BE DOWNLOADED (THE /mi SUBSECTION)
+# IN sites-too-big-to-exhaustively-crawl.txt
+#https://www.martinvrijland.nl/mi/te-mana-hinengaro/Ko-te-nuinga-ake-o-nga-tangata-kei-te-timata-ki-te-kite-kei-te-noho-tatou-i-roto-i-te-whakaata-ko-te-aha-tenei/
+#https://www.csunplugged.org/mi/principles/
+#http://www.gpedia.com/mi/gpedia/Reo_M%C4%81ori
+
+https://www.martinvrijland.nl
+https://www.csunplugged.org
+http://www.gpedia.com
+
+
Index: /other-projects/maori-lang-detection/feasibility.txt
===================================================================
--- /other-projects/maori-lang-detection/feasibility.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/feasibility.txt	(revision 33635)
@@ -0,0 +1,17 @@
+Total URLs with tld .nz at commoncrawl for July 2019's crawl: 
+6862400
+
+Total uniq domains: 126990
+
+About 54 pages on average *crawled* per site.
+
+
+https://www.quora.com/What-is-the-average-number-of-webpages-on-a-website
+https://www.internetlivestats.com/total-number-of-websites/
+https://www.webnots.com/3-ways-to-find-number-of-pages-on-a-website/
+"There are over 1.5 billion websites on the world wide web today. Of these, less than 200 million are active."
+
+"It must be noted that around 75% of websites today are not active, but parked domains or similar. [1]"
+
+https://en.wikipedia.org/wiki/Domain_parking
+"Domain parking is the registration of an Internet domain name without that domain being associated with any services such as e-mail or a website."
Index: /other-projects/maori-lang-detection/hdfs-cc-work/GS_README.TXT
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/GS_README.TXT	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/GS_README.TXT	(revision 33635)
@@ -0,0 +1,711 @@
+----------------------------------------
+INDEX: follow in sequence
+----------------------------------------
+A.  VAGRANT VM WITH HADOOP AND SPARK
+B.  Create IAM role on Amazon AWS to use S3a
+C.  Configure Spark on your vagrant VM with the AWS authentication details
+---
+Script scripts/setup.sh now is automated to do the steps in D-F below
+and prints out the main instruction for G.
+---
+D.  OPTIONAL? Further configuration for Hadoop to work with Amazon AWS
+E.  Setup cc-index-table git project
+F.  Setup warc-to-wet tools (git projects)
+G.  Getting and running our scripts
+---
+H. Austici's crawl - CLI to download web sites as WARCs, features basics to avoid crawler taps
+I. Setting up Nutch v2 on its own Vagrant VM machine
+J. Automated crawling with Nutch v2.3.1 and post-processing
+
+----------------------------------------
+
+----------------------------------------
+A. VAGRANT VM WITH HADOOP AND SPARK
+----------------------------------------
+Set up vagrant with hadoop and spark as follows
+
+1. by following the instructions at
+https://github.com/martinprobson/vagrant-hadoop-hive-spark
+
+This will eventually create the following folder, which will contain Vagrantfile
+/home/<USER>/vagrant-hadoop-hive-spark
+
+2. If there are other vagrant VMs set up according to the same instructions on the same machine, then need to change the forwarded ports (the 2nd column of ports) in the file "Vagrantfile". In the example below, excerpted from my Vagrantfile, I've incremented the forwarded ports by 1:
+
+    config.vm.network "forwarded_port", guest: 8080, host: 8081
+    config.vm.network "forwarded_port", guest: 8088, host: 8089
+    config.vm.network "forwarded_port", guest: 9083, host: 9084
+    config.vm.network "forwarded_port", guest: 4040, host: 4041
+    config.vm.network "forwarded_port", guest: 18888, host: 18889
+    config.vm.network "forwarded_port", guest: 16010, host: 16011
+
+Remember to visit the adjusted ports on the running VM.
+
+3. The most useful vagrant commands:
+vagrant up      # start up the vagrant VM if not already running.
+	        # May need to provide VM's ID if there's more than one vagrant VM
+ssh vagrant     # ssh into the sole vagrant VM, else may need to provide vagrant VM's ID
+
+vagrant halt    # to shutdown the vagrant VM. Provide VM's ID if there's more than one vagrant VM.
+
+(vagrant destroy)   # to get rid of your vagrant VM. Useful if you've edited your Vagrantfile
+
+
+4. Inside the VM, /home/<USER>/vagrant-hadoop-hive-spark will be shared and mounted as /vagrant
+Remember, this is the folder containing Vagrantfile. It's easy to use the shared folder to transfer files between the VM and the actual machine that hosts it.
+
+5. Install EMACS, FIREFOX AND MAVEN on the vagrant VM:
+Start up vagrant machine ("vagrant up") and ssh into it ("ssh vagrant") if you haven't already.
+
+
+a. sudo apt-get -y install firefox
+
+b. sudo apt-get install emacs
+
+c. sudo apt-get install maven
+  (or sudo apt update
+   sudo apt install maven)
+
+Maven is needed for the commoncrawl github projects we'll be working with.
+
+
+6. Although you can edit the Vagrantfile to have emacs and maven automatically installed when the vagrant VM is created, for firefox, you're advised to install it as above.
+
+To be able to view firefox from the machine hosting the VM, use a separate terminal and run:
+   vagrant ssh -- -Y
+[or "vagrant ssh -- -Y node1", if VM ID is node1]
+
+READING ON Vagrant:
+	* Guide: https://www.vagrantup.com/intro/getting-started/index.html
+	* Common cmds: https://blog.ipswitch.com/5-vagrant-commands-you-need-to-know
+	* vagrant reload = vagrant halt + vagrant up https://www.vagrantup.com/docs/cli/reload.html
+	* https://stackoverflow.com/questions/46903623/how-to-use-firefox-ui-in-vagrant-box
+	* https://stackoverflow.com/questions/22651399/how-to-install-firefox-in-precise64-vagrant-box
+	  sudo apt-get -y install firefox
+	* vagrant install emacs: https://medium.com/@AnnaJS15/getting-started-with-virtualbox-and-vagrant-8d98aa271d2a
+
+	* hadoop conf: sudo vi /usr/local/hadoop-2.7.6/etc/hadoop/mapred-site.xml
+	* https://data-flair.training/forums/topic/mkdir-cannot-create-directory-data-name-node-is-in-safe-mode/
+
+-------------------------------------------------
+B. Create IAM role on Amazon AWS to use S3 (S3a)
+-------------------------------------------------
+CommonCrawl (CC) crawl data is stored on Amazon S3, specifically the newest version Amazon s3a which has superceded both s3 and its earlier successor s3n.
+
+In order to have access to cc crawl data, need to create an IAM role on Dr Bainbridge's Amazon AWS account and configure its profile for commoncrawl.
+
+1. Log into Dr Bainbridge's Amazon AWS account
+- In the aws management console:
+davidb@waikato.ac.nz
+lab pwd, capital R and ! (maybe g)
+
+
+2. Create a new "iam" role or user for "commoncrawl(er)" profile
+
+3. You can create the commoncrawl profile while creating the user/role, by following the instructions at https://answers.dataiku.com/1734/common-crawl-s3
+which states
+
+"Even though the bucket is public, if your AWS key does not have your full permissions (ie if it's a restricted IAM user), you need to grant explicit access to the commoncrawl bucket: attach the following policy to your IAM user"
+
+#### START POLICY IN JSON FORMAT ###
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "Stmt1503647467000",
+            "Effect": "Allow",
+            "Action": [
+                "s3:GetObject",
+                "s3:ListBucket"
+            ],
+            "Resource": [
+                "arn:aws:s3:::commoncrawl/*",
+                "arn:aws:s3:::commoncrawl"
+            ]
+        }
+    ]
+}
+#### END POLICY ###
+
+
+--------------------------------------------------------------------------
+C. Configure Spark on your vagrant VM with the AWS authentication details
+--------------------------------------------------------------------------
+Any Spark jobs run against the CommonCrawl data stored on Amazon s3a need to be able to authenticate with the AWS IAM role you created above. In order to do this, you'll want to put the Amazon AWS access key and secret key in the SPARK configuration properties file. (Instead of configuring these values in hadoop's core-site.xml, as in the latter case, the authentication details don't get copied across when distributed jobs are run to other computers in the distributed cluster that also need to know how to authenticate):
+
+1. Inside the vagrant vm:
+
+   sudo emacs /usr/local/spark-2.3.0-bin-hadoop2.7/conf/spark-defaults.conf
+   (sudo emacs $SPARK_HOME/conf/spark-defaults.conf)
+
+2. Edit the spark properties conf file to contain these 3 new properties:
+
+    spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem  
+    spark.hadoop.fs.s3a.access.key=PASTE_IAM-ROLE_ACCESSKEY_HERE
+    spark.hadoop.fs.s3a.secret.key=PASTE_IAM-ROLE_SECRETKEY_HERE
+
+Instructions on which properties to set were taken from:
+- https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark
+- https://community.cloudera.com/t5/Community-Articles/HDP-2-4-0-and-Spark-1-6-0-connecting-to-AWS-S3-buckets/ta-p/245760
+
+[NOTE, inactive alternative: Instead of editing spark's config file to set these properties, these properties can also be set in the bash script that executes the commoncrawl Spark jobs:
+
+$SPARK_HOME/bin/spark-submit \
+   ...
+   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   ...
+
+But better not to hardcode authentication details into code, so I did it the first way.
+]
+
+----------------------------------------------------------------------
+NOTE:
+Script scripts/setup.sh now is automated to do the steps in D-F below
+and prints out the main instruction for G.
+
+
+----------------------------------------------------------------------
+D. OPTIONAL? Further configuration for Hadoop to work with Amazon AWS
+----------------------------------------------------------------------
+The following 2 pages state that additional steps are necessary to get hadoop and spark to work with AWS S3a:
+
+- https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark
+- https://community.cloudera.com/t5/Community-Articles/HDP-2-4-0-and-Spark-1-6-0-connecting-to-AWS-S3-buckets/ta-p/245760
+
+I'm not sure whether these steps were really necessary in my case, and if so, whether it was A or B below that got things working for me. However, I have both A and B below set up.
+
+
+A. Check your maven installation for necessary jars:
+
+1. Installing maven may already have got the specifically recommended version of AWS-Java-SDK (aws-java-sdk-1.7.4.jar) and v2.7.6 hadoop-aws matching the vagrant VM's hadoop version (hadoop-aws-2.7.6.jar). Check these locations, as that's where I have them:
+- /home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar
+- /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+
+The specifically recommended v.1.7.4 from the instructions can be found off https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.4 at https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar
+
+2. The script that runs the 2 Spark jobs uses the above paths for one of the spark jobs:
+   $SPARK_HOME/bin/spark-submit \
+       --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+       --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+
+However the other Spark job in the script does not set --jars or --driver-class-path, despite also referring to the s3a://commoncrawl table. So I'm not sure whether the jars are necessary or whether theywere just being ignored when provided.
+
+B. Download jar files and put them on the hadoop classpath:
+
+1. download the jar files:
+- I obtained aws-java-sdk-1.11.616.jar (v1.11) from https://aws.amazon.com/sdk-for-java/
+
+- I downloaded hadoop-aws 2.7.6 jar, as it goes with my version of hadoop, from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.6
+
+2. The easiest solution is to copy the 2 downloaded jars onto a location in the hadoop classpath.
+
+a. The command that shows the paths present on the Hadoop CLASSPATH:
+	hadoop classpath
+One of the paths this will list is /usr/local/hadoop-2.7.6/share/hadoop/common/
+
+b. SUDO COPY the 2 downloaded jar files, hadoop-aws-2.7.6.jar and aws-java-sdk-1.11.616.jar, to this location:
+
+     sudo cp hadoop-aws-2.7.6.jar /usr/local/hadoop-2.7.6/share/hadoop/common/.
+     sudo cp aws-java-sdk-1.11.616.jar /usr/local/hadoop-2.7.6/share/hadoop/common/.
+
+Any hadoop jobs run will now find these 2 jar files on the classpath.
+
+[NOTE, unused alternative: Instead of copying the 2 jar files into a system location, assuming they were downloaded into /home/vagrant/lib, you can also export a custom folder's jar files into the hadoop classpath from the bash script that runs the spark jobs. This had no effect for me, and was commented out, and is another reason why I'm not sure if the 2 jar files were even necessary.
+#export LIBJARS=/home/vagrant/lib/*                                 
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+]
+
+
+------------------------------------
+E. Setup cc-index-table git project
+------------------------------------
+Need to be inside the vagrant VM.
+
+1. Since you should have already installed maven, you can checkout and compile the cc-index-table git project.
+
+       git clone https://github.com/commoncrawl/cc-index-table.git
+
+2. Modify the toplevel pom.xml file used by maven by changing the spark version used to 2.3.0 and adding a dependency for hadoop-aws 2.7.6, as indicated below:
+
+17c17,18
+< 		<spark.version>2.4.1</spark.version>
+---
+> 		<!--<spark.version>2.4.1</spark.version>-->
+> 		<spark.version>2.3.0</spark.version>
+135a137,143
+> 		<dependency>
+> 		  <groupId>org.apache.hadoop</groupId>
+> 		  <artifactId>hadoop-aws</artifactId>
+> 		  <version>2.7.6</version>
+> 		</dependency>
+> 
+
+3. Although cc-index-table will compile successfully after the above modifications, it will nevertheless throw an exception when it's eventually run. To fix that, edit the file "cc-index-table/src/main/java/org/commoncrawl/spark/examples/CCIndexWarcExport.java" as follows:
+
+a. Set option(header) to false, since the csv file contains no header row, only data rows.
+   Change:
+	sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+                                        .load(csvQueryResult);
+   To
+	sqlDF = sparkSession.read().format("csv").option("header", false).option("inferSchema", true)
+                                        .load(csvQueryResult);
+
+b. The 4 column names are inferred as _c0 to _c3, not as url/warc_filename etc.
+   Comment out:
+	//JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+                                .toJavaRDD();
+   Replace with the default inferred column names:
+	JavaRDD<Row> rdd = sqlDF.select("_c0", "_c1", "_c2", "_c3").rdd()
+                                .toJavaRDD();
+
+// TODO: link svn committed versions of orig and modified CCIndexWarcExport.java here.
+
+4. Now (re)compile cc-index-table with the above modifications:
+
+   cd cc-index-table
+   mvn package
+
+-------------------------------
+F. Setup warc-to-wet tools
+-------------------------------
+To convert WARC files to WET (.warc.wet) files, need to checkout, set up and compile a couple more tools. These instructions are derived from those at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+1. Grab and compile the 2 git projects for converting warc to wet:
+     git clone https://github.com/commoncrawl/ia-web-commons
+     cd ia-web-commons
+     mvn install
+
+     git clone https://github.com/commoncrawl/ia-hadoop-tools
+     cd ia-hadoop-tools
+     # can't compile this yet
+     
+
+2. Add the following into ia-hadoop-tools/pom.xml, in the top of the <dependencies> element, so that maven finds an appropriate version of the org.json package and its JSONTokener (version number found off https://mvnrepository.com/artifact/org.json/json):
+
+   <dependency>
+      <groupId>org.json</groupId>
+      <artifactId>json</artifactId>
+      <version>20131018</version>
+    </dependency>
+
+[
+  UNFAMILAR CHANGES that I don't recollect making and that may have been a consequence of the change in step 1 above:
+  a. These further differences show up between the original version of the file in pom.xml.orig and the modified new pom.xml:
+  ia-hadoop-tools>diff pom.xml.orig pom.xml
+
+  <       <groupId>org.netpreserve.commons</groupId>
+  <       <artifactId>webarchive-commons</artifactId>
+  <       <version>1.1.1-SNAPSHOT</version>
+  ---
+  >       <groupId>org.commoncrawl</groupId>
+  >       <artifactId>ia-web-commons</artifactId>
+  >       <version>1.1.9-SNAPSHOT</version>
+
+  b. I don't recollect changing or manually introducing any java files. I just followed the instructions at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+  However, a diff -rq between the latest "ia-hadoop-tools" gitproject checked out a month after the "ia-hadoop-tools.orig" checkout I ran, shows the following differences in files which are not shown as recently modified in github itself in that same period.
+
+  ia-hadoop-tools> diff -rq ia-hadoop-tools ia-hadoop-tools.orig/
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/io/HDFSTouch.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/io/HDFSTouch.java differ
+  Only in ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs: ArchiveFileExtractor.java
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/CDXGenerator.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/CDXGenerator.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/JobDriver.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/JobDriver.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/WARCMetadataRecordGenerator.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/WARCMetadataRecordGenerator.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/jobs/WATGenerator.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs/WATGenerator.java differ
+  Only in ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/jobs: WEATGenerator.java
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/mapreduce/ZipNumPartitioner.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/mapreduce/ZipNumPartitioner.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/pig/ZipNumRecordReader.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/pig/ZipNumRecordReader.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/hadoop/streaming/ZipNumRecordReader.java and ia-hadoop-tools.orig/src/main/java/org/archive/hadoop/streaming/ZipNumRecordReader.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/server/FileBackedInputStream.java and ia-hadoop-tools.orig/src/main/java/org/archive/server/FileBackedInputStream.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/server/GZRangeClient.java and ia-hadoop-tools.orig/src/main/java/org/archive/server/GZRangeClient.java differ
+  Files ia-hadoop-tools/src/main/java/org/archive/server/GZRangeServer.java and ia-hadoop-tools.orig/src/main/java/org/archive/server/GZRangeServer.java differ
+]
+
+3. Now can compile ia-hadoop-tools:
+   cd ia-hadoop-tools
+   mvn package
+
+4. Can't run it until guava.jar is on hadoop classpath. Locate a guava.jar and put it into an existing location checked for by hadoop classpath:
+
+   locate guava.jar
+       # found in /usr/share/java/guava.jar and /usr/share/maven/lib/guava.jar
+   diff /usr/share/java/guava.jar /usr/share/maven/lib/guava.jar   
+	# identical/no difference, so can use either
+   sudo cp /usr/share/java/guava.jar /usr/local/hadoop/share/hadoop/common/.
+   	# now guava.jar has been copied into a location on hadoop classpath
+
+
+Having done the above, our bash script will now be able to convert WARC to WET files when it runs:
+    $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz hdfs:///user/vagrant/PATH/TO/warc/*.warc.gz
+Our script expects a specific folder structure: there should be a "warc" folder (containing the warc files), which is supplied as above, but also an empty "wet" and "wat" folder at the same level as the "warc" folder.
+
+
+When the job is running, can visit the Spark Context at http://node1:4040/jobs/ (http://node1:4041/jobs/ for me first time, since I forwarded the vagrant VM's ports at +1. However, subsequent times it was on node1:4040/jobs?)
+
+-----------------------------------
+G. Getting and running our scripts
+-----------------------------------
+
+1. Grab our 1st bash script and put it into the /home/vagrant/cc-index-table/src/script:
+       cd cc-index-table/src/script
+       wget http://svn.greenstone.org/gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh
+       chmod u+x get_maori_WET_records_for_crawl.sh
+
+RUN AS:
+cd cc-index-table
+./src/script/get_maori_WET_records_for_crawl.sh <crawl-timestamp>
+				where crawl-timestamp of form "CC-MAIN-YYYY-##" >= September 2019
+
+OUTPUT:
+After hours of processing (leave it to run overnight), you should end up with:
+      hdfs dfs -ls /user/vagrant/<crawl-timestamp>
+In particular, the zipped wet records at hdfs:///user/vagrant/<crawl-timestamp>/wet/
+that we want would have been copied into /vagrant/<crawl-timestamp>-wet-files/
+
+
+The script get_maori_WET_records_for_crawl.sh
+- takes a crawl timestamp of the form "CC-MAIN-YYYY-##" from Sep 2018 onwards (before which content_languages were not indexed). The legitimate crawl timestampts are listed in the first column at http://index.commoncrawl.org/
+- runs a spark job against CC's AWS bucket over s3a to create a csv table of MRI language records
+- runs a spark job to download all the WARC records from CC's AWS that are denoted by the csv file's records into zipped warc files
+- converts WARC to WET: locally converts the downloaded warc.gz files into warc.wet.gz (and warc.wat.gz) files
+
+
+2. Grab our 2nd bash script and put it into the top level of cc-index-table (/home/vagrant/cc-index/table):
+
+       cd cc-index-table
+       wget http://svn.greenstone.org/gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh
+       chmod u+x get_Maori_WET_records_from_CCSep2018_on.sh
+
+RUN FROM cc-index-table DIRECTORY AS:
+    (cd cc-index-table)
+    ./get_Maori_WET_records_from_CCSep2018_on.sh
+
+This script just runs the 1st script cc-index-table/src/script/get_maori_WET_records_for_crawl.sh (above) to process all listed common-crawls since September 2018.
+If any fails, then the script will terminate. Else it runs against each common-crawl in sequence.
+
+NOTE: If needed, update the script with more recent crawl timestamps from http://index.commoncrawl.org/
+
+OUTPUT:
+After days of running, will end up with:
+      hdfs:///user/vagrant/<crawl-timestamp>/wet/
+for each crawl-timestamp listed in the script,
+which at present would have got copied into
+      /vagrant/<crawl-timestamp>-wet-files/
+
+Each of these output wet folders can then be processed in turn by CCWETProcessor.java from http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java
+
+-----------------------------------
+H. Austici crawl
+-----------------------------------
+Austici's crawl: CLI to download web sites as WARCs, features basics to avoid crawler taps.
+
+Out of several software to do site mirroring, Autistici's "crawl" seemed promising:
+https://anarc.at/services/archive/web/
+
+- CLI.
+- Can download a website quite simply, though flags for additional settings are available.
+- Coded to prevent common traps.
+- Downloads website as WARC file
+- Now I have the WARC to WET process working for the WARC file it produced for the usual test site (Dr Bainbridge's home page)
+
+Need to have Go installed in order to install and run Autistici's crawl.
+Not a problem, because I can do it on the remote machine (which also hosts the hdfs) where I have sudo powers.
+
+INSTRUCTIONS
+
+1. Install go 1.11 by following instructions at https://medium.com/better-programming/install-go-1-11-on-ubuntu-18-04-16-04-lts-8c098c503c5f
+2. Create go environment:
+#!/bin/bash
+# environment vars for golang               
+export GOROOT=/usr/local/go
+export GOPATH=$HOME/go
+export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
+3. The https://git.autistici.org/ale/crawl/README.md instructions on installing are not very clear and don't work as is at this stage.
+
+These steps work:
+
+cd $GOPATH
+mkdir bin
+mkdir src
+cd src
+
+4. Since trying to go install the crawl url didn't work
+https://stackoverflow.com/questions/14416275/error-cant-load-package-package-my-prog-found-packages-my-prog-and-main
+[https://stackoverflow.com/questions/26694271/go-install-doesnt-create-any-bin-file]
+
+vagrant@node2:~/go/src$
+  mkdir -p git.autistici.org/ale
+  cd git.autistici.org/ale
+  git clone https://git.autistici.org/ale/crawl.git
+
+[Now can run the install command in README.md:]
+  cd $GOPATH/src
+  go install git.autistici.org/ale/crawl/cmd/crawl
+
+Now we should have a $GOPATH/bin folder containing the "crawl" binary
+
+5. Run a crawl:
+  cd $GOPATH/bin
+  ./crawl https://www.cs.waikato.ac.nz/~davidb/
+
+which downloads the site and puts the warc file into the $GOPATH/bin folder.
+
+More options, including output folder, WARC filename pattern for huge sites so that multiple warc files created for one site follow the same pattern are all in the instructions in README.md
+
+6. To view the RAW contents of a WARC file:
+https://github.com/ArchiveTeam/grab-site/blob/master/README.md#viewing-the-content-in-your-warc-archives
+
+zless <warc-file-name>
+
+zless already installed on vagrant file
+
+
+-----------------------------------------------------------------------------------------------
+How to run warc-to-wet conversion on sites downloaded as WARCs by Austici's "crawl" 
+-----------------------------------------------------------------------------------------------
+ISSUES CONVERTING WARC to WET:
+---
+WARC files produced by Autistici crawl are of a somewhat different format to CommonCrawl WARCs.
+- missing elements in header
+- different header elements
+- ordering different (if that matters)
+
+But WET is an official format, not CommonCrawl specific, as indicated by
+
+https://library.stanford.edu/projects/web-archiving/research-resources/data-formats-and-apis
+"WET (parsed text)
+
+WARC Encapsulated Text (WET) or parsed text consists of the extracted plaintext, delimited by archived document. Each record retains the associated URL and timestamp. Common Crawl provides details on the format and Internet Archive provides documentation on usage, though they use different names for the format."
+
+So must be possible to get WARC to WET conversion used for CommonCrawl data to work on Autistici crawl's WARC files.
+
+
+RESOLUTION:
+---
+I made changes to 2 java source files in the 2 github projects ia-web-commons and ia-hadoop-tools, which we use for the WARC to WET processing of CommonCrawl data. These gitprojects (with modifications for commoncrawl) are already on http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/gitprojects.
+
+The changed files are as follows:
+1. patches/WATExtractorOutput.java
+   put into ia-web-commons/src/main/java/org/archive/extract
+   after renaming existing to .orig
+
+THEN RECOMPILE ia-web-commons with:
+   mvn install
+
+2. patches/GZRangeClient.java
+   put into ia-hadoop-tools/src/main/java/org/archive/server
+   after renaming existing to .orig
+  
+THEN RECOMPILE ia-hadoop-tools with:
+   mvn package
+
+Make sure to first compile ia-web-commons, then ia-hadoop-tools.
+
+
+The modifications made to the above 2 files are as follows:
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+1. ia-web-commons/src/main/java/org/archive/extract/WATExtractorOutput.java
+
+[diff src/main/java/org/archive/extract/WATExtractorOutput.orig src/main/java/org/archive/extract/WATExtractorOutput.java]
+
+162,163c162,163
+< 			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Filename");
+< 		} else {
+---
+> 			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Warcinfo-ID");
+> 		} else {
+
+
+2. ia-hadoop-tools/src/main/java/org/archive/server/GZRangeClient.java
+
+[diff src/main/java/org/archive/server/GZRangeClient.orig src/main/java/org/archive/server/GZRangeClient.java]
+
+76,83c76,82
+< 		"WARC/1.0\r\n" +
+< 		"WARC-Type: warcinfo\r\n" +
+< 		"WARC-Date: %s\r\n" +
+< 		"WARC-Filename: %s\r\n" +
+< 		"WARC-Record-ID: <urn:uuid:%s>\r\n" +
+< 		"Content-Type: application/warc-fields\r\n" +
+< 		"Content-Length: %d\r\n\r\n";
+< 
+---
+> 	    "WARC/1.0\r\n" +
+> 	    "Content-Type: application/warc-fields\r\n" +
+> 	    "WARC-Type: warcinfo\r\n" +
+> 	    "WARC-Warcinfo-ID: <urn:uuid:%s>\r\n" +
+> 	    "Content-Length: %d\r\n\r\n" +
+> 	    "WARC-Record-ID: <urn:uuid:%s>\r\n" +	    
+> 	    "WARC-Date: %s\r\n";
+115,119c114,119
+< 	private static String DEFAULT_WARC_PATTERN = "software: %s Extractor\r\n" +
+< 	"format: WARC File Format 1.0\r\n" +
+< 	"conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf\r\n" +
+< 	"publisher: Internet Archive\r\n" +
+< 	"created: %s\r\n\r\n";
+---
+> 	private static String DEFAULT_WARC_PATTERN = "Software: crawl/1.0\r\n" +
+> 	"Format: WARC File Format 1.0\r\n" +
+> 	    "Conformsto: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf\r\n\r\n";
+>     // +
+>     //"publisher: Internet Archive\r\n" +
+>     //"created: %s\r\n\r\n";
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+
+3. To run WARC to WET, the warc needs to live on hdfs in a warc folder and there should be wet and wat folders at the same level.
+
+For example, assume that running Autistici's crawl generated $GOPATH/bin/crawl.warc.gz
+(default location and filename unless you pass flags to crawl CLI to control these)
+
+a. Ensure you get crawl.warc.gz onto the vagrant VM that has the git projects for WARC-to-WET installed, recompiled with the above modifications.
+
+b. Now, create the folder structure needed for warc-to-wet conversion:
+   hdfs dfs -mkdir /user/vagrant/warctest
+   hdfs dfs -mkdir /user/vagrant/warctest/warc
+   hdfs dfs -mkdir /user/vagrant/warctest/wet
+   hdfs dfs -mkdir /user/vagrant/warctest/wat
+
+c. Put crawl.warc.gz into the warc folder on hfds:
+   hdfs dfs -put crawl.warc.gz /user/vagrant/warctest/warc/.
+
+d. Finally, time to run the actual warc-to-wet conversion from ia-hadoop-tools:
+   cd ia-hadoop-tools
+   WARC_FOLDER=/user/vagrant/warctest/warc
+   $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz $WARC_FOLDER/crawl*.warc.gz
+
+More meaningful when the WARC_FOLDER contains multiple *.warc.gz files,
+as the above will use map-reduce to generate a *.warc.wet.gz file in the output wet folder for each input *.warc.gz file.
+
+e. Copy the generated wet files across from /user/vagrant/warctest/wet/:
+
+   (cd /vagrant or else
+   cd /home/vagrant
+   )
+   hdfs dfs -get /user/vagrant/warctest/wet/crawl.warc.wet.gz .
+
+or, when dealing with multiple input warc files, we'll have multiple wet files:
+    hdfs dfs -get /user/vagrant/warctest/wet/*.warc.wet.gz
+
+
+f. Now can view the contents of the WET files to confirm they are what we want:
+   gunzip crawl.warc.wet.gz
+   zless crawl.warc.wet
+
+The wet file contents should look good now: the web pages as WET records without html tags.
+
+
+----------------------------------------------------
+I. Setting up Nutch v2 on its own Vagrant VM machine
+----------------------------------------------------
+1. Untar vagrant-for-nutch2.tar.gz
+2. Follow the instructions below. A copy is also in vagrant-for-nutch2/GS_README.txt
+
+---
+REASONING FOR THE NUTCH v2 SPECIFIC VAGRANT VM:
+---
+We were able to get nutch v1 working on a regular machine.
+
+From a few pages online starting with https://stackoverflow.com/questions/33354460/nutch-clone-website, it appeared that "./bin/nutch fetch -all" was the nutch command to mirror a web site. Nutch v2 introduced the -all flag to the ./bin/nutch fetch command. And nutch v2 required HBase which presupposes hadoop.
+
+Our vagrant VM for commoncrawl had an incompatible version of HBase but this version was needed for that VM's version of hadoop and spark. So Dr Bainbridge came up with the idea of having a separate Vagrant VM for Nutch v2 which would have the version of HBase it needed and a Hadoop version matching that. Compatible versions with nutch 2.3.1 are mentioned at https://waue0920.wordpress.com/2016/08/25/nutch-2-3-1-hbase-0-98-hadoop-2-5-solr-4-10-3/
+(Another option was MongoDB instead of HBase, https://lobster1234.github.io/2017/08/14/search-with-nutch-mongodb-solr/, but that was not even covered in the apache nutch 2 installation guide.)
+
+---
+ Vagrant VM for Nutch2
+---
+This vagrant virtual machine is based on https://github.com/martinprobson/vagrant-hadoop-hive-spark
+
+However:
+- It comes with the older versions of hadoop 2.5.2 and hbase 0.98.21, and no spark or hive or other packages.
+- the VM is called node2 with IP 10.211.55.102 (instead of node1 with IP 10.211.55.101)
+- Since not all packages are installed, fewer ports needed forwarding. And they're forwarded to portnumber+2 to not conflict with any vagrant VM that used the original vagrant image's forwarded port numbers.
+- scripts/common.sh uses HBASE_ARCHIVE of the form "hbase-${HBASE_VERSION}-hadoop2-bin.tar.gz" (the -hadoop2 suffix is specific to v0.98.21)
+- and hbase gets installed as /usr/local/hbase-$HBASE_VERSION-hadoop2 (again with the additional -hadoop2 suffix specific to v0.98.21) in scripts/setup-hbase.sh, so the symbolic link creation there needed to refer to a path of this form.
+
+INSTRUCTIONS:
+a. mostly follow the "Getting Started" instructions at https://github.com/martinprobson/vagrant-hadoop-hive-spark
+b. but after step 3, replace the github cloned Vagrantfile, scripts and resources folders with their modified counterparts included in the zip file that can be downloaded by visiting http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/vagrant-for-nutch2.tar.gz.
+c. wherever the rest of that git page refers to "node1", IP "10.211.55.101" and specific port numbers, use instead "node2", IP "10.211.55.102" and the forwarded port numbers in the customised Vagrantfile.
+If there's already a node2/if IP "10.211.55.102" vagrant VM set up, then adjust all files in the git cloned vagrant vm already modified by the contents of this vagrant-for-nutch2 folder as follows:
+- increment all occurrences of node2 and "10.211.55.102" to node3 and IP "10.211.55.103", if not already taken, and
+- in the Vagrantfile increment forwarded ports by another 2 or so from the highest port number values already in use by other vagrant VMs.
+d. After doing "vagrant up --provider=virtualbox" to create the VM, do "vagrant ssh" or "vagrant ssh node<#>" (e.g. "vagrant ssh node2") for step 8 in the "Getting Started" section.
+e. Inside the VM, install emacs, maven, firefox:
+
+   sudo apt-get install emacs
+
+   sudo apt update
+   sudo apt install maven
+
+   sudo apt-get -y install firefox
+
+f. We set up nutch 2.3.1, which can be downloaded from https://archive.apache.org/dist/nutch/2.3.1/, as that version worked as per the nutch2 tutorial instructions with the configuration of specific versions of hadoop, hbase and gora for the vagrant VM described here.
+
+After untarring the nutch 2.3.1 source tarball,
+  1. move apache-nutch-2.3.1/conf/regex-urlfilter.txt to apache-nutch-2.3.1/conf/regex-urlfilter.txt.orig
+  2. download the two files nutch-site.xml and regex-urlfilter.GS_TEMPLATE from http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/conf
+and put them into the apache-nutch-2.3.1/conf folder.
+  3. Then continue following the nutch tutorial 2 instructions at https://cwiki.apache.org/confluence/display/NUTCH/Nutch2Tutorial to set up nutch2 (and apache-solr, if needed, but I didn't install apache-solr for nutch v2).
+     - nutch-site.xml has already been configured to do as much optimisation and speeding up of the crawling as we know about concerning nutch
+     - for each site that will be crawled, regex-urlfilter.GS_TEMPLATE will get copied as the live "regex-urlfilter.txt" file, and lines of regex filters will be appended to its end.
+
+------------------------------------------------------------------------
+J. Automated crawling with Nutch v2.3.1 and post-processing
+------------------------------------------------------------------------
+1. When you're ready to start crawling with Nutch 2.3.1,
+- copy the batchcrawl.sh file (from http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts) into the vagrant machine at top level. Make the script executable.
+- copy the to_crawl.tar.gz file containing the "to_crawl" folder (generated by CCWETProcessor.java running of the common-crawl downloaded data where MRI was the primary language) and put it into the vagrant machine at toplevel.
+- run batchcrawl.sh on a site or range of sites not yet crawled, e.g.
+    ./batchcrawl.sh 00485-00500
+
+2. When crawling is done, the above will have generated the "crawled" folder containing a subfolder for each of the crawled sites, e.g. subfolders 00485 to 00500. Each crawled site folder will contain a dump.txt with the text output of the site's web pages. The "crawled" folder with site subfolders each containing a dump.txt file can be processed with NutchTextDumpProcessor.java.
+
+
+--------------------------------------------------------
+K. Reading data from hbase tables and backing up hbase
+--------------------------------------------------------
+
+* Backing up HBase database:
+https://blogs.msdn.microsoft.com/data_otaku/2016/12/21/working-with-the-hbase-import-and-export-utility/
+
+* From an image at http://dwgeek.com/read-hbase-table-using-hbase-shell-get-command.html/
+to see the contents of a table, inside hbase shell, type:
+
+   scan 'tablename'
+
+e.g. scan '01066_webpage' and hit enter.
+
+
+To list tables and see their "column families" (I don't yet understand what this is):
+
+hbase shell
+hbase(main):001:0> list
+
+hbase(main):002:0> describe '01066_webpage'
+Table 01066_webpage is ENABLED                                                                                                                                                                              
+01066_webpage                                                                                                                                                                                               
+COLUMN FAMILIES DESCRIPTION                                                                                                                                                                                 
+{NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCK
+CACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                            
+{NAME => 'h', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCK
+CACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                            
+{NAME => 'il', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
+KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                           
+{NAME => 'mk', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
+KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                           
+{NAME => 'mtdt', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BL
+OCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                         
+{NAME => 'ol', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
+KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                           
+{NAME => 'p', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCK
+CACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                            
+{NAME => 's', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCK
+CACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                                                                                            
+8 row(s) in 0.1180 seconds
+
+
+-----------------------EOF------------------------
+
Index: /other-projects/maori-lang-detection/hdfs-cc-work/conf/ia-hadoop-tools-pom.xml
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/conf/ia-hadoop-tools-pom.xml	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/conf/ia-hadoop-tools-pom.xml	(revision 33635)
@@ -0,0 +1,173 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>org.archive</groupId>
+  <artifactId>ia-hadoop-tools</artifactId>
+  <version>1.0-SNAPSHOT</version>
+  <packaging>jar</packaging>
+
+  <name>ia-hadoop-tools</name>
+  <url>http://maven.apache.org</url>
+
+  <properties>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+  </properties>
+
+  <dependencies>
+    <!-- START GS TEAM INSERT -->
+    <dependency>
+      <groupId>org.json</groupId>
+      <artifactId>json</artifactId>
+      <version>20131018</version>
+    </dependency>
+    <!-- END GS TEAM INSERT -->
+    <dependency>
+      <groupId>junit</groupId>
+      <artifactId>junit</artifactId>
+      <version>4.10</version>
+      <type>jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.commoncrawl</groupId>
+      <artifactId>ia-web-commons</artifactId>
+      <version>1.1.9-SNAPSHOT</version>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.archive.access-control</groupId>
+      <artifactId>access-control</artifactId>
+      <version>0.1.0-SNAPSHOT</version>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.pig</groupId>
+      <artifactId>pig</artifactId>
+      <version>0.10.0</version>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.mortbay.jetty</groupId>
+      <artifactId>jetty</artifactId>
+      <version>6.1.26</version>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.httpcomponents</groupId>
+      <artifactId>httpclient</artifactId>
+      <version>4.1.3</version>
+      <type>jar</type>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.hbase</groupId>
+      <artifactId>hbase</artifactId>
+      <version>0.90.5</version>
+      <type>jar</type>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.cassandra</groupId>
+      <artifactId>cassandra-all</artifactId>
+      <version>2.0.2</version>
+    </dependency>
+    <dependency>
+      <groupId>com.datastax.cassandra</groupId>
+      <artifactId>cassandra-driver-core</artifactId>
+      <version>2.0.0-rc1</version>
+    </dependency>
+    <dependency>
+      <groupId>org.slf4j</groupId>
+      <artifactId>slf4j-log4j12</artifactId>
+      <version>1.7.5</version>
+    </dependency>
+    <dependency>
+      <groupId>commons-collections</groupId>
+      <artifactId>commons-collections</artifactId>
+      <version>3.1</version>
+      <scope>compile</scope>
+    </dependency>
+
+  </dependencies>
+  <build>
+    <pluginManagement>
+      <plugins>
+        <plugin>
+          <groupId>org.sonatype.plugins</groupId>
+          <artifactId>jarjar-maven-plugin</artifactId>
+          <version>1.5</version>
+        </plugin>
+      </plugins>
+    </pluginManagement>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-compiler-plugin</artifactId>
+        <version>2.3.2</version>
+        <configuration>
+          <source>1.6</source>
+          <target>1.6</target>
+        </configuration>
+      </plugin>
+      <plugin>
+        <artifactId>maven-assembly-plugin</artifactId>
+        <version>2.4</version>
+        <configuration>
+
+          <descriptorRefs>
+            <descriptorRef>jar-with-dependencies</descriptorRef>
+          </descriptorRefs>
+
+          <finalName>ia-hadoop-tools</finalName>
+          <archive>
+            <manifest>
+              <mainClass>org.archive.hadoop.jobs.JobDriver</mainClass>
+            </manifest>
+          </archive>
+        </configuration>
+        <executions>
+          <execution>
+            <phase>package</phase>
+            <goals>
+              <goal>single</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+    <resources>
+      <resource>
+        <directory>src/test/resources</directory>
+        <filtering>true</filtering>
+      </resource>
+    </resources>
+  </build>
+  <repositories>
+    <repository>
+      <id>internetarchive</id>
+      <name>Internet Archive Maven Repository</name>
+      <url>http://builds.archive.org:8080/maven2</url>
+      <layout>default</layout>
+
+      <releases>
+        <enabled>true</enabled>
+        <updatePolicy>daily</updatePolicy>
+        <checksumPolicy>warn</checksumPolicy>
+      </releases>
+      <snapshots>
+        <enabled>true</enabled>
+        <updatePolicy>daily</updatePolicy>
+        <checksumPolicy>fail</checksumPolicy>
+      </snapshots>
+    </repository>
+  </repositories>
+
+  <distributionManagement>
+    <repository>
+      <id>repository</id>
+      <!--Pass as command-line system property to maven -->
+      <url>${repository.url}</url>
+    </repository>
+  </distributionManagement>
+
+</project>
Index: /other-projects/maori-lang-detection/hdfs-cc-work/conf/nutch-site.xml
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/conf/nutch-site.xml	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/conf/nutch-site.xml	(revision 33635)
@@ -0,0 +1,119 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements.  See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+
+  <property>
+    <name>http.agent.name</name>
+    <value>GreenstoneTeam</value>
+  </property>
+
+  <property>
+    <name>storage.data.store.class</name>
+    <value>org.apache.gora.hbase.store.HBaseStore</value>
+    <description>Default class for storing data</description>
+  </property>
+  <!-- GS NOTE:
+       Refer to https://lucene.472066.n3.nabble.com/protocol-http-or-protocol-httpclient-td4262473.html
+       Plugin protocol-http now supports https, ssl too. Whatever problems may still exist with protocol-httpclient (reco
+       mmended below for https, back when it had more problems than at present) also exist in protocol-http.
+       In short, there's no need to turn on protocol-httpclient for https, as protocol-http works (equally well) for http
+       s pages too.
+  -->
+  <property>
+    <name>plugin.includes</name>
+    <value>protocol-http|urlfilter-(regex|validator)|parse-(html|tika)|index-(basic|anchor)|indexer-solr|scoring-opic|url
+    normalizer-(pass|regex|basic)</value>
+    <description>Regular expression naming plugin directory names to
+    include.  Any plugin not matching this expression is excluded.
+    In any case you need at least include the nutch-extensionpoints plugin. By
+    default Nutch includes crawling just HTML and plain text via HTTP,
+    and basic indexing and search plugins. In order to use HTTPS please enable
+    protocol-httpclient, but be aware of possible intermittent problems with the
+    underlying commons-httpclient library. Set parsefilter-naivebayes for classification based focused crawler.
+    </description>
+  </property>
+
+
+  <!-- https://lucene.472066.n3.nabble.com/Content-of-size-X-was-truncated-to-Y-td4003517.html -->
+  <!--
+      <property>
+      <name>parser.skip.truncated</name>
+      <value>false</value>
+      <description>Boolean value for whether we should skip parsing for truncated documents. By default this
+      property is activated due to extremely high levels of CPU which parsing can sometimes take.
+      </description>
+      </property>
+        -->  
+  
+  <property>
+    <name>http.content.limit</name>
+    <value>-1</value>
+    <description>The length limit for downloaded content using the http://
+    protocol, in bytes. If this value is nonnegative (>=0), content longer
+    than it will be truncated; otherwise, no truncation at all. Do not
+    confuse this setting with the file.content.limit setting.
+    </description>
+  </property>
+
+  <!--https://stackoverflow.com/questions/4871972/how-to-speed-up-crawling-in-nutch-->
+  <!--
+      <property>
+      <name>fetcher.threads.per.queue</name>
+      <value>1</value>
+      <description>This number is the maximum number of threads that
+      should be allowed to access a queue at one time. Setting it to
+      a value > 1 will cause the Crawl-Delay value from robots.txt to
+      be ignored and the value of fetcher.server.min.delay to be used
+      as a delay between successive requests to the same server instead
+      of fetcher.server.delay.
+      </description>
+      </property>
+  -->
+  <property>
+    <name>generate.max.count</name>
+    <value>50</value>
+    <description>The maximum number of urls in a single
+    fetchlist.  -1 if unlimited. The urls are counted according
+    to the value of the parameter generator.count.mode.
+    </description>
+  </property>
+  <property>
+    <name>fetcher.throughput.threshold.pages</name>
+    <value>1</value>
+    <description>The threshold of minimum pages per second. If the fetcher downloads less
+    pages per second than the configured threshold, the fetcher stops, preventing slow queue's
+    from stalling the throughput. This threshold must be an integer. This can be useful when
+    fetcher.timelimit.mins is hard to determine. The default value of -1 disables this check.
+    </description>
+  </property>
+  <property>
+    <name>fetcher.server.delay</name>
+    <value>0.5</value>
+    <description>The number of seconds the fetcher will delay between
+    successive requests to the same server. Note that this might get
+    overriden by a Crawl-Delay from a robots.txt and is used ONLY if
+    fetcher.threads.per.queue is set to 1.
+    </description>
+  </property>
+  
+
+</configuration>
Index: /other-projects/maori-lang-detection/hdfs-cc-work/conf/regex-urlfilter.GS_TEMPLATE
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/conf/regex-urlfilter.GS_TEMPLATE	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/conf/regex-urlfilter.GS_TEMPLATE	(revision 33635)
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# The default url filter.
+# Better for whole-internet crawling.
+
+# Each non-comment, non-blank line contains a regular expression
+# prefixed by '+' or '-'.  The first matching pattern in the file
+# determines whether a URL is included or ignored.  If no pattern
+# matches, the URL is ignored.
+
+# skip file: ftp: and mailto: urls
+-^(file|ftp|mailto):
+
+# skip image and other suffixes we can't yet parse
+# for a more extensive coverage use the urlfilter-suffix plugin
+-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
+
+# skip URLs containing certain characters as probable queries, etc.
+-[?*!@=]
+
+# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
+-.*(/[^/]+)/[^/]+\1/[^/]+\1/
+
+# accept anything else
+#+.
+
+# batchcrawl.sh will automatically append regex url filters below for each site to be crawled
Index: /other-projects/maori-lang-detection/hdfs-cc-work/conf/spark-defaults.conf.in
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/conf/spark-defaults.conf.in	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/conf/spark-defaults.conf.in	(revision 33635)
@@ -0,0 +1,28 @@
+# Default system properties included when running spark-submit.
+# This is useful for setting default environmental settings.
+
+# Example:
+# spark.master                     spark://master:7077
+# spark.eventLog.enabled           true
+# spark.eventLog.dir               hdfs://namenode:8021/directory
+# spark.serializer                 org.apache.spark.serializer.KryoSerializer
+# spark.driver.memory              5g
+# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
+
+#spark.eventLog.dir=hdfs://10.211.55.101/user/spark/applicationHistory
+#spark.eventLog.dir               hdfs://node1:8021/user/spark/applicationHistory
+spark.eventLog.dir               hdfs://node1/user/spark/applicationHistory
+#spark.eventLog.dir               hdfs:///user/spark/applicationHistory
+spark.yarn.historyServer.address=10.211.55.101:18080
+spark.eventLog.enabled=true
+spark.yarn.archive	hdfs://node1/user/spark/spark-libs.jar
+
+
+## -- START GS TEAM INSERT --- ##
+# Config settings and authentication details
+# for the created IAM role for s3a data that uses the created commoncrawl policy
+# Edit the access and secret keys below:
+spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
+spark.hadoop.fs.s3a.access.key TYPE_AWS_IAM_ROLE_ACCESSKEY_HERE
+spark.hadoop.fs.s3a.secret.key TYPE_AWS_IAM_ROLE_SECRETKEY_HERE
+## -- END GS TEAM INSERT --- ##
Index: /other-projects/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java	(revision 33635)
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.commoncrawl.spark.examples;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.Text;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.commoncrawl.spark.util.WarcFileOutputFormat;
+import org.jets3t.service.S3Service;
+import org.jets3t.service.ServiceException;
+import org.jets3t.service.impl.rest.httpclient.RestS3Service;
+import org.jets3t.service.model.S3Object;
+import org.jets3t.service.utils.ServiceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class CCIndexWarcExport extends CCIndexExport {
+
+	private static final Logger LOG = LoggerFactory.getLogger(CCIndexExport.class);
+
+	protected static final String COMMON_CRAWL_BUCKET = "commoncrawl";
+
+	protected int numRecordsPerWarcFile = 10000;
+	protected String warcPrefix = "COMMON-CRAWL-EXPORT";
+	protected String warcCreator;
+	protected String warcOperator;
+	protected String csvQueryResult;
+
+	protected CCIndexWarcExport() {
+		super();
+	}
+
+	@Override
+	protected void addOptions() {
+		options.getOption("query")
+				.setDescription("SQL query to select rows. Note: the result is required to contain the columns `url', "
+						+ "`warc_filename', `warc_record_offset' and `warc_record_length', make sure they're SELECTed.");
+		options.addOption(new Option(null, "csv", true, "CSV file to load WARC records by filename, offset and length."
+				+ "The CSV file must have column headers and the input columns `url', `warc_filename', "
+				+ "`warc_record_offset' and `warc_record_length' are mandatory, see also option --query. "));
+
+		options.addOption(
+				new Option(null, "numOutputPartitions", true, "repartition data to have <n> output partitions"));
+		options.addOption(new Option(null, "numRecordsPerWarcFile", true, "allow max. <n> records per WARC file. "
+				+ "This will repartition the data so that in average one partition contains not more than <n> rows. "
+				+ "Default is 10000, set to -1 to disable this option."
+				+ "\nNote: if both --numOutputPartitions and --numRecordsPerWarcFile are used, the former defines "
+				+ "the minimum number of partitions, the latter the maximum partition size."));
+
+		options.addOption(new Option(null, "warcPrefix", true, "WARC filename prefix"));
+		options.addOption(new Option(null, "warcCreator", true, "(WARC info record) creator of WARC export"));
+		options.addOption(new Option(null, "warcOperator", true, "(WARC info record) operator of WARC export"));
+	}
+
+	protected int parseOptions(String[] args, List<String> arguments) {
+
+		CommandLineParser parser = new PosixParser();
+		CommandLine cli;
+
+		try {
+			cli = parser.parse(options, args);
+			if (cli.hasOption("numRecordsPerWarcFile")) {
+				numRecordsPerWarcFile = Integer.parseInt(cli.getOptionValue("numRecordsPerWarcFile"));
+			}
+			if (cli.hasOption("warcPrefix")) {
+				warcPrefix = cli.getOptionValue("warcPrefix");
+			}
+			if (cli.hasOption("warcCreator")) {
+				warcCreator = cli.getOptionValue("warcCreator");
+			}
+			if (cli.hasOption("warcOperator")) {
+				warcOperator = cli.getOptionValue("warcOperator");
+			}
+			if (cli.hasOption("csv")) {
+				if (cli.hasOption("query")) {
+					LOG.error("Options --csv and --query are mutually exclusive.");
+					return 1;
+				}
+				csvQueryResult = cli.getOptionValue("csv");
+			}
+		} catch (ParseException e) {
+			// ignore, handled in call to super.parseOptions(...)
+		}
+
+		return super.parseOptions(args, arguments);
+	}
+
+	protected static byte[] getCCWarcRecord(S3Service s3, String filename, int offset, int length) {
+		long from = offset;
+		long to = offset + length - 1;
+		S3Object s3obj = null;
+		try {
+			s3obj = s3.getObject(COMMON_CRAWL_BUCKET, filename, null, null, null, null, from, to);
+			byte[] bytes = ServiceUtils.readInputStreamToBytes(s3obj.getDataInputStream());
+			s3obj.closeDataInputStream();
+			return bytes;
+		} catch (IOException | ServiceException e) {
+			LOG.error("Failed to fetch s3://{}/{} (bytes = {}-{}): {}", COMMON_CRAWL_BUCKET, filename, from, to, e);
+		} finally {
+			if (s3obj != null) {
+				try {
+					s3obj.closeDataInputStream();
+				} catch (IOException e) {
+				}
+			}
+		}
+		return null;
+	}
+
+	@Override
+	protected int run(String tablePath, String outputPath) {
+
+		Dataset<Row> sqlDF;
+		if (csvQueryResult != null) {
+		    // ************* START GS TEAM CHANGES 1 ************ //
+		        //sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+			sqlDF = sparkSession.read().format("csv").option("header", false).option("inferSchema", true)
+					.load(csvQueryResult);
+		    // ************** END GS TEAM CHANGES 1 ************ //
+		} else {
+			loadTable(sparkSession, tablePath, tableName);
+			sqlDF = executeQuery(sparkSession, sqlQuery);
+		}
+		sqlDF.persist();
+
+		long numRows = sqlDF.count();
+		LOG.info("Number of records/rows matched by query: {}", numRows);
+		if (numRecordsPerWarcFile > 0) {
+			int n = 1 + (int) (numRows / numRecordsPerWarcFile);
+			if (n > numOutputPartitions) {
+				numOutputPartitions = n;
+				LOG.info("Distributing {} records to {} output partitions (max. {} records per WARC file)", numRows,
+						numOutputPartitions, numRecordsPerWarcFile);
+			} else {
+				// more output partitions requested
+				LOG.info("Distributing {} records to {} output partitions", numRows, numOutputPartitions);
+			}
+		}
+		if (numOutputPartitions > 0) {
+			LOG.info("Repartitioning data to {} output partitions", numOutputPartitions);
+			sqlDF = sqlDF.repartition(numOutputPartitions);
+		}
+
+		// ************* START GS TEAM CHANGES 2 ************ //
+		//JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+		JavaRDD<Row> rdd = sqlDF.select("_c0", "_c1", "_c2", "_c3").rdd()
+				.toJavaRDD();
+		// ************** END GS TEAM CHANGES 2 ************* //
+
+		// fetch WARC content from s3://commoncrawl/ and map to paired RDD
+		//   <Text url, byte[] warc_record>
+		JavaPairRDD<Text,byte[]> res = rdd.mapPartitionsToPair((Iterator<Row> rows) -> {
+			ArrayList<scala.Tuple2<Text, byte[]>> reslist = new ArrayList<>();
+			S3Service s3 = new RestS3Service(null);
+			while (rows.hasNext()) {
+				Row row = rows.next();
+				String url = row.getString(0);
+				String filename = row.getString(1);
+				int offset = row.getInt(2);
+				int length = row.getInt(3);
+				LOG.info("Fetching WARC record {} {} {} for {}", filename, offset, length, url);
+				byte[] bytes = getCCWarcRecord(s3, filename, offset, length);
+				if (bytes != null) {
+					reslist.add(new scala.Tuple2<Text, byte[]>(new Text(url), bytes));
+				}
+			}
+			return reslist.iterator();
+		}, false);
+
+		// save data as WARC files
+		Configuration conf = sparkSession.sparkContext().hadoopConfiguration();
+		conf.set("warc.export.prefix", warcPrefix);
+		if (warcCreator != null) {
+			conf.set("warc.export.creator", warcCreator);
+		}
+		if (warcOperator != null) {
+			conf.set("warc.export.operator", warcOperator);
+		}
+		conf.set("warc.export.software",
+				getClass().getCanonicalName() + " (Spark " + sparkSession.sparkContext().version() + ")");
+		conf.set("warc.export.description", "Common Crawl WARC export from " + tablePath + " for query: " + sqlQuery);
+
+		res.saveAsNewAPIHadoopFile(outputPath, String.class, byte[].class, WarcFileOutputFormat.class, conf);
+		LOG.info("Wrote {} WARC files with {} records total", numOutputPartitions, numRows);
+		sparkStats.report();
+
+		return 0;
+	}
+
+	public static void main(String[] args) throws IOException {
+		CCIndexExport job = new CCIndexWarcExport();
+		int success = job.run(args);
+		System.exit(success);
+	}
+
+}
Index: /other-projects/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java.orig
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java.orig	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/patches/CCIndexWarcExport.java.orig	(revision 33635)
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.commoncrawl.spark.examples;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.Text;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.commoncrawl.spark.util.WarcFileOutputFormat;
+import org.jets3t.service.S3Service;
+import org.jets3t.service.ServiceException;
+import org.jets3t.service.impl.rest.httpclient.RestS3Service;
+import org.jets3t.service.model.S3Object;
+import org.jets3t.service.utils.ServiceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class CCIndexWarcExport extends CCIndexExport {
+
+	private static final Logger LOG = LoggerFactory.getLogger(CCIndexExport.class);
+
+	protected static final String COMMON_CRAWL_BUCKET = "commoncrawl";
+
+	protected int numRecordsPerWarcFile = 10000;
+	protected String warcPrefix = "COMMON-CRAWL-EXPORT";
+	protected String warcCreator;
+	protected String warcOperator;
+	protected String csvQueryResult;
+
+	protected CCIndexWarcExport() {
+		super();
+	}
+
+	@Override
+	protected void addOptions() {
+		options.getOption("query")
+				.setDescription("SQL query to select rows. Note: the result is required to contain the columns `url', "
+						+ "`warc_filename', `warc_record_offset' and `warc_record_length', make sure they're SELECTed.");
+		options.addOption(new Option(null, "csv", true, "CSV file to load WARC records by filename, offset and length."
+				+ "The CSV file must have column headers and the input columns `url', `warc_filename', "
+				+ "`warc_record_offset' and `warc_record_length' are mandatory, see also option --query. "));
+
+		options.addOption(
+				new Option(null, "numOutputPartitions", true, "repartition data to have <n> output partitions"));
+		options.addOption(new Option(null, "numRecordsPerWarcFile", true, "allow max. <n> records per WARC file. "
+				+ "This will repartition the data so that in average one partition contains not more than <n> rows. "
+				+ "Default is 10000, set to -1 to disable this option."
+				+ "\nNote: if both --numOutputPartitions and --numRecordsPerWarcFile are used, the former defines "
+				+ "the minimum number of partitions, the latter the maximum partition size."));
+
+		options.addOption(new Option(null, "warcPrefix", true, "WARC filename prefix"));
+		options.addOption(new Option(null, "warcCreator", true, "(WARC info record) creator of WARC export"));
+		options.addOption(new Option(null, "warcOperator", true, "(WARC info record) operator of WARC export"));
+	}
+
+	protected int parseOptions(String[] args, List<String> arguments) {
+
+		CommandLineParser parser = new PosixParser();
+		CommandLine cli;
+
+		try {
+			cli = parser.parse(options, args);
+			if (cli.hasOption("numRecordsPerWarcFile")) {
+				numRecordsPerWarcFile = Integer.parseInt(cli.getOptionValue("numRecordsPerWarcFile"));
+			}
+			if (cli.hasOption("warcPrefix")) {
+				warcPrefix = cli.getOptionValue("warcPrefix");
+			}
+			if (cli.hasOption("warcCreator")) {
+				warcCreator = cli.getOptionValue("warcCreator");
+			}
+			if (cli.hasOption("warcOperator")) {
+				warcOperator = cli.getOptionValue("warcOperator");
+			}
+			if (cli.hasOption("csv")) {
+				if (cli.hasOption("query")) {
+					LOG.error("Options --csv and --query are mutually exclusive.");
+					return 1;
+				}
+				csvQueryResult = cli.getOptionValue("csv");
+			}
+		} catch (ParseException e) {
+			// ignore, handled in call to super.parseOptions(...)
+		}
+
+		return super.parseOptions(args, arguments);
+	}
+
+	protected static byte[] getCCWarcRecord(S3Service s3, String filename, int offset, int length) {
+		long from = offset;
+		long to = offset + length - 1;
+		S3Object s3obj = null;
+		try {
+			s3obj = s3.getObject(COMMON_CRAWL_BUCKET, filename, null, null, null, null, from, to);
+			byte[] bytes = ServiceUtils.readInputStreamToBytes(s3obj.getDataInputStream());
+			s3obj.closeDataInputStream();
+			return bytes;
+		} catch (IOException | ServiceException e) {
+			LOG.error("Failed to fetch s3://{}/{} (bytes = {}-{}): {}", COMMON_CRAWL_BUCKET, filename, from, to, e);
+		} finally {
+			if (s3obj != null) {
+				try {
+					s3obj.closeDataInputStream();
+				} catch (IOException e) {
+				}
+			}
+		}
+		return null;
+	}
+
+	@Override
+	protected int run(String tablePath, String outputPath) {
+
+		Dataset<Row> sqlDF;
+		if (csvQueryResult != null) {
+			sqlDF = sparkSession.read().format("csv").option("header", true).option("inferSchema", true)
+					.load(csvQueryResult);
+		} else {
+			loadTable(sparkSession, tablePath, tableName);
+			sqlDF = executeQuery(sparkSession, sqlQuery);
+		}
+		sqlDF.persist();
+
+		long numRows = sqlDF.count();
+		LOG.info("Number of records/rows matched by query: {}", numRows);
+		if (numRecordsPerWarcFile > 0) {
+			int n = 1 + (int) (numRows / numRecordsPerWarcFile);
+			if (n > numOutputPartitions) {
+				numOutputPartitions = n;
+				LOG.info("Distributing {} records to {} output partitions (max. {} records per WARC file)", numRows,
+						numOutputPartitions, numRecordsPerWarcFile);
+			} else {
+				// more output partitions requested
+				LOG.info("Distributing {} records to {} output partitions", numRows, numOutputPartitions);
+			}
+		}
+		if (numOutputPartitions > 0) {
+			LOG.info("Repartitioning data to {} output partitions", numOutputPartitions);
+			sqlDF = sqlDF.repartition(numOutputPartitions);
+		}
+
+		JavaRDD<Row> rdd = sqlDF.select("url", "warc_filename", "warc_record_offset", "warc_record_length").rdd()
+				.toJavaRDD();
+
+		// fetch WARC content from s3://commoncrawl/ and map to paired RDD
+		//   <Text url, byte[] warc_record>
+		JavaPairRDD<Text,byte[]> res = rdd.mapPartitionsToPair((Iterator<Row> rows) -> {
+			ArrayList<scala.Tuple2<Text, byte[]>> reslist = new ArrayList<>();
+			S3Service s3 = new RestS3Service(null);
+			while (rows.hasNext()) {
+				Row row = rows.next();
+				String url = row.getString(0);
+				String filename = row.getString(1);
+				int offset = row.getInt(2);
+				int length = row.getInt(3);
+				LOG.info("Fetching WARC record {} {} {} for {}", filename, offset, length, url);
+				byte[] bytes = getCCWarcRecord(s3, filename, offset, length);
+				if (bytes != null) {
+					reslist.add(new scala.Tuple2<Text, byte[]>(new Text(url), bytes));
+				}
+			}
+			return reslist.iterator();
+		}, false);
+
+		// save data as WARC files
+		Configuration conf = sparkSession.sparkContext().hadoopConfiguration();
+		conf.set("warc.export.prefix", warcPrefix);
+		if (warcCreator != null) {
+			conf.set("warc.export.creator", warcCreator);
+		}
+		if (warcOperator != null) {
+			conf.set("warc.export.operator", warcOperator);
+		}
+		conf.set("warc.export.software",
+				getClass().getCanonicalName() + " (Spark " + sparkSession.sparkContext().version() + ")");
+		conf.set("warc.export.description", "Common Crawl WARC export from " + tablePath + " for query: " + sqlQuery);
+
+		res.saveAsNewAPIHadoopFile(outputPath, String.class, byte[].class, WarcFileOutputFormat.class, conf);
+		LOG.info("Wrote {} WARC files with {} records total", numOutputPartitions, numRows);
+		sparkStats.report();
+
+		return 0;
+	}
+
+	public static void main(String[] args) throws IOException {
+		CCIndexExport job = new CCIndexWarcExport();
+		int success = job.run(args);
+		System.exit(success);
+	}
+
+}
Index: /other-projects/maori-lang-detection/hdfs-cc-work/patches/GZRangeClient.java
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/patches/GZRangeClient.java	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/patches/GZRangeClient.java	(revision 33635)
@@ -0,0 +1,417 @@
+package org.archive.server;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.net.URLConnection;
+import java.nio.charset.Charset;
+import java.text.ParseException;
+import java.util.List;
+import java.util.UUID;
+import java.util.logging.Logger;
+import java.net.URI;
+import java.net.URISyntaxException;
+
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.archive.util.HMACSigner;
+import org.archive.format.gzip.GZIPFormatException;
+import org.archive.format.gzip.GZIPMemberSeries;
+import org.archive.format.gzip.GZIPMemberWriter;
+import org.archive.format.gzip.GZIPSeriesMember;
+import org.archive.streamcontext.SimpleStream;
+import org.archive.util.IAUtils;
+import org.archive.util.DateUtils;
+import org.archive.util.FileNameSpec;
+import com.google.common.io.ByteStreams;
+
+public class GZRangeClient {
+
+	private final static Logger LOGGER = 
+		Logger.getLogger(GZRangeClient.class.getName());
+
+	private static final Charset UTF8 = Charset.forName("UTF-8");
+	private static int CR = 13;
+	private static int LF = 10;
+	private static final long DEFAULT_MAX_ARC_SIZE = 1024 * 1024 * 100;
+	private static final long DEFAULT_MAX_WARC_SIZE = 1024 * 1024 * 1024;
+
+	private File targetDir;
+	private long maxArcSize;
+	private long maxWarcSize;
+	private String timestamp14;
+	private String timestampZ;
+	private FileNameSpec warcNamer;
+	private FileNameSpec arcNamer;
+	private File currentArc;
+	private File currentArcTmp;
+
+	private String hmacName = "";
+	private String hmacSignature = "";
+
+	protected static FileSystem hdfsSys = null;
+
+	private File currentWarc;
+	private File currentWarcTmp;
+
+	private FileOutputStream currentArcOS;
+	private long currentArcSize = 0;
+	private FileOutputStream currentWarcOS;
+	private long currentWarcSize = 0;
+	private byte[] warcHeaderContents;
+	private boolean exitOnError = false;
+
+	private final static String ARC_PATTERN = 
+		"filedesc://%s 0.0.0.0 %s text/plain 76\n" +
+		"1 0 InternetArchive\n" +
+		"URL IP-address Archive-date Content-type Archive-length\n\n";
+
+	private final static String WARC_PATTERN =
+	    "WARC/1.0\r\n" +
+	    "Content-Type: application/warc-fields\r\n" +
+	    "WARC-Type: warcinfo\r\n" +
+	    "WARC-Warcinfo-ID: <urn:uuid:%s>\r\n" +
+	    "Content-Length: %d\r\n\r\n" +
+	    "WARC-Record-ID: <urn:uuid:%s>\r\n" +	    
+	    "WARC-Date: %s\r\n";
+
+	/*
+filedesc://IQ-125-20061126082604-03075-crawling08.us.archive.org.arc 0.0.0.0 20061126082604 text/plain 1447
+1 1 InternetArchive
+URL IP-address Archive-date Content-type Archive-length
+
+
+
+WARC-Date: 2009-10-10T21:33:10Z
+WARC-Filename: LOC-WEEKLY-008-20091010213310-06162-crawling110.us.archive.org.warc.gz
+WARC-Record-ID: <urn:uuid:776a760b-f456-48f1-97e3-1d29967c75d2>
+Content-Type: application/warc-fields
+Content-Length: 599
+
+software: Heritrix/1.15.4 http://crawler.archive.org
+ip: 207.241.235.29
+hostname: crawling110.us.archive.org
+format: WARC File Format 0.17
+conformsTo: http://crawler.archive.org/warc/0.17/WARC0.17ISO.doc
+operator: Vinay Goel
+publisher: Internet Archive
+audience: Library of Congress
+isPartOf: LOC-WEEKLY-008-RECOVER
+created: 2009-09-30T12:21:49Z
+description: Library of Congress Monthly Harvest
+robots: ignore
+http-header-user-agent: Mozilla/5.0 (compatible; archive.org_bot/1.5.0 +http://www.loc.gov/minerva/crawl.html)
+http-header-from: archive-crawler-agent@lists.sourceforge.net
+
+	 */
+
+	private static String DEFAULT_WARC_PATTERN = "Software: crawl/1.0\r\n" +
+	"Format: WARC File Format 1.0\r\n" +
+	    "Conformsto: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf\r\n\r\n";
+    // +
+    //"publisher: Internet Archive\r\n" +
+    //"created: %s\r\n\r\n";
+
+	private static final String defaultWarcHeaderString = String.format(
+			DEFAULT_WARC_PATTERN, 
+			IAUtils.COMMONS_VERSION, 
+			DateUtils.getLog17Date(System.currentTimeMillis()));
+
+	private static final byte[] DEFAULT_WARC_HEADER_BYTES = 
+		defaultWarcHeaderString.getBytes(UTF8);
+
+	public GZRangeClient(File targetDir, String prefix, String timestamp14)
+	throws ParseException {
+
+		this.targetDir = targetDir;
+		arcNamer = new FileNameSpec(prefix, ".arc.gz");
+		warcNamer = new FileNameSpec(prefix, ".warc.gz");
+		this.timestamp14 = timestamp14;
+		long msse = DateUtils.parse14DigitDate(timestamp14).getTime();
+		this.timestampZ = DateUtils.getLog17Date(msse);
+		maxArcSize = DEFAULT_MAX_ARC_SIZE;
+		maxWarcSize = DEFAULT_MAX_WARC_SIZE;
+		warcHeaderContents = DEFAULT_WARC_HEADER_BYTES;
+	}
+	
+	public GZRangeClient(File targetDir, String prefix, String timestamp14, String hmacName, String hmacSignature)
+	throws ParseException {
+
+		this.targetDir = targetDir;
+		arcNamer = new FileNameSpec(prefix, ".arc.gz");
+		warcNamer = new FileNameSpec(prefix, ".warc.gz");
+		this.timestamp14 = timestamp14;
+		long msse = DateUtils.parse14DigitDate(timestamp14).getTime();
+		this.timestampZ = DateUtils.getLog17Date(msse);
+		maxArcSize = DEFAULT_MAX_ARC_SIZE;
+		maxWarcSize = DEFAULT_MAX_WARC_SIZE;
+		warcHeaderContents = DEFAULT_WARC_HEADER_BYTES;
+		this.hmacName = hmacName;
+		this.hmacSignature = hmacSignature;
+	}
+
+	public void finish() throws IOException {
+		closeArc();
+		closeWarc();
+	}
+
+	private long getGZLength(InputStream is) 
+	throws IOException, GZIPFormatException {
+
+		SimpleStream s = new SimpleStream(is);
+		GZIPMemberSeries gzs = new GZIPMemberSeries(s,"range",0,true);
+		GZIPSeriesMember m = gzs.getNextMember();
+		m.skipMember();
+		return m.getCompressedBytesRead();
+	}
+
+	public void append(long offset, List<String> urls) throws IOException {
+		boolean isArc = false;
+		String first = urls.get(0);
+		if(first.endsWith(".arc.gz")) {
+			isArc = true;
+		} else if(first.endsWith(".warc.gz")) {
+
+		} else {
+			throw new IOException("URL (" + first +
+					") must end with '.arc.gz' or '.warc.gz'");
+		}
+		
+		HMACSigner signer = null;
+		if(hmacName != null && hmacSignature != null && !hmacName.isEmpty() && !hmacSignature.isEmpty())
+			signer = new HMACSigner(hmacSignature, hmacName);
+		
+		for(String url : urls) {
+			FileBackedInputStream fbis = null;
+			InputStream is = null;
+			try {
+				if(url.startsWith("http://")) {
+					URL u = new URL(url);
+					URLConnection conn = u.openConnection();
+					conn.setRequestProperty("Range", String.format("bytes=%d-", offset));
+					if(signer != null)	
+						conn.setRequestProperty("Cookie", signer.getHMacCookieStr(1000));
+					LOGGER.info(String.format("Attempting(%d) from(%s)",offset,url));
+					conn.connect();
+					is = conn.getInputStream();
+				} else if(url.startsWith("hdfs://")){
+					URI u = new URI(url);
+					//only initialize the FS once
+					if (hdfsSys == null) {
+						Configuration conf = new Configuration();
+						URI defaultURI = new URI(u.getScheme() + "://" + u.getHost() + ":"+ u.getPort() + "/");
+						hdfsSys = FileSystem.get(defaultURI, conf);
+					}
+					Path path = new Path(u.getPath());
+					FSDataInputStream fis = hdfsSys.open(path);
+					fis.seek(offset);
+					is = fis;
+				}
+				fbis = new FileBackedInputStream(is);
+				long length = getGZLength(fbis);
+				InputStream orig = fbis.getInputStream();
+				if(isArc) {
+					writeARCRecord(orig, length);
+				} else {
+					writeWARCRecord(orig, length);					
+				}
+				LOGGER.info(String.format("Wrote record(%d) from(%s)",
+						offset,url));
+				return;
+			} catch (IOException e) {
+				LOGGER.warning("FAILED URL-OFFSET("+url+")(" + offset+")");
+			} catch (URISyntaxException e) { 
+				LOGGER.warning("FAILED URL-OFFSET("+url+")(" + offset+")");
+			} finally {
+				if(is != null) {
+					is.close();
+				}
+				if(fbis != null) {
+					fbis.resetBacker();
+				}
+			}
+		}
+		StringBuilder sb = new StringBuilder();
+		for(String u : urls) {
+			if(sb.length() != 0) {
+				sb.append(",");
+			}
+			sb.append(u);
+		}
+		String errMsg = String.format("Unable to get offset(%d) from (%s)",
+				offset,sb.toString());
+		if(exitOnError) {
+			throw new IOException(errMsg);
+		} else {
+			LOGGER.severe(errMsg);
+		}
+	}
+
+
+
+	private String getWARCRecordID() {
+		return "urn:uuid:" + UUID.randomUUID().toString();
+	}
+	private byte[] getARCHeader(String name) {
+		return String.format(ARC_PATTERN,name,timestamp14).getBytes(UTF8);
+	}
+	private byte[] getWARCHeader(String name) {
+		String t = String.format(WARC_PATTERN,
+				timestampZ,name,getWARCRecordID(),warcHeaderContents.length + 4);
+		byte[] b = t.getBytes(UTF8);
+		ByteArrayOutputStream baos = new ByteArrayOutputStream();
+		try {
+			baos.write(b);
+			baos.write(warcHeaderContents);
+		} catch(IOException e) {
+			// not gonna happen
+		}
+		baos.write(CR);
+		baos.write(LF);
+		baos.write(CR);
+		baos.write(LF);
+		return baos.toByteArray();
+	}
+
+	private void writeWARCRecord(InputStream is, long length) throws IOException {
+		if(currentWarcSize == 0) {
+			nextWarc();
+		}
+		InputStream lis = ByteStreams.limit(is, length);
+		ByteStreams.copy(lis, currentWarcOS);
+		currentWarcSize += length;
+		if(currentWarcSize > maxWarcSize) {
+			closeWarc();
+		}
+	}
+	private void writeARCRecord(InputStream is, long length) throws IOException {
+		if(currentArcSize == 0) {
+			nextArc();
+		}
+		InputStream lis = ByteStreams.limit(is, length);
+		ByteStreams.copy(lis, currentArcOS);
+		currentArcSize += length;
+		if(currentArcSize > maxArcSize) {
+			closeArc();
+		}
+	}
+
+	private void closeArc() throws IOException {
+		if(currentArcOS == null) {
+			return;
+		}
+		currentArcOS.close();
+		if(!currentArcTmp.renameTo(currentArc)) {
+			throw new IOException(String.format("Failed rename(%s) to (%s)",
+					currentArcTmp.getAbsolutePath(),
+					currentArc.getAbsolutePath()));
+		}
+		currentArcOS = null;
+		currentArcSize = 0;
+		LOGGER.info(String.format("Closed(%s)",currentArc.getAbsolutePath()));
+	}
+	private void closeWarc() throws IOException {
+		if(currentWarcOS == null) {
+			return;
+		}
+		currentWarcOS.close();
+		if(!currentWarcTmp.renameTo(currentWarc)) {
+			throw new IOException(String.format("Failed rename(%s) to (%s)",
+					currentWarcTmp.getAbsolutePath(),
+					currentWarc.getAbsolutePath()));
+		}
+		currentWarcOS = null;
+		currentWarcSize = 0;
+		LOGGER.info(String.format("Closed(%s)",currentWarc.getAbsolutePath()));
+	}
+	private void nextArc() throws IOException {
+		String newArcName = arcNamer.getNextName();
+		currentArc = new File(targetDir,newArcName);
+		String tmpArcName = newArcName + ".OPEN";
+		currentArcTmp = new File(targetDir,tmpArcName);
+		currentArcOS = new FileOutputStream(currentArcTmp);
+		byte[] header = getARCHeader(newArcName);
+		GZIPMemberWriter w = new GZIPMemberWriter(currentArcOS);
+                w.write(new ByteArrayInputStream(header));
+		currentArcSize = w.getBytesWritten();
+		LOGGER.info(String.format("Openned(%s)",currentArc.getAbsolutePath()));
+	}
+
+	private void nextWarc() throws IOException {
+		String newWarcName = warcNamer.getNextName();
+		currentWarc = new File(targetDir,newWarcName);
+		String tmpWarcName = newWarcName + ".OPEN";
+		currentWarcTmp = new File(targetDir,tmpWarcName);
+		currentWarcOS = new FileOutputStream(currentWarcTmp);
+
+		byte[] header = getWARCHeader(newWarcName);
+		GZIPMemberWriter w = new GZIPMemberWriter(currentWarcOS);
+                w.write(new ByteArrayInputStream(header));
+		currentWarcSize = w.getBytesWritten();
+		LOGGER.info(String.format("Openned(%s)",currentWarc.getAbsolutePath()));
+	}
+
+	/**
+	 * @return the warcHeaderContents
+	 */
+	public byte[] getWarcHeaderContents() {
+		return warcHeaderContents;
+	}
+
+	/**
+	 * @param warcHeaderContents the warcHeaderContents to set
+	 */
+	public void setWarcHeaderContents(byte[] warcHeaderContents) {
+		this.warcHeaderContents = warcHeaderContents;
+	}
+
+	/**
+	 * @return the maxArcSize
+	 */
+	public long getMaxArcSize() {
+		return maxArcSize;
+	}
+
+	/**
+	 * @param maxArcSize the maxArcSize to set
+	 */
+	public void setMaxArcSize(long maxArcSize) {
+		this.maxArcSize = maxArcSize;
+	}
+
+	/**
+	 * @return the maxWarcSize
+	 */
+	public long getMaxWarcSize() {
+		return maxWarcSize;
+	}
+
+	/**
+	 * @param maxWarcSize the maxWarcSize to set
+	 */
+	public void setMaxWarcSize(long maxWarcSize) {
+		this.maxWarcSize = maxWarcSize;
+	}
+
+
+	/**
+	 * @return the exitOnError
+	 */
+	public boolean isExitOnError() {
+		return exitOnError;
+	}
+
+
+	/**
+	 * @param exitOnError the exitOnError to set
+	 */
+	public void setExitOnError(boolean exitOnError) {
+		this.exitOnError = exitOnError;
+	}
+}
Index: /other-projects/maori-lang-detection/hdfs-cc-work/patches/WATExtractorOutput.java
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/patches/WATExtractorOutput.java	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/patches/WATExtractorOutput.java	(revision 33635)
@@ -0,0 +1,211 @@
+package org.archive.extract;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.nio.charset.Charset;
+import java.text.ParseException;
+import java.net.UnknownHostException;
+import java.util.Date;
+
+import org.archive.format.gzip.GZIPMemberWriter;
+import org.archive.format.gzip.GZIPMemberWriterCommittedOutputStream;
+import org.archive.format.http.HttpHeaders;
+import org.archive.format.json.JSONUtils;
+import org.archive.format.warc.WARCRecordWriter;
+import org.archive.resource.MetaData;
+import org.archive.resource.Resource;
+import org.archive.util.IAUtils;
+import org.archive.util.DateUtils;
+import org.archive.util.StreamCopy;
+import org.archive.util.io.CommitedOutputStream;
+import com.github.openjson.JSONException;
+
+import java.net.InetAddress;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+
+import java.util.logging.Logger;
+
+public class WATExtractorOutput implements ExtractorOutput {
+	WARCRecordWriter recW;
+	private boolean wroteFirst;
+	private GZIPMemberWriter gzW;
+	private static int DEFAULT_BUFFER_RAM = 1024 * 1024;
+	private int bufferRAM = DEFAULT_BUFFER_RAM;
+	private final static Charset UTF8 = Charset.forName("UTF-8");
+	private String outputFile;
+	
+	private static final Logger LOG = Logger.getLogger(WATExtractorOutput.class.getName());
+	
+	public WATExtractorOutput(OutputStream out) {
+		this(out, null);
+	}
+
+	public WATExtractorOutput(OutputStream out, String outputFile) {
+		gzW = new GZIPMemberWriter(out);
+		recW = new WARCRecordWriter();
+		wroteFirst = false;
+		this.outputFile = outputFile;
+	}
+
+	private CommitedOutputStream getOutput() {
+		return new GZIPMemberWriterCommittedOutputStream(gzW,bufferRAM);
+	}
+
+	public void output(Resource resource) throws IOException {
+		StreamCopy.readToEOF(resource.getInputStream());
+		MetaData top = resource.getMetaData().getTopMetaData();
+		CommitedOutputStream cos;
+		if(!wroteFirst) {
+			cos = getOutput();
+			writeWARCInfo(cos,top);
+			cos.commit();
+			wroteFirst = true;
+		}
+		String envelopeFormat = JSONUtils.extractSingle(top, "Envelope.Format");
+		if(envelopeFormat == null) {
+			// hrm...
+			throw new IOException("Missing Envelope.Format");
+		}
+
+		// remove the text extracts if it exists
+		JSONUtils.removeObject(top, "Envelope.Payload-Metadata.HTTP-Response-Metadata.HTML-Metadata", "Text");
+
+		cos = getOutput();
+		if(envelopeFormat.startsWith("ARC")) {
+			writeARC(cos,top);
+		} else if(envelopeFormat.startsWith("WARC")) {
+			writeWARC(cos,top);
+		} else {
+			// hrm...
+			throw new IOException("Unknown Envelope.Format");
+		}
+		cos.commit();
+	}
+
+	private void writeWARCInfo(OutputStream recOut, MetaData md) throws IOException {
+		// filename is given in the command line
+		String filename = outputFile;
+		if (filename == null || filename.length() == 0) {
+			// if no filename by command line, we construct a default filename base on container filename
+			filename = JSONUtils.extractSingle(md, "Container.Filename");
+			if (filename == null) {
+				throw new IOException("No Container.Filename...");
+			}
+			if (filename.endsWith(".warc") || filename.endsWith(".warc.gz")) {
+				filename = filename.replaceFirst("\\.warc$", ".warc.wat.gz");
+				filename = filename.replaceFirst("\\.warc\\.gz$", ".warc.wat.gz");
+			} else if (filename.endsWith(".arc") || filename.endsWith(".arc.gz")) {
+				filename = filename.replaceFirst("\\.arc$", ".arc.wat.gz");
+				filename = filename.replaceFirst("\\.arc\\.gz$", ".arc.wat.gz");
+			}
+		}
+		// removing path from filename
+		File tmpFile = new File(filename);
+		filename = tmpFile.getName();
+		HttpHeaders headers = new HttpHeaders();
+		headers.add("Software-Info", IAUtils.COMMONS_VERSION);
+		headers.addDateHeader("Extracted-Date", new Date());
+
+		// add ip, hostname
+		try {
+			InetAddress host = InetAddress.getLocalHost();
+			headers.add("ip", host.getHostAddress());
+			headers.add("hostname", host.getCanonicalHostName());
+		} catch (UnknownHostException e) {
+			LOG.warning("unable to obtain local crawl engine host :\n"+e.getMessage());
+        }
+
+		headers.add("format", IAUtils.WARC_FORMAT);
+		headers.add("conformsTo", IAUtils.WARC_FORMAT_CONFORMS_TO);
+		// optional arguments
+		if(IAUtils.OPERATOR != null && IAUtils.OPERATOR.length() > 0) {
+			headers.add("operator", IAUtils.OPERATOR);
+		}
+		if(IAUtils.PUBLISHER != null && IAUtils.PUBLISHER.length() > 0) {
+			headers.add("publisher", IAUtils.PUBLISHER);
+		}
+		if(IAUtils.WAT_WARCINFO_DESCRIPTION != null && IAUtils.WAT_WARCINFO_DESCRIPTION.length() > 0) {
+			headers.add("description", IAUtils.WAT_WARCINFO_DESCRIPTION);
+		}
+		
+		ByteArrayOutputStream baos = new ByteArrayOutputStream();
+		headers.write(baos);
+                recW.writeWARCInfoRecord(recOut,filename,baos.toByteArray());
+	}
+
+	private String extractOrIO(MetaData md, String path) throws IOException {
+		String value = JSONUtils.extractSingle(md, path);
+		if(value == null) {
+			throw new IOException("No "+path+" found.");
+		}
+		return value;
+	}
+
+	private void writeARC(OutputStream recOut, MetaData md) throws IOException {
+		String targetURI = extractOrIO(md, "Envelope.ARC-Header-Metadata.Target-URI");
+		String capDateString = extractOrIO(md, "Envelope.ARC-Header-Metadata.Date");
+		String filename = extractOrIO(md, "Container.Filename");
+		String offset = extractOrIO(md, "Container.Offset");
+		String recId = String.format("<urn:arc:%s:%s>",filename,offset);
+		writeWARCMDRecord(recOut,md,targetURI,capDateString,recId);
+	}
+
+	private void writeWARC(OutputStream recOut, MetaData md) throws IOException {
+		String warcType = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Type");
+		String targetURI;
+		if(warcType.equals("warcinfo")) {
+			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Warcinfo-ID");
+		} else {		
+			targetURI = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Target-URI");
+		}
+		// handle date of generation in WARC format
+		DateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");
+		String capDateString = dateFormat.format(new Date());
+		String recId = extractOrIO(md, "Envelope.WARC-Header-Metadata.WARC-Record-ID");
+		writeWARCMDRecord(recOut,md,targetURI,capDateString,recId);
+	}
+
+	private void writeWARCMDRecord(OutputStream recOut, MetaData md, 
+			String targetURI, String capDateString, String recId)
+	throws IOException {
+
+                ByteArrayOutputStream bos = new ByteArrayOutputStream();
+
+		OutputStreamWriter osw = new OutputStreamWriter(bos, UTF8);
+		String contents = md.toString();
+		osw.write(contents, 0, contents.length());
+		osw.flush();
+//		ByteArrayInputStream bais = new ByteArrayInputStream(md.toString().getBytes("UTF-8"));
+		Date capDate;
+		try {
+			capDate = DateUtils.getSecondsSinceEpoch(capDateString);
+
+		} catch (ParseException e) {
+			e.printStackTrace();
+			// TODO... not the write thing...
+			capDate = new Date();
+		}
+		
+		recW.writeJSONMetadataRecord(recOut, bos.toByteArray(),
+				targetURI, capDate, recId);
+	}
+
+	private static String transformWARCDate(final String input) {
+		
+		StringBuilder output = new StringBuilder(14);
+		
+		output.append(input.substring(0,4));
+		output.append(input.substring(5,7));
+		output.append(input.substring(8,10));
+		output.append(input.substring(11,13));
+		output.append(input.substring(14,16));
+		output.append(input.substring(17,19));
+		
+		return output.toString();
+	}
+}
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/GS_README
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/GS_README	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/GS_README	(revision 33635)
@@ -0,0 +1,11 @@
+1. Most of the scripts in this folder are based on https://github.com/commoncrawl/cc-index-table/tree/master/src/script/convert_url_index.sh
+
+to set the environment and setup SPARK etc, in order to run cc-index-table related tasks to process data that we're particularly interested in.
+
+Several of the modifications are further based on the instructions and examples at https://github.com/commoncrawl/cc-index-table
+
+2. The third/final phase of the script get_maori_WET_records_for_crawl.sh is based on the instructions at https://groups.google.com/forum/#!topic/common-crawl/hsb90GHq6to
+
+3. The script get_Maori_WET_records_from_CCSep2018_on.sh is merely a batch processing script.
+
+4. Script limit10_export_index.sh never really worked, but it was only scripted for testing purposes, and has no other purpose.
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/batchcrawl.sh	(revision 33635)
@@ -0,0 +1,230 @@
+#!/bin/bash
+
+sitesDir=to_crawl/sites
+echo "SITES DIR (INPUT): $sitesDir"
+outputDir=crawled
+mkdir -p $outputDir
+echo "OUTPUT DIR: $outputDir"
+
+
+NUTCH_HOME=apache-nutch-2.3.1
+NUTCH_CONF_DIR=$NUTCH_HOME/runtime/local/conf
+NUTCH_URLFILTER_TEMPLATE=$NUTCH_CONF_DIR/regex-urlfilter.GS_TEMPLATE
+NUTCH_URLFILTER_FILE=$NUTCH_CONF_DIR/regex-urlfilter.txt
+
+CRAWL_COMMAND=$NUTCH_HOME/runtime/local/bin/crawl
+NUTCH_COMMAND=$NUTCH_HOME/runtime/local/bin/nutch
+
+CRAWL_ITERATIONS=10
+
+function crawlSite() {
+    siteDir=$1
+    crawlId=$2
+
+    echo "processing site $siteDir"
+
+    #echo "processing site $siteDir with crawlId: $crawlId"
+
+    echo "Copying over template $NUTCH_URLFILTER_TEMPLATE to live version of file"
+    cp $NUTCH_URLFILTER_TEMPLATE $NUTCH_URLFILTER_FILE
+
+    echo "Appending contents of regex-urlfilter file for site $siteDir to url-filter file:"
+    cat ${siteDir}regex-urlfilter.txt >> $NUTCH_URLFILTER_FILE
+
+    #echo "Contents of seedURLs.txt file for site:"
+    #cat ${siteDir}seedURLs.txt                       
+    
+    # $siteDir parameter is the folder containing seedURLs.txt
+    crawl_cmd="./$CRAWL_COMMAND $siteDir $crawlId $CRAWL_ITERATIONS"
+
+    # Since we're going to crawl from scratch, create log.out file
+    # Logging to terminal and log file simultaenously
+    # https://stackoverflow.com/questions/418896/how-to-redirect-output-to-a-file-and-stdout
+    echo "Going to run nutch crawl command (and copy output to ${siteDir}log.out):" 2>&1 | tee ${siteDir}log.out
+    # append to log.out file hereafter
+    echo "  $crawl_cmd" 2>&1 | tee -a ${siteDir}log.out
+    echo "--------------------------------------------------" 2>&1 | tee -a ${siteDir}log.out
+
+    # append output of $crawl_cmd to log.out
+    $crawl_cmd 2>&1 | tee -a ${siteDir}log.out
+    result=$?
+
+    if [ "x$result" = "x0" ]; then
+	# nutch finished crawling successfully.
+	
+	# But check if the site was crawled thoroughly within $CRAWL_ITERATIONS
+	# If not, create file UNFINISHED to indicate a more thorough crawl needed
+	tail -10 ${siteDir}log.out | grep "no more URLs to fetch now" > /dev/null
+	result=$?
+	if [ "x$result" != "x0" ]; then
+	    echo "A crawl of $CRAWL_ITERATIONS iterations was insufficient for crawlId $crawlId" 2>&1 | tee ${siteDir}UNFINISHED
+	    echo "" 2>&1 | tee -a ${siteDir}UNFINISHED
+	    echo "To re-run crawl of site with crawlId $crawlId with a larger number of iterations:" 2>&1 | tee -a ${siteDir}UNFINISHED
+	    echo "1. delete $outputDir/$crawlId" 2>&1 | tee -a ${siteDir}UNFINISHED
+	    echo "2. copy the regex-urlfilter file:" 2>&1 | tee -a ${siteDir}UNFINISHED
+	    echo "   cp $NUTCH_URLFILTER_TEMPLATE $NUTCH_URLFILTER_FILE" 2>&1 | tee -a ${siteDir}UNFINISHED
+	    echo "3. Adjust # crawl iterations in old crawl command:" 2>&1 | tee -a ${siteDir}UNFINISHED
+	    echo "   $crawl_cmd" 2>&1 | tee -a ${siteDir}UNFINISHED
+	fi
+	
+	# outputDir/$crawlId should not yet exist
+     	./$NUTCH_COMMAND readdb -dump $outputDir/$crawlId -text -crawlId $crawlId
+     	./$NUTCH_COMMAND readdb -stats -crawlId $crawlId > $outputDir/$crawlId/stats
+     	cat $outputDir/$crawlId/part-r-* > $outputDir/$crawlId/dump.txt
+    else
+	# appending to log.out
+     	echo "CRAWL FAILED." 2>&1 | tee -a ${siteDir}log.out
+    fi
+
+
+    # move the peripheral crawl products (the log.out and optional UNFINISHED file)
+    # from the input to the output folder. This way we can re-run the crawl and
+    # these files will still have been preserved as long as the output folder
+    # isn't deleted
+    mv ${siteDir}log.out $outputDir/$crawlId/log.out
+    if [ -e "${siteDir}UNFINISHED" ]; then
+	mv ${siteDir}UNFINISHED $outputDir/$crawlId/UNFINISHED
+    fi
+
+    # finally append the current crawlId to siteIDsCrawled.txt
+    echo $crawlId >> $outputDir/siteIDsCrawled.txt
+}
+
+function prepareForCrawl() {
+    siteDir=$1
+    crawlId=$2
+
+    echo "Processing siteDir $siteDir with crawlId: $crawlId"
+    
+    if [ -d "$outputDir/$crawlId" ]; then
+	# Skip site already processed. *Append* this msg to log.out
+	echo "" 2>&1 | tee -a ${siteDir}log.out
+	echo "**** $siteDir already processed. Skipping...." 2>&1 | tee -a ${siteDir}log.out
+	echo "Delete $outputDir/$crawlId if you want to reprocess it." 2>&1 | tee -a ${siteDir}log.out	
+
+    else	
+	crawlSite $siteDir $crawlId
+
+    fi
+    echo "--------------------------------------------------"
+    
+
+    
+}
+
+function crawlAll() {
+
+    # https://stackoverflow.com/questions/4000613/perform-an-action-in-every-sub-directory-using-bash
+    for siteDir in $sitesDir/*/; do
+	
+	# to get crawl_id like 00001 from $siteDir like to_crawl/sites/00001/
+	# Remove the $sitesDir prefix of to_crawl/sites followed by /,
+	# Next remove the / suffix that remains
+	crawlId=${siteDir#"$sitesDir/"}
+	crawlId=${crawlId%/}
+	
+	prepareForCrawl $siteDir $crawlId
+	#break
+    done
+}
+
+function crawlRange() {
+    
+    startCrawlId=$1
+    endCrawlId=$2
+
+    # https://unix.stackexchange.com/questions/232384/argument-string-to-integer-in-bash/301285
+
+    # sadly, the numeric value is in octal in both the following
+    #end=$(($endCrawlId+0))    
+    #printf -v end '%d\n' $endCrawlId 2>/dev/null
+    # removes a single 0 prefix
+    #end=${endCrawlId##+(0)}
+    #${old##+(0)}
+    # https://stackoverflow.com/questions/19861394/stripping-leading-zeros-using-bash-substring-removal-only/51196007
+    start=${startCrawlId#"${startCrawlId%%[!0]*}"}
+    end=${endCrawlId#"${endCrawlId%%[!0]*}"}
+
+    echo "Start: $start ($startCrawlId), End: $end ($endCrawlId)"
+
+    # Generate a range of numbers between start and end
+    #https://stackoverflow.com/questions/966020/how-to-produce-a-range-with-step-n-in-bash-generate-a-sequence-of-numbers-with/966026
+    for (( COUNTER=$start; $COUNTER<=$end; COUNTER=$COUNTER+1 )); do
+	# Now pad back with zeroes to get crawlId
+	# https://stackoverflow.com/questions/1117134/padding-zeros-in-a-string	
+	crawlId=`printf %05d $COUNTER`
+	#echo $COUNTER - $crawlId
+	# $siteDir needs / at end to work with existing code
+	siteDir=$sitesDir/$crawlId/
+	#echo "siteDir $siteDir (crawlId $crawlId)"
+	prepareForCrawl $siteDir $crawlId
+    done
+    
+}
+
+function printUsage() {
+    echo "Run as:"
+    echo "  $0 -all|<ids>"
+    echo "    where an id is a folder name in to_crawl/sites"
+    echo "    and ids can be a comma or space separated list of"
+    echo "    individual ids and/or ranges"
+    echo "  Examples:"
+    echo "    $0 00008-00022,00025,00026,00028-00034 00050"
+    echo "    $0 -all"
+    echo "    $0 00312"
+    
+}
+
+function parseArgs() {
+    
+    # for i in $*; do
+    # 	echo "Pinky"
+    # 	echo $i
+    # done
+
+    args="$1"
+    #echo "Got arg string: $args"
+
+    if [ "x$args" = "x" ]; then	
+	printUsage
+	exit 0
+    fi
+    
+    # works - split args on comma or space
+    # https://stackoverflow.com/questions/918886/how-do-i-split-a-string-on-a-delimiter-in-bash
+    IFS=', ' read -ra IDS <<< "$args"
+    for id in "${IDS[@]}"; do
+	echo "id: |$id|"
+	if [ "x$id" = "x-all" ]; then
+	    echo "crawlAll"
+	    crawlAll
+	elif [[ $id == *"-"* ]]; then
+	    # https://stackoverflow.com/questions/229551/how-to-check-if-a-string-contains-a-substring-in-bash
+	    echo "$id is a range"
+	    startCrawlId=${id%%-*}
+	    endCrawlId=${id##*-}
+	    echo "crawlRange $startCrawlId $endCrawlId"
+	    crawlRange $startCrawlId $endCrawlId	
+	    
+	else
+	    echo "$id is singleton"
+	    crawlId=$id
+	    # $siteDir needs / at end to work with existing code
+	    siteDir=$sitesDir/$crawlId/
+	    echo "prepareForCrawl $siteDir $crawlId"
+	    prepareForCrawl $siteDir $crawlId	    
+	fi
+    done
+}
+
+
+
+# Passing as string instead of individual arguments
+# Because one can only have 9 individual args without using shift, see
+# https://www.unix.com/shell-programming-and-scripting/57225-there-limit-no-arguments-shell-script.html
+
+parseArgs "$*"
+
+# old. testing
+#crawlRange "00010" "00015"
+#parseArgs "00008-00022,00025,00026,00028-00034"
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/exportHBase.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/exportHBase.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/exportHBase.sh	(revision 33635)
@@ -0,0 +1,79 @@
+#!/bin/bash
+
+# This script exports the Hbase tables for the crawled websites' webpages to files called part-m-*
+# which presumably can be reimported into Hbase.
+# See https://blogs.msdn.microsoft.com/data_otaku/2016/12/21/working-with-the-hbase-import-and-export-utility/
+# We want to backup our HBase tables that were created by crawling, so we can finally retire the
+# vagrant Virtual Machines we used to crawl web sites.
+
+
+# When doing this on the command line:
+#
+# hdfs dfs -mkdir /user/vagrant
+# EXPORT CMD:
+# vagrant@node5:~/crawled$ hbase org.apache.hadoop.hbase.mapreduce.Export "01066_webpage" "/user/vagrant/01066"
+#
+# Note that when the script runs the above with double quotes around the tablename or outputdir
+# hadoop doesn't like it. But such a cmd was okay from the command line.
+#
+# OUTPUT:
+# vagrant@node5:~/crawled$ hdfs dfs -ls /user/vagrant/01066
+# Found 2 items
+# -rw-r--r--   1 vagrant supergroup          0 2019-10-30 08:29 /user/vagrant/01066/_SUCCESS
+# -rw-r--r--   1 vagrant supergroup  184788017 2019-10-30 08:29 /user/vagrant/01066/part-m-00000
+
+
+sitesDir=crawled
+echo "SITES DIR (INPUT): $sitesDir"
+
+logOut=$sitesDir/exportHBase.log
+
+# output dir is on hdfs (i.e. hdfs:///user/vagrant)
+outputDir=/user/vagrant
+
+exportCmd="hbase org.apache.hadoop.hbase.mapreduce.Export"
+
+function exportAll() {
+
+    if $(hdfs dfs -test -d "$outputDir"); then
+	echo "Output directory on hdfs $outputDir already exists."
+    else
+	echo "Creating directory on hfds $outputDir..."
+	hdfs dfs -mkdir $outputDir
+    fi
+    
+    # https://stackoverflow.com/questions/4000613/perform-an-action-in-every-sub-directory-using-bash
+    for siteDir in $sitesDir/*/; do
+	
+	# to get crawl_id like 00001 from $siteDir like to_crawl/sites/00001/
+	# Remove the $sitesDir prefix of to_crawl/sites followed by /,
+	# Next remove the / suffix that remains
+	crawlId=${siteDir#"$sitesDir/"}
+	crawlId=${crawlId%/}
+	
+	echo "$siteDir $crawlId"
+
+	fullExportCmd="$exportCmd ${crawlId}_webpage $outputDir/$crawlId"
+	
+	echo "Going to run command:"
+	#echo "   $exportCmd \"${crawlId}_webpage\" \"$outputDir/$crawlId\"" 2>&1 | tee $logOut
+	echo "    $fullExportCmd" 2>&1 | tee $logOut
+
+	# run the export from hbase - produces a file called part-m-00000 (perhaps multiple files called part-m-*)
+	$fullExportCmd 2>&1 | tee -a $logOut
+
+	
+	# if export was successful, should have file _SUCCESS in outputdir
+	# In that case, copy the outputfile(s) named part-m-* into crawl folder
+	if $(hdfs dfs -test -f "$outputDir/$crawlId/_SUCCESS"); then
+	    hdfs dfs -get $outputDir/$crawlId/part-m-* ${sitesDir}/$crawlId/.
+	else
+	    echo "ERROR: EXPORT OF $crawlId FAILED" 2>&1 | tee -a $logOut
+	fi
+
+    done
+}
+
+
+# script begins here
+exportAll
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_index_csv.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_index_csv.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_index_csv.sh	(revision 33635)
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+#DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+CRAWL_ID=$1
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-csv"
+OUTPUT_PARENTDIR=hdfs:///user/vagrant/${CRAWL_ID}
+     # or just OUTPUT_PARENTDIR=/user/vagrant/${CRAWL_ID}, since /user/vagrant is on hdfs:
+
+# https://stackoverflow.com/questions/26513861/checking-if-directory-in-hdfs-already-exists-or-not
+hdfs dfs -test -d $OUTPUT_PARENTDIR
+if [ $? == 0 ]; then
+    echo "Directory $OUTPUT_PARENTDIR already exists."        
+else
+    echo "Creating directory $OUTPUT_PARENTDIR..."    
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR
+fi
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+#    --conf spark.hadoop.fs.s3a.access.key=AKIA2EVQBWSTBJ2M4BLM \
+#    --conf spark.hadoop.fs.s3a.secret.key=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 \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+# https://www.patricia-anong.com/blog/2017/11/1/extend-vmdk-on-virtualbox
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --conf spark.sql.parquet.mergeSchema=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length   
+            FROM ccindex                                                        
+            WHERE crawl = '${CRAWL_ID}' AND subset = 'warc' AND content_languages = 'mri'" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    $OUTPUTDIR
+
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset.sh	(revision 33635)
@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+#DATA="$1"
+# Output directory
+CRAWL_ID=$1
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+#OUTPUTDIR="$2"
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-subset"
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+# $SPARK_HOME/bin/spark-submit \
+#     $SPARK_ON_YARN \
+#     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+#     --conf spark.core.connection.ack.wait.timeout=600s \
+#     --conf spark.network.timeout=120s \
+#     --conf spark.task.maxFailures=20 \
+#     --conf spark.shuffle.io.maxRetries=20 \
+#     --conf spark.shuffle.io.retryWait=60s \
+#     --conf spark.driver.memory=$DRIVER_MEM \
+#     --conf spark.executor.memory=$EXECUTOR_MEM \
+#     $SPARK_EXTRA_OPTS \
+#     --num-executors $NUM_EXECUTORS \
+#     --executor-cores $EXECUTOR_CORES \
+#     --executor-memory $EXECUTOR_MEM \
+#     --conf spark.hadoop.parquet.enable.dictionary=true \
+#     --conf spark.sql.parquet.filterPushdown=true \
+#     --conf spark.sql.parquet.mergeSchema=false \
+#     --conf spark.sql.hive.metastorePartitionPruning=true \
+#     --conf spark.hadoop.parquet.enable.summary-metadata=false \
+#     --class org.commoncrawl.spark.CCIndex2Table $_APPJAR \
+#     --outputCompression=$COMPRS \
+#     --outputFormat=$FORMAT $NESTED \
+#     --partitionBy=$PARTITION_BY \
+#     "$DATA" "$OUTPUTDIR"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+   # --conf spark.hadoop.parquet.enable.dictionary=true \
+   # --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   # --conf spark.sql.hive.metastorePartitionPruning=true \
+   # --conf spark.sql.parquet.filterPushdown=true \
+   # --conf spark.sql.parquet.mergeSchema=true \
+
+
+
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --csv hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-unzipped-csv/cc-mri.csv \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-${CRAWL_ID} \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   $OUTPUTDIR
+
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset_from_scratch.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset_from_scratch.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/export_maori_subset_from_scratch.sh	(revision 33635)
@@ -0,0 +1,98 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+#DATA="$1"
+# Output directory
+CRAWL_ID=$1
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-subset"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+# $SPARK_HOME/bin/spark-submit \
+#     $SPARK_ON_YARN \
+#     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+#     --conf spark.core.connection.ack.wait.timeout=600s \
+#     --conf spark.network.timeout=120s \
+#     --conf spark.task.maxFailures=20 \
+#     --conf spark.shuffle.io.maxRetries=20 \
+#     --conf spark.shuffle.io.retryWait=60s \
+#     --conf spark.driver.memory=$DRIVER_MEM \
+#     --conf spark.executor.memory=$EXECUTOR_MEM \
+#     $SPARK_EXTRA_OPTS \
+#     --num-executors $NUM_EXECUTORS \
+#     --executor-cores $EXECUTOR_CORES \
+#     --executor-memory $EXECUTOR_MEM \
+#     --conf spark.hadoop.parquet.enable.dictionary=true \
+#     --conf spark.sql.parquet.filterPushdown=true \
+#     --conf spark.sql.parquet.mergeSchema=false \
+#     --conf spark.sql.hive.metastorePartitionPruning=true \
+#     --conf spark.hadoop.parquet.enable.summary-metadata=false \
+#     --class org.commoncrawl.spark.CCIndex2Table $_APPJAR \
+#     --outputCompression=$COMPRS \
+#     --outputFormat=$FORMAT $NESTED \
+#     --partitionBy=$PARTITION_BY \
+#     "$DATA" "$OUTPUTDIR"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+	  
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --conf spark.hadoop.parquet.enable.dictionary=true \
+   --conf spark.hadoop.parquet.enable.summary-metadata=false \
+   --conf spark.sql.hive.metastorePartitionPruning=true \
+   --conf spark.sql.parquet.filterPushdown=true \
+   --conf spark.sql.parquet.mergeSchema=true \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --query "SELECT url, warc_filename, warc_record_offset, warc_record_length
+            FROM ccindex
+            WHERE crawl = '${CRAWL_ID}' AND subset = 'warc' AND content_languages LIKE '%mri%'" \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-${CRAWL_ID} \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   $OUTPUTDIR
+
+#   hdfs:///user/vagrant/cc-mri-subset
+#   s3://commoncrawl/cc-index/table/cc-main/warc/ \
+# Since s3:// deprecated, moved to using s3a:// now
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh	(revision 33635)
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# This script should be put into cc-index-table (github project) toplevel folder
+# It works with the get_maori_WET_records_for_crawl.sh script, which has to be
+# placed in cc-index-table/src/script
+
+# crawl_ids are from http://index.commoncrawl.org/
+# We only want the crawl_ids from Sep 2018 and onwards as that's when
+# the content_languages field was included in CommonCrawl's columnar index
+
+# https://www.cyberciti.biz/faq/bash-for-loop-array/
+# (else chain commands as at https://superuser.com/questions/237072/wrapping-long-bash-commands-in-script-files)
+crawl_ids=( "CC-MAIN-2019-35" "CC-MAIN-2019-30" "CC-MAIN-2019-26" \
+      "CC-MAIN-2019-22" "CC-MAIN-2019-18" "CC-MAIN-2019-13" \
+      "CC-MAIN-2019-09" "CC-MAIN-2019-04" "CC-MAIN-2018-51" \
+      "CC-MAIN-2018-47" "CC-MAIN-2018-43" "CC-MAIN-2018-39" )
+
+for crawl_id in "${crawl_ids[@]}"
+do
+    echo "About to start off index and WARC download process for CRAWL ID: $crawl_id"
+    ./src/script/get_maori_WET_records_for_crawl.sh $crawl_id
+    result=$?
+    if [ $result != 0 ]; then
+	echo "Processing common-crawl $crawl_id failed with exit value: $result"
+	echo "Will cease to process remaining cc crawls. Exitting..."
+	exit $result
+    fi
+done
+
+# Important note from
+# https://www.tldp.org/LDP/abs/html/exit-status.html
+# "When a script ends with an exit that has no parameter, the exit
+# status of the script is the exit status of the last command executed
+# in the script (previous to the exit)."
+
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh	(revision 33635)
@@ -0,0 +1,205 @@
+#!/bin/bash
+
+# This script is BASED ON the cc-index-table github project's convert_url_index script at
+# https://github.com/commoncrawl/cc-index-table/blob/master/src/script/convert_url_index.sh
+# That script is described as
+# "A Spark job converts the Common Crawl URL index files (a sharded gzipped index in CDXJ format)
+# into a table in Parquet or ORC format." (https://github.com/commoncrawl/cc-index-table)."
+# If you want to run that script, then modify its variables to have the following values before
+# running it, in order for it to work on our machine for doing analytics:
+# EXECUTOR_MEM=3g
+# EXECUTOR_CORES=2
+# NUM_EXECUTORS=2
+# DRIVER_MEM=3g
+# Since that script was copied here, as a result, a lot of such variables (like executor
+# and memory related) are unused here, as they were just copied directly across. Such unused
+# variables can probably be removed from this file.
+
+# This script was modified to do the following:
+# SQL query CommonCrawl's distributed cc-index table on Amazon S3 for the parameterised crawl timestamp
+# and get all those records for which the primary language in the content_languages field is MRI for Maori.
+# Only the WARC related fields (url, filename, offset and length fields) of each record are requested.
+# The matching records' fields are then constructed into a distributed csv file on the local hdfs system
+# A second phase then requests the warc files at those offsets and downloads them onto the local hdfs.
+# We still get zipped WARC files, but they only contain the pages of that crawl where the primary language
+# was identified as MRI.
+# A third phase converts those WARC files into WET (and WAT) files and copies these zipped files onto the
+# mounted shared space on vagrant.
+
+
+#---------------------------- START UNUSED VARIABLES---------------------------#
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+#--- Dr Bainbridge modified the above variables in the original script, convert_url_index.sh,
+# as follows in order to get that spark job to run. Not used in this script. ---#
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+#--- VARIABLES PROBABLY ALSO NOT OF USE IN THIS SCRIPT ---#
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+#---------------------------- END UNUSED VARIABLES---------------------------#
+
+
+# The crawl timestamp, of the form CC-MAIN-2019-26
+# Obtain from http://index.commoncrawl.org/
+CRAWL_ID=$1
+if [ "x$CRAWL_ID" == "x" ]; then
+    echo "No crawl timestamp provided. Should be of the form CC-MAIN-YYYY-COUNT."
+    echo "e.g. CC-MAIN-2019-26. Choose a crawl timestamp from http://index.commoncrawl.org/"
+    exit
+fi
+# Output directory
+OUTPUT_PARENTDIR=hdfs:///user/vagrant/${CRAWL_ID}
+     # or just OUTPUT_PARENTDIR=/user/vagrant/${CRAWL_ID}, since /user/vagrant is on hdfs:
+
+# https://stackoverflow.com/questions/26513861/checking-if-directory-in-hdfs-already-exists-or-not
+#hdfs dfs -test -d $OUTPUT_PARENTDIR
+#if [ $? == 0 ]; then
+if $(hdfs dfs -test -d "$OUTPUT_PARENTDIR"); then
+    echo "Directory $OUTPUT_PARENTDIR already exists."        
+else
+    echo "Creating directory $OUTPUT_PARENTDIR..."    
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR
+fi
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+# PHASE 1: querying this crawl's massive index with an SQL query that requests just the references to warc files
+# for those crawled web pages where the content_languages field's primary language is MRI (3 letter code for Maori)
+# The output is a distributed .csv file which will be stored in a "cc-mri-csv" subfolder of the $OUTPUT_PARENTDIR.
+
+#OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/cc-mri-csv"
+OUTPUTDIR="${OUTPUT_PARENTDIR}/cc-mri-csv"
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+# https://www.patricia-anong.com/blog/2017/11/1/extend-vmdk-on-virtualbox
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --conf spark.sql.parquet.mergeSchema=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length   
+            FROM ccindex                                                        
+            WHERE crawl = '${CRAWL_ID}' AND subset = 'warc' AND content_languages = 'mri'" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    $OUTPUTDIR
+
+
+# The above generates ZIPPED part*.csv files in $OUTPUTDIR (folder cc-mri-csv).
+# First create a folder and unzip into it:
+#hdfs dfs -test -d $OUTPUT_PARENTDIR/cc-mri-unzipped-csv
+#if [ $? == 0 ]; then
+if $(hdfs dfs -test -d "$OUTPUT_PARENTDIR/cc-mri-unzipped-csv"); then
+    echo "Directory cc-mri-unzipped-csv already exists for crawl ${CRAWL_ID}."
+    echo "Assuming cc-mri.csv also exists inside $OUTPUT_PARENTDIR"
+else
+    echo "Creating directory $OUTPUT_PARENTDIR/cc-mri-unzipped-csv..."
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR/cc-mri-unzipped-csv
+
+    echo "Unzipping ${OUTPUTDIR}/part files into $OUTPUT_PARENTDIR/cc-mri-unzipped-csv/cc-mri.csv"
+    hdfs dfs -cat $OUTPUTDIR/part* | gzip -d | hdfs dfs -put - $OUTPUT_PARENTDIR/cc-mri-unzipped-csv/cc-mri.csv
+fi
+
+
+
+# PHASE 2, which uses the index of MRI warc URLs and offsets,
+# stored in the now unzipped .csv file,
+# to get all the WARC records it specifies at the specified warc offsets.
+# This will get us all the MRI pages of the commoncrawl for CRAWL_ID
+
+# Change OUTPUTDIR
+OUTPUTDIR="hdfs:///user/vagrant/${CRAWL_ID}/warc"
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+
+
+#    $SPARK_ON_YARN \
+$SPARK_HOME/bin/spark-submit \
+   --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+   --class org.commoncrawl.spark.examples.CCIndexWarcExport $_APPJAR \
+   --csv $OUTPUT_PARENTDIR/cc-mri-unzipped-csv/cc-mri.csv \
+   --numOutputPartitions 10 \
+   --numRecordsPerWarcFile 5000 \
+   --warcPrefix MAORI-${CRAWL_ID} \
+   s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+   $OUTPUTDIR
+
+
+# PHASE 3: convert warc files to wet files and copy the wet files into the mounted shared area
+
+#hdfs dfs -test -f $OUTPUTDIR/_SUCCESS
+#if [ $? == 0 ]; then
+if $(hdfs dfs -test -f "$OUTPUTDIR/_SUCCESS"); then
+    # ia-hadoop-tools converts warc files into wet (and wat) files but expects a particular folder structure
+    # Create the expected folder structure: a "wet" and a "wat" folder should exist
+    # at the same level as the "warc" folder ($OUTPUTDIR) that contains the warc gz files.
+    
+    echo "Creating wet (text) and wat (metadata) directories to put converted warc files into..."
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR/wet
+    hdfs dfs -mkdir $OUTPUT_PARENTDIR/wat
+    # move the _SUCCESS file from /warc to its parent folder
+    hdfs dfs -mv $OUTPUTDIR/_SUCCESS $OUTPUT_PARENTDIR/.
+
+    # Move from the "/home/vagrant/cc-index-table" github project to the "ia-hadoop-tools" github project:
+    cd /home/vagrant/ia-hadoop-tools
+    # PWD is now ia-hadoop-tools folder
+    # To run the following, guava.jar should be on the hadoop classpath
+    # This can be achieved with:
+    # vagrant@node1:~/ia-hadoop-tools$ sudo cp /usr/share/java/guava.jar /usr/local/hadoop/share/hadoop/common/.
+    $HADOOP_MAPRED_HOME/bin/hadoop jar $PWD/target/ia-hadoop-tools-jar-with-dependencies.jar WEATGenerator -strictMode -skipExisting batch-id-xyz $OUTPUTDIR/*.warc.gz
+
+    # Now copy the zipped wet-files into the mounted folder, so we can scp the files from there to where WETProcessor.java is
+    echo "Copying and tarring up the wet files"
+    mkdir /vagrant/${CRAWL_ID}-wet-files
+    hdfs dfs -get $OUTPUT_PARENTDIR/wet/*.warc.wet.gz /vagrant/${CRAWL_ID}-wet-files/.
+    cd /vagrant
+    #tar -cvf ${CRAWL_ID}.tar /vagrant/${CRAWL_ID}-wet-files
+    #rm -rf /vagrant/${CRAWL_ID}-wet-files
+fi
+
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/limit10_export_index.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/limit10_export_index.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/limit10_export_index.sh	(revision 33635)
@@ -0,0 +1,76 @@
+#!/bin/bash
+
+# convert URL index with Spark on Yarn
+
+# Table format configuration
+FORMAT=${FORMAT:-parquet} # parquet, orc
+NESTED="$NESTED"          # "" (empty) or --useNestedSchema
+COMPRS=${COMPRS:-gzip}    # gzip, snappy, lzo, none
+PARTITION_BY="crawl,subset"
+
+# Input spec (URL index files to convert)
+#DATA=s3a://commoncrawl/cc-index/collections/CC-MAIN-2017-51/indexes/cdx-*.gz
+DATA="$1"
+# Output directory
+#OUTPUTDIR=hdfs:///user/ubuntu/cc-index-table/
+OUTPUTDIR="$2"
+
+
+# Spark configuration
+SPARK_HOME="$SPARK_HOME"
+# EXECUTOR_MEM=44g
+# EXECUTOR_CORES=12
+# NUM_EXECUTORS=4
+# DRIVER_MEM=4g
+
+EXECUTOR_MEM=3g
+EXECUTOR_CORES=2
+NUM_EXECUTORS=2
+DRIVER_MEM=3g
+
+
+SPARK_ON_YARN="--master yarn"
+SPARK_EXTRA_OPTS=""
+
+# source specific configuration file
+## test -e $(dirname $0)/convert_url_index_conf.sh && . $(dirname $0)/convert_url_index_conf.sh
+
+
+_APPJAR=$PWD/target/cc-spark-0.2-SNAPSHOT-jar-with-dependencies.jar
+
+##export LIBJARS=/home/vagrant/lib/hadoop-aws-2.7.6.jar
+#export LIBJARS=/home/vagrant/lib/*
+#export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
+
+set -e
+set -x
+
+
+#   --conf spark.hadoop.fs.s3a.access.key=ACCESSKEY \
+#   --conf spark.hadoop.fs.s3a.secret.key=SECRETKEY \
+#    --conf spark.hadoop.fs.s3a.access.key=AKIA2EVQBWSTBJ2M4BLM \
+#    --conf spark.hadoop.fs.s3a.secret.key=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 \
+
+
+#    $SPARK_ON_YARN \
+# /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar
+#    --jars file:/home/vagrant/aws-java-sdk-1.7.4.jar,file:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+#    --driver-class-path=/home/vagrant/lib/aws-java-sdk-1.7.4.jar:/home/vagrant/lib/hadoop-aws-2.7.6.jar \
+
+#    --conf spark.sql.parquet.mergeSchema=true \
+
+$SPARK_HOME/bin/spark-submit \
+    --jars file:/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar,file:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --driver-class-path=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6/hadoop-aws-2.7.6.jar \
+    --conf spark.hadoop.parquet.enable.dictionary=true \
+    --conf spark.hadoop.parquet.enable.summary-metadata=false \
+    --conf spark.sql.hive.metastorePartitionPruning=true \
+    --conf spark.sql.parquet.filterPushdown=true \
+    --class org.commoncrawl.spark.examples.CCIndexExport $_APPJAR \
+    --query "SELECT * FROM ccindex LIMIT 10" \
+    --outputFormat csv \
+    --numOutputPartitions 10 \
+    --outputCompression gzip \
+    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
+    hdfs:///user/vagrant/cc-limit10-csv
+
Index: /other-projects/maori-lang-detection/hdfs-cc-work/scripts/setup.sh
===================================================================
--- /other-projects/maori-lang-detection/hdfs-cc-work/scripts/setup.sh	(revision 33635)
+++ /other-projects/maori-lang-detection/hdfs-cc-work/scripts/setup.sh	(revision 33635)
@@ -0,0 +1,88 @@
+#!/bin/bash
+
+# set value to "copy_extra_jars" if you want certain additional jars copied over,
+# a step that may be optional or may be necessary
+copy_extra_jars=$1
+
+echo "Checking out hdfs-cc-work from GS svn's gs3-extensions/maori-lang-detection"
+svn co http://svn.greenstone.org/gs3-extensions/maori-lang-detection/hdfs-cc-work
+
+echo "Moving the tarred 3 git projects, cc-index-table (modified), ia-hadoop-tools (modified) and ia-web-commons into the current folder and extracting..."
+mv hdfs-cc-work/gitprojects/cc-index-table.tar .
+mv hdfs-cc-work/gitprojects/ia-hadoop-tools.tar .
+mv hdfs-cc-work/gitprojects/ia-web-commons.tar .
+
+tar -xvf cc-index-table.tar
+tar -xvf ia-hadoop-tools.tar
+tar -xvf ia-web-commons.tar
+
+
+echo "Copying scripts into cc-index-table and making them executable"
+# the script that does the actual work
+cp hdfs-cc-work/scripts/get_maori_WET_records_for_crawl.sh cc-index-table/src/script/.
+chmod u+x cc-index-table/src/script/get_maori_WET_records_for_crawl.sh
+# top level script belonging in top level of cc-index-table
+cp hdfs-cc-work/scripts/get_Maori_WET_records_from_CCSep2018_on.sh cc-index-table/.
+chmod u+x cc-index-table/get_Maori_WET_records_from_CCSep2018_on.sh
+
+
+echo "Compiling the modified cc-index-table"
+cd cc-index-table
+mvn package
+
+# ia-web-commons has to be compiled up before ia-hadoop-tools
+echo "Compiling ia-web-commons"
+cd ../ia-web-commons
+mvn install
+
+echo "Compiling the modified ia-hadoop-tools"
+cd ../ia-hadoop-tools
+mvn package
+
+
+echo "Putting jar files in place"
+
+echo "1. For RUNNING warc to wet conversion (ia-hadoop-tools): Sudo copying hdfs-cc-work/jars/guava.jar into hadoop classpath $hadoop_common_dir"
+sudo cp hdfs-cc-work/jars/guava.jar $hadoop_common_dir/.
+
+# Some discussion about == and = allowed in if test for equality
+# https://stackoverflow.com/questions/4277665/how-do-i-compare-two-string-variables-in-an-if-statement-in-bash
+if [ "x$copy_extra_jars" == "xcopy_extra_jars" ]; then
+    hadoop_common_dir=/usr/local/hadoop/share/hadoop/common
+    echo "Copying extra jars"
+    echo "2. For cc-index-table: Sudo copying hadoop-aws-2.7.6.jar and aws-java-sdk-1.11.616.jar"
+    echo "from hdfs-cc-work/jars into hadoop classpath $hadoop_common_dir"
+    sudo cp hadoop-aws-2.7.6.jar $hadoop_common_dir/.
+    sudo cp aws-java-sdk-1.11.616.jar $hadoop_common_dir/.
+    
+    dest_dir=/home/vagrant/.m2/repository/com/amazonaws/aws-java-sdk/1.7.4
+    if [ ! -e "$dest_dir/aws-java-sdk-1.7.4.jar" ]; then
+	echo "3. For cc-index-table: Copying aws-java-sdk-1.7.4.jar into $dest_dir/"
+	mkdir -p $dest_dir
+	cp hdfs-cc-work/jars/aws-java-sdk-1.7.4.jar $dest_dir/.
+    fi
+    
+    dest_dir=/home/vagrant/.m2/repository/org/apache/hadoop/hadoop-aws/2.7.6
+    if [ ! -e "$dest_dir/hadoop-aws-2.7.6.jar" ]; then
+	echo "4. For cc-index-table: Copying hadoop-aws-2.7.6.jar into $dest_dir/"
+	mkdir -p $dest_dir
+	cp hdfs-cc-work/jars/aws-java-sdk-1.7.4.jar $dest_dir/.
+    fi
+fi
+
+echo "Done compiling and automated parts of setting up."
+echo "NEXT STEP:"
+echo "Ensure you have sudo edited $SPARK_HOME/conf/spark-defaults.conf"
+echo "  (/usr/local/spark-2.3.0-bin-hadoop2.7/conf/spark-defaults.conf)"
+echo "to contain the following 3 lines with YOUR Amazon AWS IAM Role access and secret keys:"
+echo "   spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem"
+echo "   spark.hadoop.fs.s3a.access.key=YOUR_AWS_IAM-ROLE_ACCESSKEY_HERE"
+echo "   spark.hadoop.fs.s3a.secret.key=YOUR_AWS_IAM-ROLE_SECRETKEY_HERE"
+echo "Consult GS_README.TXT section B (and C) for instructions on setting up an AWS IAM role."
+echo "Only when that's done will you be ready to run the following script."
+echo ""
+echo "THEN:"
+echo "To get MRI warc to wet for a particular crawl timestamp, cd into cc-index-table and RUN:"
+echo "./get_maori_WET_records_for_crawl.sh CC-MAIN-<YYYY-##>"
+echo "   where YYYY is the year and ## is the common crawl number for that year."
+
Index: /other-projects/maori-lang-detection/lib/LICENSE.txt
===================================================================
--- /other-projects/maori-lang-detection/lib/LICENSE.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/lib/LICENSE.txt	(revision 33635)
@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
Index: /other-projects/maori-lang-detection/lib/NOTICE.txt
===================================================================
--- /other-projects/maori-lang-detection/lib/NOTICE.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/lib/NOTICE.txt	(revision 33635)
@@ -0,0 +1,5 @@
+Apache Commons CSV
+Copyright 2005-2019 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
Index: /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/mri-sent.train
===================================================================
--- /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/mri-sent.train	(revision 33635)
+++ /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/mri-sent.train	(revision 33635)
@@ -0,0 +1,25 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Index: /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/sample_maori_shorttext.txt
===================================================================
--- /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/sample_maori_shorttext.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/sample_maori_shorttext.txt	(revision 33635)
@@ -0,0 +1,5 @@
+
+Nau Mai Haere Mai
+
+Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei.
+
Index: /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt
===================================================================
--- /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt	(revision 33635)
+++ /other-projects/maori-lang-detection/models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt	(revision 33635)
@@ -0,0 +1,1 @@
+Kua whakaemitia mai ki konei he kohikohinga niupepa i tÄngia mÅ ngÄ kaipÄnui MÄori o ngÄ tau 1842-1932. E taea te pÄnui niupepa mÄ te rapu kupu, i te rÄrangi taitara, me te rÄrangi wÄtaka hoki. He mea i whakatÅ«ria e te kaupapa New Zealand Digital Library, i te Tari Rorohiko, o te Whare WÄnanga o Waikato. Kei raro nei he kupu whakamÄrama e pÄ ana ki te kohikohinga nei. Ka nui te mihi ki ngÄ rÅpÅ« tautoko i te kaupapa nei. NÄ tÄ rÄtou tautoko kua kore he utu, mÄ koutou, mÅ te titiro ki ngÄ niupepa nei. Ko te TÄhuhu o te MÄtauranga tÄrÄ, me Ätehi whare pukapuka e noho mai nei ki ngÄ whare wÄnanga o te motu. Tirohia ngÄ rÅpÅ« tautoko. 
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/CCWETProcessor.java	(revision 33635)
@@ -0,0 +1,936 @@
+package org.greenstone.atea;
+
+
+import java.io.*;
+import java.nio.charset.StandardCharsets;
+import java.util.Properties;
+import java.util.zip.GZIPInputStream;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+
+import org.apache.commons.csv.*; // https://commons.apache.org/proper/commons-csv/download_csv.cgi
+import org.apache.log4j.Logger;
+
+
+/**
+ * The main() method of this class takes a folder of warc.wet(.gz) files and goes through
+ * the WET records in each, putting each WET record into a file. Each file is put into a
+ * keep or discard or greyListed folder, and its url listed written into a keep, discard
+ * or greylisted text file, based on based on
+ * 
+ * 1. whether it's whitelisted, else greylisted else blacklisted
+ * 2. and if explicitly whitelisted or else not greylisted or blacklisted and there's
+ * enough content. Formerly, content-length and number of lines were used to determine if
+ * the content was sufficient. Now it's just word count and number of MAX characters
+ * (not MINIMUM characters) that determine a string is a word. These settings can be adjusted
+ * in conf/config.properties.
+ *
+ * Put a url-blacklist-filter.txt and/or url-greylist-filter.txt and/or url-whitelist-filter.txt
+ * into the conf folder to control any url patterns that are explicitly included or excluded or
+ * set aside for inspecting later. These filter text files don't use regexes, instead their
+ * format is:
+ * - precede URL by ^ to blacklist urls that match the given prefix
+ * - succeed URL by $ to blacklist urls that match the given suffix
+ * - ^url$ will blacklist urls that match the given url completely
+ * - Without either ^ or $ symbol, urls containing the given url will get blacklisted
+ * 
+ * WETProcessor.java's current implementation is that explicit whitelisting has precedence
+ * over greylisting and which takes precedence over blacklisting in turn. However, even
+ * explicitly whitelisted urls still need to have sufficient content to end up in keepURLs.txt
+ * and in the seedURLs.txt file used for nutch, along with its domain in regex-urlfilter.txt
+ * also for nutch.
+ *
+ * A CCWETProcessor instance can be configured to process all the .warc.wet(.gz) files
+ * in the given input folder. Then use a single instance of the WETProcessor class to process
+ * each single unzipped warc.wet file.
+ *
+ * To compile, including the jars in lib/ for compiling.
+ *      maori-lang-detection/src$ javac -cp ".:../lib/*" org/greenstone/atea/CCWETProcessor.java 
+ *
+ * To run, passing the log4j and other properties files in conf/ folder:
+ *      maori-lang-detection/src$ java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor &lt;folder containing commoncrawls subfolders containing warc.wet(.gz) files&gt; &lt;outputFolder&gt;
+ *
+ * e.g. (from maori-lang-detection/src)
+ *    
+ *    - java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor ../ccrawl-data /Scratch/ak19/gs3-extensions/maori-lang-detection/to_crawl
+ *    - java -cp ".:../conf:../lib/*" org.greenstone.atea.CCWETProcessor ../ccrawl-data /Scratch/ak19/gs3-extensions/maori-lang-detection/to_crawl 2&gt;&amp;1 | less
+ *
+*/
+
+public class CCWETProcessor {
+    private static Logger logger = Logger.getLogger(org.greenstone.atea.CCWETProcessor.class.getName());
+
+    // Properties shared across WETProcessor instances
+    public final int MAX_WORD_LENGTH;
+    public final int MIN_NUM_WORDS;
+    public final int MAX_WORDS_CAMELCASE;
+
+    // constants for the possible fixed values in sites-too-big-to-exhaustively-crawl.txt file
+    public final String SUBDOMAIN_COPY = "SUBDOMAIN-COPY";
+    public final String SINGLEPAGE = "SINGLEPAGE";
+    public final String FOLLOW_LINKS_WITHIN_TOPSITE = "FOLLOW-LINKS-WITHIN-TOPSITE";
+    
+    /**
+     * Characters that need escaping if used as a string literal in a regex
+     * https://stackoverflow.com/questions/399078/what-special-characters-must-be-escaped-in-regular-expressions
+     * https://www.regular-expressions.info/refcharacters.html
+     * Put the \\ (escape char) at start so we don't double-escape chars already escaped,
+     * as would happen for any chars appearing earlier in this list than \\
+    */    
+    public final String ESCAPE_CHARS_FOR_RE = "\\.^$*+?()[{|";
+    //public final String[] ESCAPE_CHARS_FOR_RE = ["\\", ".", "^", "$", "*", "+", "?", "(", ")", "[", "{", "|"];
+    
+    private Properties configProperties = new Properties();
+
+    // File paths shared across WETProcessor instances
+    public final File commoncrawlDir;
+    public final File outputFolder;
+    public final File discardFolder;
+    public final File keepFolder;
+    public final File greyListedFolder;
+    public final File keepURLsFile;
+    public final File discardURLsFile;
+    public final File greyListedFile;
+
+    /** Possible values stored in the blackList/whiteList/greyList Maps */
+    private final Integer LIST_ENTRY_CONTAINS = new Integer(0);
+    private final Integer LIST_ENTRY_STARTSWITH = new Integer(1);
+    private final Integer LIST_ENTRY_ENDSWITH = new Integer(2);
+    private final Integer LIST_ENTRY_MATCHES = new Integer(3);
+
+    /**
+     * Store url patterns as keys and values indicated whether a url should
+     * match it exactly, start/end with it, or contain it
+     */
+    private HashMap<String, Integer> blackList;
+    private HashMap<String, Integer> greyList;
+    private HashMap<String, Integer> whiteList;
+
+    /** map of topsites with allowable regexes: sites too big to exhaustively crawl
+     * with optional regex defining allowed exceptions, like subdomains or url suffixes
+     * off that top site. For example, wikipedia.org is a topsite, but mi.wikipedia.org
+     * is relevant. Or blogspot.com is a top site, but someone's pages in Maori off blogspot
+     * would be relevant.
+     * The map would store top site domain suffix and an optional regex string for allowable
+     * url patterns.
+    */
+    private HashMap<String, String> topSitesMap;
+    
+    /** Map of domains we keep and the full urls we're keeping that are of that domain. 
+     * No need to use a TreeMap which preserves natural (alphabetical) ordering of keys,
+     * while a HashMap has no notion of ordering, because we just need to store urls with
+     * their domains. Whether the domains are sorted or the urls per domain are sorted becomes
+     * irrelevant. (Does it really? What if we have urls followed vs preceded by urls with the
+     * same prefix, e.g. pinky.com/toto/index.html and pinky.com/toto/nono/file.html
+     * Is there any benefit to nutch when crawling if these seedURLs are ordered or not?)
+     */
+    private Map<String, Set<String>> domainsToURLsMap;
+    
+    // Keep a count of all the records that all WETProcessors instantiated
+    // by our main method combined have processed
+    private int totalRecordCount = 0;
+
+    private int wetFileCount = 0;
+
+    private static ClassLoader MY_CLASSLOADER = org.greenstone.atea.CCWETProcessor.class.getClassLoader();
+	
+    public CCWETProcessor(File inFolder, File outFolder) throws Exception {
+	this.commoncrawlDir = inFolder;
+	this.outputFolder = outFolder;
+
+	// load up the properties from the config file
+	try (InputStream infile = MY_CLASSLOADER.getResourceAsStream("config.properties")) {
+	    configProperties = new Properties();
+	    configProperties.load(infile);	    
+	    //infile.close(); // not explicitly called in examples of try-with-resources
+	    
+	} catch(Exception e) {
+	    logger.error("Exception attempting to read properties from config.properties.", e);
+	}
+
+	if(configProperties.size() == 0) {
+	    logger.warn("*** Warning: no values read into config properties. Using defaults.");
+	}
+	
+	MAX_WORD_LENGTH = Integer.parseInt(configProperties.getProperty("WETprocessor.max.word.length", "15"));
+	MIN_NUM_WORDS = Integer.parseInt(configProperties.getProperty("WETprocessor.min.num.words", "20"));
+	MAX_WORDS_CAMELCASE = Integer.parseInt(configProperties.getProperty("WETprocessor.max.words.camelcase", "10"));
+
+
+	this.discardFolder = new File(outFolder, "discard");
+	if(!discardFolder.exists()) {
+	    discardFolder.mkdir();
+	}	
+	this.keepFolder = new File(outFolder, "keep");
+	if(!keepFolder.exists()) {
+	    keepFolder.mkdir();
+	}
+	
+	this.greyListedFolder = new File(outFolder, "greylisted");
+	if(!greyListedFolder.exists()) {
+	    greyListedFolder.mkdir();
+	}
+	
+	this.keepURLsFile = new File(outFolder, "keepURLs.txt");	
+	if(keepURLsFile.exists() && !keepURLsFile.delete()) {
+	    throw new Exception("Warning: Unable to delete " + this.keepURLsFile + ". Unable to proceed.");
+	}
+	this.discardURLsFile = new File(outFolder, "discardURLs.txt");
+	if(discardURLsFile.exists() && !discardURLsFile.delete()) {
+	    throw new Exception ("Warning Unable to delete " + discardURLsFile + ". Unable to proceed.");
+	}
+	this.greyListedFile = new File(outFolder, "greyListed.txt");
+	if(greyListedFile.exists() && !greyListedFile.delete()) {
+	    throw new Exception ("Warning Unable to delete " + greyListedFile + ". Unable to proceed.");
+	}
+
+	// prepare our blacklist, greylist (for inspection) and whitelist
+	logger.info("Loading blacklist.");
+	blackList = new HashMap<String, Integer>();
+	initURLFilterList(blackList, "url-blacklist-filter.txt");
+	
+	logger.info("Loading greylist.");
+	greyList = new HashMap<String, Integer>();
+	initURLFilterList(greyList, "url-greylist-filter.txt");
+	
+	logger.info("Loading whitelist.");
+	whiteList = new HashMap<String, Integer>();
+	initURLFilterList(whiteList, "url-whitelist-filter.txt");
+
+	// Create the map of topSites
+	logger.info("Loading map of topsites with regex of allowable url patterns for each topsite.");
+	topSitesMap = new HashMap<String, String>();
+	
+	// Read in our csv file of topsites and what to do when one hits a match with a seedURL
+	// and put these in our topSitesMap
+	// https://commons.apache.org/proper/commons-csv/apidocs/index.html
+	// https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVParser.html	
+	//https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVFormat.html
+	CSVFormat customisedCSVFormat = CSVFormat.DEFAULT
+	    .withCommentMarker('#')
+	    .withSkipHeaderRecord()
+	    .withIgnoreSurroundingSpaces();
+	
+	File topSitesCSVData = new File(MY_CLASSLOADER.getResource("sites-too-big-to-exhaustively-crawl.txt").getFile());
+	// CSVParser is AutoCloseable and throws exceptions, so putting it in a try-with-resources
+	try (
+	     CSVParser parser = CSVParser.parse(topSitesCSVData, StandardCharsets.UTF_8, customisedCSVFormat);
+	     ) {
+	    for (CSVRecord csvRecord : parser) {
+		String topsite = csvRecord.get(0);
+		String allowed_url_pattern = (csvRecord.size() >= 2) ? csvRecord.get(1) : "";
+		topSitesMap.put(topsite, allowed_url_pattern);
+
+		//logger.debug("@@@@ topsite: " + topsite + " - " + allowed_url_pattern);
+		
+	    }
+	} catch(Exception e) {
+	    logger.error("@@@@@@@@@ Error attempting to parse CSV format of text file " + topSitesCSVData, e);
+	}	
+ 
+	//logger.debug("Prematurely terminating for testing purposes.");
+	//System.exit(-1);
+    }
+
+
+    /** Utility function to help escape regex characters in URL to go into regex-urlfilter.txt */
+    private String escapeStringForRegex(String str) {	
+	for(int i = 0; i < ESCAPE_CHARS_FOR_RE.length(); i++) {
+	    char c = ESCAPE_CHARS_FOR_RE.charAt(i);
+	    str = str.replace(Character.toString(c), "\\"+c);
+	}
+	return str;	
+    }
+    
+    /**
+     * Using the keepURLs.txt file generated by running WETProcessor instances, this produces
+     * as output the URL seed list and regex-urlfilter text files required by nutch, see
+     * https://cwiki.apache.org/confluence/display/nutch/NutchTutorial
+     *
+     * This method creates seedURLs files and url-regexfilter files needed by nutch, instructing
+     * it what portion to crawl of each site.
+     *
+     * The topSiteMatches file also gets created, listing sites excluded from crawling as
+     * they're too large to exhaustively crawl. The user will be told to inspect this file
+     * after this program has finished running.
+     *
+     * If checkForPossibleProductSites, then any urls containing /mi(/) that are outside of NZ
+     * or whose geolocation isn't known will end up in the file denoted by possibleProductSitesFile
+     *
+     */
+    public void prepareSitesForNutchCrawling(File seedURLsFile, File urlFilterFile,
+				    File domainURLsFile, File topSiteMatchesFile,
+				    boolean checkForPossibleProductSites, File possibleProductSitesFile) {
+	// Maintain a Map of unique domains mapped to seed urls at that domain
+	// TreeSet: by default, "the elements are ordered using their natural ordering"
+	// (or by a Comparator provided at set creation time).
+	// Whereas HashSet doesn't guarantee ordering.
+	// So we get alphabetic sorting for free. And guaranteed log(n) for basic operations.
+	// Would be a similar distinction for Maps.
+	domainsToURLsMap = new TreeMap<String, Set<String>>();
+
+	final String PROTOCOL_REGEX_PREFIX = "+^https?://";
+	final String FILTER_REGEX_PREFIX = PROTOCOL_REGEX_PREFIX + "([a-z0-9-]+\\.)*"; // https?://([a-z0-9-]+\.)* for nutch's regex-urlfilter.txt
+
+	// keep an eye out on URLs we need to inspect later
+	Set<String> possibleProductDomains = new TreeSet<String>();
+	File geoLiteCityDatFile = new File(MY_CLASSLOADER.getResource("GeoLiteCity.dat").getFile());
+	
+	try (
+	     BufferedReader reader = new BufferedReader(new FileReader(this.keepURLsFile));
+	     BufferedWriter possibleProductSitesWriter = new BufferedWriter(new FileWriter(possibleProductSitesFile));
+	     ) {
+		
+	    // read a URL at a time from urlsFile
+	    String url = null;
+	    String domainWithProtocol = null;
+	    while((url = reader.readLine()) != null) { // readLine removes newline separator
+		
+		// work out domain. This retains any www. or subdomain prefix
+		// passing true to further also retain the http(s) protocol
+		domainWithProtocol = Utility.getDomainForURL(url, true);
+
+		Set<String> urlsSet;
+		if(!domainsToURLsMap.containsKey(domainWithProtocol)) {
+		    urlsSet = new TreeSet<String>();
+		    urlsSet.add(url);
+		    domainsToURLsMap.put(domainWithProtocol, urlsSet);		    
+		} else {
+		    urlsSet = domainsToURLsMap.get(domainWithProtocol);
+		    urlsSet.add(url);
+		}
+
+		if(checkForPossibleProductSites) {		    
+		    // Dr Nichols said that a url that was located outside the country and
+		    // which had /mi/ URLs was more likely to be an autotranslated (product) site.
+		    // Following Dr Nichols' idea, let's keep a look out for more product sites:
+		    // if any URL contains /mi AND the tld of its domain is outside of New Zealand
+		    // then add that domain (if not already added) and that url into a file
+		    // for later manual inspection
+		    if(!domainWithProtocol.endsWith(".nz")
+		       && (url.contains("/mi/") || url.endsWith("/mi"))) {
+			
+			if(!possibleProductDomains.contains(domainWithProtocol)) {
+			    
+			    String countryCode = "";
+			    try {
+				// more expensive test, so do this only if above conditions are true:
+				countryCode = Utility.getCountryCodeOfDomain(domainWithProtocol, geoLiteCityDatFile);
+				System.err.println("@@@@ Got country code: " + countryCode);
+			    } catch(Exception exceptObj) {
+				countryCode = ""; // forces domain to be included for inspection
+				
+				logger.error("Could not check if domain " + domainWithProtocol
+					     + " was in country: " + countryCode,
+					     exceptObj);
+			    }
+			    
+			    boolean isInNZ = countryCode.toLowerCase().equals("nz");
+			    
+			    
+			    //if(!Utility.isDomainInCountry(domainWithProtocol, "nz", geoLiteCityDatFile)) {
+			    if(!isInNZ) {
+				possibleProductDomains.add(domainWithProtocol);
+				// write both domain and a sample seedURL on that site out to file
+				possibleProductSitesWriter.write(countryCode + " : " + domainWithProtocol + "\n");			    
+				possibleProductSitesWriter.write("\t" + url + "\n");
+			    }
+			}
+			//else {
+			// already wrote out domain to file at some point, write just the URL out to file
+			//possibleProductSitesWriter.write("\t" + url + "\n");
+			//}			
+		    }
+		}
+	    }
+	} catch (IOException ioe) {
+	    logger.error("@@@@@@@@@ Error reading in urls from file " + this.keepURLsFile, ioe);
+	}
+
+	// We'd have pruned out duplicates by now and have a sorted list of domains,
+	// each of which maps to seed URLs in the commoncrawl for that domain
+
+	int domainCount = 0;
+	File sitesFolder = new File(outputFolder, "sites");
+	if(!sitesFolder.exists()) {
+	    sitesFolder.mkdir();
+	}
+	final String FORMATSTR = "%05d";
+	
+	// write out each domain followed in sequence by all urls we found in that domain
+	// (urls with tab up front)
+	try (
+	     // global lists of all domains, seedURLs and regex-urlfilters across all wet files of all commoncrawls
+	     // Also a global file listing any urls that matched top sites that didn't specify
+	     // allowed regex patterns
+	     BufferedWriter domainURLsWriter = new BufferedWriter(new FileWriter(domainURLsFile)); 
+	     BufferedWriter seedURLsWriter = new BufferedWriter(new FileWriter(seedURLsFile));
+	     BufferedWriter urlFilterWriter = new BufferedWriter(new FileWriter(urlFilterFile));
+	     BufferedWriter topSiteMatchesWriter = new BufferedWriter(new FileWriter(topSiteMatchesFile));	     
+	     ) {
+
+	    // initialise topSiteMatchesFile with some instructional text.
+	    topSiteMatchesWriter.write("The following domain with seedURLs are on a major/top 500 site\n");
+	    topSiteMatchesWriter.write("for which no allowed URL pattern regex has been specified.\n");
+	    topSiteMatchesWriter.write("Specify one for this domain in the tab-spaced sites-too-big-to-exhaustively-crawl.txt file\n");
+	    
+	    //Set<Map.Entry<String, Set<String>>> domainsSet = domainsToURLsMap.keySet();
+	    Set<String> domainsSet = domainsToURLsMap.keySet();
+	    Iterator<String> domainIterator = domainsSet.iterator();
+
+	    /*
+	    // DEBUG
+	String value = topSitesMap.get("wikipedia.org");
+	if(value == null) {
+	    logger.debug("### wikipedia.org had null value");
+	} else {
+	    logger.debug("### wikipedia.org had value: " + value);
+	} // DEBUG
+	    */
+	    
+	    while(domainIterator.hasNext()) {
+		String domainWithProtocol = domainIterator.next();
+		// Also get domain without protocol prefix
+		int startIndex = domainWithProtocol.indexOf("//"); // http:// or https:// prefix
+		startIndex = (startIndex == -1) ? 0 : (startIndex+2); // skip past the protocol's // portion
+		String domain = domainWithProtocol.substring(startIndex);
+
+		/*if(domain.contains("docs.google.com")) {
+		    logger.debug("domain with protocol: " + domainWithProtocol);
+		    logger.debug("domain: " + domain);
+		    }*/
+		
+		String allowedURLPatternRegex = isURLinTopSitesMap(domain);		
+		// If the domain is of a topsite for which no allowed URL pattern has been provided
+		// in sites-too-big-to-exhaustively-crawl.txt,
+		// then we don't know how to crawl the site. Warn the user by writing the affected
+		// domain and seedURLs to the topSiteMatchesFile.		
+		if(allowedURLPatternRegex != null && allowedURLPatternRegex.equals("")) {
+		    
+		    // topsite, but we don't (yet) know what portion can be crawled
+		    // Append the top site and url to a global/toplevel file that
+		    // the user needs to check later and we're done with this domain as it
+		    // won't go into any other file hereafter
+
+		    Set<String> urlsForDomainSet = domainsToURLsMap.get(domainWithProtocol);
+		    for(String url : urlsForDomainSet) {
+			topSiteMatchesWriter.write("\t" + url + "\n"); 
+		    }
+		   
+		    continue; // done with this domain
+		}
+
+		// start counting the domains we're actually going to process
+		domainCount++;		
+		
+		String siteID = String.format(FORMATSTR, domainCount);
+		File domainFolder = new File(sitesFolder, siteID);
+		domainFolder.mkdir();
+		
+		// write out the domain		
+		//seedURLsWriter.write(domainWithProtocol + "\n");
+		
+
+		// for every domain, we need a sites/0000x/ folder, where x is domain#, containing
+		// its own INDIVIDUAL seedURLs.txt and regex-urlfilter.txt
+		// We still have a global seedURLs.txt and regex-urlfilter.txt too.
+		File siteSeedsFile = new File(domainFolder, "seedURLs.txt"); // e.g. sites/00001/seedURLs.txt
+		File siteRegexFile = new File(domainFolder, "regex-urlfilter.txt"); // e.g. sites/00001/regex-urlfilter.txt
+		try (
+		     BufferedWriter siteURLsWriter = new BufferedWriter(new FileWriter(siteSeedsFile));
+		     BufferedWriter siteRegexWriter = new BufferedWriter(new FileWriter(siteRegexFile));
+		     ) {
+
+		    // write all sorted unique domains into global domains file
+		    // Using the domain withuot protocol since the global domains file is for
+		    // informational purposes
+		    domainURLsWriter.write(domain + "\n");
+
+		    // Only write urls and no domain into single global seedurls file
+		    // But write domain and tabbed urls into individual sites/0000#/seedURLs.txt
+		    // files (and write regexed domain into each sites/0000#/regex-urlfilter.txt)
+		    // If we ever run nutch on a single seedURLs listing containing
+		    // all seed pages to crawl sites from, the above two files will work for that.
+
+		    // first write out the urls for the domain into the sites/0000x/seedURLs.txt file
+		    // also write into the global seeds file (with a tab prefixed to each?)
+		    Set<String> urlsForDomainSet = domainsToURLsMap.get(domainWithProtocol);
+		    for(String url : urlsForDomainSet) {
+			seedURLsWriter.write(url + "\n"); // global seedURLs file
+			siteURLsWriter.write(url + "\n");
+		    }
+		    
+		    if(allowedURLPatternRegex == null) { // entire site can be crawled
+			siteURLsWriter.write(domainWithProtocol + "\n");
+			
+			// Write out filter in the following form for a site, e.g. for nutch.apache.org:
+			// nutch.apache.org => +^https?://([a-z0-9-]+\.)*nutch\.apache\.org/
+			String regexed_domain = FILTER_REGEX_PREFIX + escapeStringForRegex(domain) + "/";		
+			//String regexed_domain = FILTER_REGEX_PREFIX + domain.replace(".", "\\.") + "/";		
+			urlFilterWriter.write(regexed_domain + "\n"); //global file
+			siteRegexWriter.write(regexed_domain + "\n"); // site file
+		    }
+		    else { // domain belongs to a top site where only portion of site can be crawled
+			
+			if(allowedURLPatternRegex.equals(SUBDOMAIN_COPY)) { // COPY existing domain as url-filter
+			    siteURLsWriter.write(domainWithProtocol + "\n");
+			    // e.g. pinky.blogspot.com will add a filter for pinky.blogspot.com
+			    // and not for all of blogspot.com
+			    
+			    String regexed_domain = PROTOCOL_REGEX_PREFIX+escapeStringForRegex(domain) + "/";
+			    //String regexed_domain = PROTOCOL_REGEX_PREFIX+domain.replace(".", "\\.") + "/";
+			    urlFilterWriter.write(regexed_domain + "\n");
+			    siteRegexWriter.write(regexed_domain + "\n");
+			    
+			} else if(allowedURLPatternRegex.equals(SINGLEPAGE)) {
+			    // don't write out domain. We want individual pages
+			    //DON'T DO THIS HERE: siteURLsWriter.write(domainWithProtocol + "\n");
+			    
+			    // don't write out domain as a regex expression url filter either,
+			    // write out the individual seed urls for the domain instead
+			    // since we will only be downloading the single page
+			    
+			    urlsForDomainSet = domainsToURLsMap.get(domainWithProtocol);
+			    for(String urlInDomain : urlsForDomainSet) {
+				// don't append slash to end this time
+				String regexed_url = "+^"+escapeStringForRegex(urlInDomain);
+				//String regexed_url = "+^"+urlInDomain.replace(".", "\\.");
+				urlFilterWriter.write(regexed_url + "\n");
+				siteRegexWriter.write(regexed_url + "\n");
+			    }
+			} else if(allowedURLPatternRegex.equals(FOLLOW_LINKS_WITHIN_TOPSITE)) {
+			    
+			    // DON'T write out domain into siteURLs file,
+			    // BUT DO write it into urlFilter file
+			    String regexed_domain = PROTOCOL_REGEX_PREFIX + escapeStringForRegex(domain) + "/";
+
+			    urlFilterWriter.write(regexed_domain + "\n");
+			    siteRegexWriter.write(regexed_domain + "\n");
+			} else { // allowedURLPatternRegex is a url-form - convert to regex
+			    if(!allowedURLPatternRegex.endsWith("/")) {
+				allowedURLPatternRegex += "/";
+			    }
+			    String regexed_pattern = PROTOCOL_REGEX_PREFIX+escapeStringForRegex(allowedURLPatternRegex);
+			    //String regexed_pattern = PROTOCOL_REGEX_PREFIX+allowedURLPatternRegex.replace(".", "\\.");
+			    siteURLsWriter.write(domainWithProtocol + "\n");
+			    urlFilterWriter.write(regexed_pattern + "\n");
+			    siteRegexWriter.write(regexed_pattern + "\n");
+
+			}
+		    }
+		    
+		} catch (IOException ioe) {
+		    logger.error("@@@@@@@@@ Error writing to one of:" + siteSeedsFile + " or " + siteRegexFile, ioe);
+		}
+		
+	    }
+
+	} catch (IOException ioe) {
+	    logger.error("\n@@@@@@@@@ Error writing to one of:\n\t" + seedURLsFile
+			       + "\n\t" + urlFilterFile
+			       + "\n\t" + domainURLsFile
+			       + "\n\t" + topSiteMatchesFile, ioe);
+	}	
+
+	/*
+	// BEGIN DEBUG
+	logger.debug("@@@@ TopSitesMap contains: ");
+	for(Map.Entry<String, String> entry : topSitesMap.entrySet()) {
+	    String topSite = entry.getKey();
+	    String urlPattern = entry.getValue();	    
+	    logger.debug(topSite + " - " + urlPattern);
+	} // END DEBUG
+	*/
+    }
+
+    private String stripSubDomain(String url) {
+	int index = url.indexOf(".");
+	if(index != -1) {
+	    url = url.substring(index+1);
+	}
+	return url;
+    }
+
+
+    /** 
+     * @return true when a seedURL's domain exactly matches a topsite such as blogspot.com,
+     * with or without www. prefix. This method tests for such as case as it would be dangerous
+     * to do a SUBDOMAIN-COPY on such a site and thereby crawl that entire domain.
+     */
+    private boolean isExactDomainMatch(String seedURLDomain, String domain) {
+	// check for an exact match as-is
+	if(seedURLDomain.equals(domain)) {
+	    return true;
+	}
+
+	// else check if with or without a www. prefix we have an exact match with domain
+	if(seedURLDomain.startsWith("www.")) {
+	    if(seedURLDomain.substring(4).equals(domain)) {
+		return true;
+	    }
+	} else {
+	    if(domain.equals("www."+seedURLDomain)) {
+		return true;
+	    }
+	}
+
+	return false;
+     }
+    
+
+    /**
+     * Check if the domain of the seedurl, either in its entirety or when stripped of 
+     * www/subdomains, is in the list of top sites.
+     * If it is, and the given url matches the regex for that topsite, then add the url to the
+     * whitelist and a regex disallowing the rest of the topsite to the url regex filter file.
+     * @param fullSeedDomain: domain of seedURL without the protocol. May include www. prefix.
+     * @return one of the following values:
+     *  - This function returns null if the seedURL's domain does not match any of the topsites.
+     *  - The empty String is returned if the seedURL's domain matched a topsite but no (allowed-
+     * url-pattern) value was defined for it. The empty String is also returned if the seedURL's
+     * domain exactly matched a topsite and had a value of SUBDOMAIN-COPY, because we still don't
+     * want to blindly crawl a topsite (as would happen with SUBDOMAIN-COPY).
+     *  - A non-emptry String is returned if the seedURL's domain matched a topsite and a value
+     * was defined for it. (The value will be one of "SUBDOMAIN-COPY", "SINGLEPAGE" or an allowed
+     * URL pattern.
+     */
+    private String isURLinTopSitesMap(String fullSeedDomain) {
+	boolean keepLooping = true;
+
+	String domain = fullSeedDomain;
+	
+	// domain aprameter will have retained www or subdomains, but is stripped of protocol
+	
+	// keep looping, stripping subdomains from url and checking if it matches a topsite domain
+	// if it does, return the value for that topsite domain in the topSitesMap
+	// If no match at all, return null.
+	do {
+	    
+	    String allowed_url_pattern = topSitesMap.get(domain);
+	    if(allowed_url_pattern != null) { // if topSitesMap.containsKey(domain);
+		// there's an entry for the URL in the topSitesMap
+		logger.debug("##### A top site matches URL domain " + domain);
+
+		// if we're dealing with SUBDOMAIN-COPY, then the fullSeedDomain, with or without
+		// www prefix, should not exactly match the topSitesMap domain
+		// e.g. we don't want to crawl a seed URL with domain www.blogspot.com
+		// despite it matching topsite blogspot.com with a value of SUBDOMAIN-COPY.
+		
+		if(allowed_url_pattern.equals(SUBDOMAIN_COPY) && isExactDomainMatch(fullSeedDomain, domain)) {
+		    return ""; // means don't crawl site, write url into unprocessed-topsite-matches file
+		}
+		return allowed_url_pattern;
+	    }
+	    // else, no entry for the URL in the topSitesMap
+	    // We're not done yet: strip subDomain from URL and check it against topSitesMap again
+	    
+	    String newDomain = stripSubDomain(domain);
+	    if(domain.equals(newDomain)) {
+		keepLooping = false;
+	    } else {
+		domain = newDomain;
+	    }
+	} while(keepLooping);
+
+	// url in entirety or stripped of subdomains did not match any of the topsites
+	return null;
+    }
+
+    private boolean isListedInFilterList(Map<String, Integer> filterListMap, String url) {
+	//Set<Map.Entry<String,Integer>> entries = filterListMap.entrySet();
+	//Iterator<Map.Entry<String, Integer>> i = entries.iterator();
+	//while(i.hasNext()) {
+	// Map.Entry<String, Integer> entry = i.next();
+	for(Map.Entry<String,Integer> entry : filterListMap.entrySet()) {
+	    String urlPattern = entry.getKey();
+	    Integer matchRule = entry.getValue();
+
+	    if(matchRule == LIST_ENTRY_CONTAINS && url.contains(urlPattern)) {
+		return true;
+	    }
+	    else if(matchRule == LIST_ENTRY_STARTSWITH && url.startsWith(urlPattern)) {
+		return true;
+	    }
+	    else if(matchRule == LIST_ENTRY_ENDSWITH && url.endsWith(urlPattern)) {
+		return true;
+	    }
+	    else if(matchRule == LIST_ENTRY_MATCHES && url.equals(urlPattern)) {
+		return true;
+	    }
+	    // else check the rest of the filter list against this url
+	    // before returning false to be certain it's not been listed in the filter list
+	}
+	
+	return false;
+    }
+
+    /**
+     * Returns true if the url or pattern is found in the blacklist file.
+     * Note that if eventually the same url pattern is found in the greylist or whitelist too,
+     * it won't get blacklisted after all. But that's not implemented here.
+     */
+    public boolean isBlacklisted(String url) {
+	boolean isBlackListed = isListedInFilterList(blackList, url);
+
+	// if any portion of the URL contains the word "livejasmin", or even "jasmin" actually,
+	// then it's an adult site, so blacklist the entire domain if it wasn't already blacklisted
+	String domainWithoutProtocol = Utility.getDomainForURL(url, false); // remove protocol
+	if(!isBlackListed && url.contains("jasmin")) {
+	    logger.warn("### Blacklisting additional domain (likely an adult site): " + domainWithoutProtocol);
+	    blackList.put(domainWithoutProtocol, LIST_ENTRY_CONTAINS);
+	}
+	return isBlackListed;
+    }
+    
+    /**
+     * Returns true if the url or pattern is explicitly mentioned in the greylist file.
+     * Will eventually take precedence over if the same URL pattern was mentioned in the blacklist.
+     * Will eventually be pre-empted into the whitelist if mentioned in the whitelist.
+     */
+    public boolean isGreylisted(String url) {
+	// auto-translated product sites
+	return isListedInFilterList(greyList, url);
+    }
+
+    /**
+     * Returns true if the url or pattern is explicitly mentioned in the whitelist file
+     * Its mention in a whitelist moreover overrides any mention in the blacklist and greylist.
+     */
+    public boolean isWhitelisted(String url) {
+	return isListedInFilterList(whiteList, url);
+    }
+    
+    /**
+     * Checks URL parameter against each line ("filter") of conf/url-black|grey|whitelist-filter.txt to decide
+     * whether it is in the mentioned black|grey|white list.
+     * Filters don't represent actual regex, just ^ and $ as start and end terminators.
+     * By not having this method deal with actual regex for filters, this has the advantage that
+     * we don't have to remember to escape or double escape each filter to turn it into a regex.
+     */
+    public void initURLFilterList(Map<String, Integer> list, String filterListFilename) {
+	
+	// if filterListFilename does not exist in the conf folder, just return
+	if(MY_CLASSLOADER.getResource(filterListFilename) == null) {
+	    logger.warn("Filter list filename: " + filterListFilename + " does not exist");
+	    return;	    
+	}
+
+	try (
+	     BufferedReader reader = new BufferedReader(new InputStreamReader(MY_CLASSLOADER.getResourceAsStream(filterListFilename), "UTF-8"));
+	     ) {
+	    String filter = null;
+	    while((filter = reader.readLine()) != null) {
+		// skip comments and empty lines
+		filter = filter.trim();
+		if(filter.equals("") || filter.startsWith("#")) {
+		    continue;
+		}
+		
+		if(filter.startsWith("^") && filter.endsWith("$")) {
+		    filter = filter.substring(1, filter.length()-1);
+		    list.put(filter, LIST_ENTRY_MATCHES);
+		}
+		else if(filter.startsWith("^")) {
+		    filter = filter.substring(1);
+		    list.put(filter, LIST_ENTRY_STARTSWITH);
+		    //logger.debug("Match filter startswith: " + filter);
+		}
+		else if(filter.endsWith("$")) {
+		    filter = filter.substring(0, filter.length()-1);
+		    list.put(filter, LIST_ENTRY_ENDSWITH);
+		    //logger.debug("@@@ Match filter endswith: " + filter);
+		}
+		else {
+		    list.put(filter, LIST_ENTRY_CONTAINS);
+		}
+		//logger.debug("Got filter: " + filter);
+	    }
+		
+	} catch (IOException ioe) {
+	    logger.error("@@@@@@@@@ Error reading into map from file " + filterListFilename, ioe);
+	}
+	
+    }
+
+    /** Maintain a count of all WET files processed. */
+    public void setWETFileCount(int count) { this.wetFileCount = count; }
+    
+    /** Maintain a count of all WET records processed. */
+    //public int getRecordCount() { return this.totalRecordCount; }
+    //public void addToRecordCount(int count) { this.totalRecordCount += count; }
+    public void setRecordCount(int count) { this.totalRecordCount = count; }
+
+    public void processAllWETFilesOfCrawl(File ccrawlWETFileDir) {
+	
+	// Will list all the warc.wet files in the input directory or else their gzipped versions
+	File[] WETFiles = ccrawlWETFileDir.listFiles(new WETFilenameFilter());
+
+	int wetRecordCount = 0;
+	int wetFileCount = 0;
+	
+	for(int i = 0; i < WETFiles.length; i++) {
+	    File WETFile = WETFiles[i];	    
+	    logger.debug("Processing WETfile: " + WETFile);
+
+	    // Any .gz files listed means they haven't been unzipped yet. So unzip.
+	    String WETFilename = WETFile.toString();
+	    if(WETFilename.endsWith(".gz")) {
+		File GZippedWETFile = WETFile;
+		String WETGZippedFilename = WETFilename;
+		WETFilename = WETFilename.substring(0, WETFilename.lastIndexOf(".gz"));		
+
+		WETFile = new File(WETFilename);
+		Utility.unzipFile(GZippedWETFile, WETFile);
+	    }
+	    // hereafter all WETFiles should refer to the unzipped version
+	    // Check the unzipped WETFile exists	    
+
+	    if(!WETFile.exists() || !WETFile.isFile()) {		
+		logger.error("Error: " + WETFile + " does not exist (failure to unzip?)");
+		return;
+	    }
+
+	    // Finally, we can process this WETFile's records into the keep and discard pile
+	    wetFileCount++;
+	    logger.debug("Off to process " + WETFile);
+	    String crawlID = ccrawlWETFileDir.getName(); // something like CC-MAIN-YYYY-##-wet-files
+	    crawlID = crawlID.substring("CC-MAIN-".length(), crawlID.indexOf("-wet-files")); // YYYY-##	    
+	    WETProcessor wetFileProcessor = new WETProcessor(WETFile, crawlID, this);
+	    wetFileProcessor.processWETFile();
+	    wetRecordCount += wetFileProcessor.getRecordCount();
+	}
+
+	// for information purposes
+	this.setWETFileCount(wetFileCount);
+	this.setRecordCount(wetRecordCount);
+    }
+
+
+    // --------------- STATIC METHODS AND INNER CLASSED USED BY MAIN -------------- //    
+    public static void printUsage() {
+	System.err.println("Run this program as:");
+	System.err.println("\tCCWetProcessor <path to 'ccrawl-data' input folder> <output folder path> [--check-for-product-sites]");	
+    }
+
+    /** Filename filter to only list warc.wet files or else warc.wet.gz files
+     * for which unzipped warc.wet equivalents don't yet exist.
+     */
+    private static class WETFilenameFilter implements FilenameFilter {
+	
+	public boolean accept(File dir, String name) {
+	    if(name.endsWith(".warc.wet")) {
+		logger.debug("Will include " + name + " for processing.");
+		return true;
+	    }
+
+	    if(name.endsWith(".warc.wet.gz")) {
+		String nameWithoutGZext = name.substring(0, name.lastIndexOf(".gz"));
+		File unzippedVersion = new File(dir, nameWithoutGZext);
+		if(unzippedVersion.exists()) {
+		    logger.debug("--- Unzipped version " + unzippedVersion + " exists.");
+		    logger.debug("Skipping " + name);
+		    return false; // don't count gzipped version if unzipped version exists.
+		}
+		else {
+		    logger.debug("Only zipped version " + name + " exists.");
+		    return true; // No unzipped version, so have to work with gzipped version
+		}
+	    }
+
+	    // we're not even interested in any other file extensions
+	    logger.debug("Not a WET file. Skipping " + name);
+	    return false;
+	}
+    }
+
+
+    private static class CCrawlWETFolderFilenameFilter implements FilenameFilter {
+	
+	public boolean accept(File dir, String name) {
+	    File f = new File (dir, name);
+	    if(f.isDirectory()) {
+		if(name.matches("CC-MAIN-\\d{4}-\\d{2}-wet-files")) {
+		    return true;
+		}
+	    }
+	    else {
+		logger.info("File " + f + " is not a directory");
+	    }
+	    return false;				   
+	}
+    }
+
+    public static void main(String[] args) {
+	if(args.length < 2 || args.length > 3) {
+	    printUsage();
+	    return;
+	}
+	
+	boolean checkForPossibleProductSites = false;
+	if(args.length == 3) {
+	    if(!args[2].equals("--check-for-product-sites")) {
+		printUsage();
+		return;
+	    } else {
+		checkForPossibleProductSites = true;
+	    }
+	}
+	
+	File commoncrawlDir = new File(args[0]);
+	if(!commoncrawlDir.exists() || !commoncrawlDir.isDirectory()) {
+	    logger.error("Error: " + args[0] + " does not exist or is not a directory");
+	    return;
+	}
+	
+	File outFolder = new File(args[1]);
+	if(!outFolder.exists() || !outFolder.isDirectory()) {
+	    logger.error("Error: " + args[1] + " does not exist or is not a directory.");
+	    return;
+	}
+
+
+    try {
+	CCWETProcessor ccWETFilesProcessor = new CCWETProcessor(commoncrawlDir, outFolder);
+
+	File[] ccrawlFolders = commoncrawlDir.listFiles(new CCrawlWETFolderFilenameFilter());
+	
+	for(int i = 0; i < ccrawlFolders.length; i++) {
+	    File ccrawlFolder = ccrawlFolders[i];
+	    logger.info("About to process commoncrawl WET files folder: " + ccrawlFolder);
+	    ccWETFilesProcessor.processAllWETFilesOfCrawl(ccrawlFolder);	    
+	}
+	
+	// create the global files of all domains, seedURLs and regex-urlfilters across all wet files of all commoncrawls
+	// The former is the only unique one. seedURLs and regex-urlfilters are
+	// repeated on a per site/domain basis too, stored in the sites folder
+	File seedURLsFile = new File(outFolder, "seedURLs.txt");
+	File urlFilterFile = new File(outFolder, "regex-urlfilter.txt");
+	File domainURLsFile = new File(outFolder, "all-domain-urls.txt");
+	File topSitesMatchedFile = new File(outFolder, "unprocessed-topsite-matches.txt");
+	File possibleProductSitesFile = new File(outFolder, "possible-product-sites.txt");
+
+	
+	ccWETFilesProcessor.prepareSitesForNutchCrawling(seedURLsFile, urlFilterFile, domainURLsFile, topSitesMatchedFile, checkForPossibleProductSites, possibleProductSitesFile);
+
+	logger.info("\n*** Inspect urls in greylist at " + ccWETFilesProcessor.greyListedFile + "\n");
+
+	if(checkForPossibleProductSites) {
+	    logger.info("\n*** Check " + topSitesMatchedFile + " for sites not prepared for crawling because they matched top sites for which no regex of allowed url patterns were specified in sites-too-big-to-exhaustively-crawl.txt.\n");
+	} else {
+	    possibleProductSitesFile.delete();
+	}
+	
+	
+    } catch(Exception e) {
+	// can get an exception when instantiating CCWETProcessor instance
+	logger.error(e.getMessage(), e);
+    }
+	
+	return;	
+    
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/MRIWebPageStats.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/MRIWebPageStats.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/MRIWebPageStats.java	(revision 33635)
@@ -0,0 +1,58 @@
+package org.greenstone.atea;
+
+
+//import org.apache.log4j.Logger;
+
+
+public class MRIWebPageStats {
+    //private static Logger logger = Logger.getLogger(org.greenstone.atea.MRIWebPageStats.class.getName());
+
+    public final String siteID; // crawled site's folder name e.g. 00510
+    public final String URL; // URL of webpage
+    public final int pageID; // index into NutchTextDumpProcessor::pages ArrayList
+
+    public final boolean isMRI;
+    public final int numSentences; // count of all sentences in the webpage's body
+    public final int numSentencesInMRI; // count of sentences in the webpage's body in MÄori (mri)
+
+    
+    public MRIWebPageStats(String siteID, String url, int pageID, boolean isMRI,
+			   int numSentences, int numSentencesInMRI)
+    {
+	this.siteID = siteID;
+	this.URL = url;
+	this.pageID = pageID;
+
+	this.isMRI = isMRI;
+	this.numSentences = numSentences;
+	this.numSentencesInMRI = numSentencesInMRI;
+    }
+
+    public String toString() {
+	StringBuilder str = new StringBuilder();
+	str.append("URL: " + this.URL);
+	str.append("\nsiteID: " + this.siteID);
+	str.append("\nnum sentences in MRI: " + this.numSentencesInMRI+"/"+this.numSentences);	
+	if(this.isMRI && this.numSentencesInMRI <= 0) {
+	    // one or more pages in the site were MRI, but they didn't contain proper sentences
+	    str.append(" (no PROPER sentences in MRI)");
+	}
+	return str.toString();
+    }
+
+    /** for converting to csv */
+    /*
+      Unused.
+    public String[] toCSV() {
+	String[] csvRecord = { Integer.toString(pageID),
+			       siteID, // foreign key
+			       URL,
+			       Boolean.toString(isMRI),
+			       Integer.toString(numSentences),
+			       Integer.toString(numSentencesInMRI)
+	};
+
+	return csvRecord;
+    }
+    */
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/MaoriTextDetector.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/MaoriTextDetector.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/MaoriTextDetector.java	(revision 33635)
@@ -0,0 +1,305 @@
+/**
+ * Class that uses OpenNLP with the Language Detection Model to determine, with a default
+ * or configurable level of confidence, whether text (from a file or stdin) is in MÄori or not.
+ * July 2019.
+ *
+ * Oct 2019:
+ * - Uses a Sentence Model that we trained for MÄori (see bin/script/gen_SentenceDetection_model.sh)
+ * for being able to split MÄori language text into sentences.
+ * - Refactored into TextLanguageDetector as base class with this class now inheriting from it.
+ */
+
+package org.greenstone.atea;
+
+
+import opennlp.tools.langdetect.*;
+import opennlp.tools.sentdetect.*;
+import opennlp.tools.util.*;
+
+import java.io.*;
+import java.util.ArrayList;
+
+//import org.apache.log4j.Logger;
+
+
+/**
+ * EXPORT OPENNLP_HOME environment variable to be your apache OpenNLP installation.
+ * Create a folder called "models" within the $OPENNLP_HOME folder, and put the file "langdetect-183.bin" in there
+ *    (which is the language detection model zipped up and renamed to .bin extension).
+ * Ensure that the mri-sent_trained.bin sentence model for MÄori that we trained also lives
+ * in the "models" folder.
+ *
+ * Then, to compile this program, do the following from the "src" folder (the folder containing this java file):
+ *    maori-lang-detection/src$ javac -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/MaoriTextDetector.java
+ *
+ * To run this program, issue one of the following commands from the "src" folder (the folder containing this java file):
+ *
+ *    maori-lang-detection/src$ java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org.greenstone.atea.MaoriTextDetector --help
+ *
+ *    maori-lang-detection/src$ java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org.greenstone.atea.MaoriTextDetector --file &lt;full/path/to/textfile&gt;
+ *
+ *    maori-lang-detection/src$ java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org.greenstone.atea.MaoriTextDetector -
+ *       Press enter. This variant of the program expects text to stream in from standard input.
+ *       If entering text manually, then remember to press Ctrl-D to indicate the usual end of StdIn.
+ *
+ * https://stackoverflow.com/questions/219585/including-all-the-jars-in-a-directory-within-the-java-classpath
+ * Also has information on how to run this class if it's in a Java package.
+ */
+public class MaoriTextDetector extends TextLanguageDetector {
+    //static Logger logger = Logger.getLogger(org.greenstone.atea.MaoriTextDetector.class.getName());
+    
+    /** The 3 letter language code for Maori in ISO 639-2 or ISO 639-3 */ 
+    public static final String MAORI_3LETTER_CODE = "mri";
+
+    /** String taken from our university website, https://www.waikato.ac.nz/maori/ */
+    public static final String TEST_MRI_INPUT_TEXT = "Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei.";
+
+    /** test input string for a negative result */
+    public static final String TEST_ENG_INPUT_TEXT = "The main program exits with -1 if an Exception occurred when attempting to detect the text's language";
+    
+    /** Constructor with default confidence for language detection.
+     * Uses the trained Maori sentence model.
+     */
+    public MaoriTextDetector(boolean silentMode) throws Exception {
+	super(silentMode, DEFAULT_MINIMUM_CONFIDENCE, "mri-sent_trained.bin");
+    }
+
+    /** Constructor with configurable confidence level in language detection
+     * that uses the sentence Model we trained for MÄori */
+    public MaoriTextDetector(boolean silentMode, double min_confidence) throws Exception {
+	super(silentMode, min_confidence, "mri-sent_trained.bin");
+    }
+
+    /**
+     * Function that takes a text and returns those sentences in MÄori.
+     * @param text: the string of text from which sentences in the requested
+     * language are to be identified and returned.
+     * @return an ArrayList where:
+     *   - the first element is the total number of sentences in the text parameter
+     *   - remaining elements are the sentences in the text parameter that were in the
+     *     requested language.
+     */
+    public ArrayList<String> getAllSentencesInMaori(String text) {
+	// big assumption here: that we can split incoming text into sentences
+	// for any language (using the MÄori language trained sentence model),
+	// despite not knowing what language those sentences are in
+	// Hinges on MRI sentences detection being similar to at least ENG equivalent
+
+
+	// we'll be storing just those sentences in the text that are in MÄori. 
+	
+	// OpenNLP language detection works best with a minimum of 2 sentences
+	// See https://opennlp.apache.org/news/model-langdetect-183.html
+	// "It is important to note that this model is trained for and works well with
+	// longer texts that have at least 2 sentences or more from the same language."
+	
+	// For evaluating single languages, I used a very small data set and found that
+	// if the primary language detected is MRI AND if the confidence is >= 0.1, the
+	// results appear reasonably to be in te reo MÄori.
+	
+	final double confidenceCutoff = 0.1;
+	return getAllSentencesInLanguage(MAORI_3LETTER_CODE, text, confidenceCutoff);
+    }
+
+    
+    /**
+     * @return true if the input text is Maori (mri) with MINIMUM_CONFIDENCE levels of confidence (if set,
+     * else DEFAULT_MINIMUM_CONFIDENCE levels of confidence).
+     */
+    public boolean isTextInMaori(String text) {	
+	return isTextInLanguage(MAORI_3LETTER_CODE, text);
+    }
+
+    
+    /**
+     * Handle "smaller" textfiles/streams of text read in.
+     * Return value is the same as for isTextInMaori(String text);
+     */
+    public boolean isTextInMaori(BufferedReader reader) throws Exception {
+	return isTextInLanguage(MAORI_3LETTER_CODE, reader);
+    }
+    
+    /*
+     * Need better handling of "larger" textfiles/streams of text read in:
+     * what if multiple languages with high confidence every NUM_LINES read in?
+     * Does this mean the file is multi-lingual with each section dominated by a different language?
+     * How best to convey such information to the user?
+     */
+    /**
+     * Rudimentary attempt to deal with very large files.
+     * Return value is the same as for isTextInMaori(String text);
+     */
+    public boolean isLargeTextInMaori(BufferedReader reader) throws Exception {
+	return isLargeTextInLanguage(MAORI_3LETTER_CODE, reader);
+    }
+
+
+    /********** STATIC METHODS *************/
+
+    public static void doPrint(boolean runSilent, String msg) {
+	if(!runSilent) System.out.println(msg);	
+    }
+    public static void doPrintErr(boolean runSilent, String msg) {
+	if(!runSilent) System.err.println(msg);
+    }
+    
+    public static void printUsage() {
+	System.err.println("Run this program with:");
+	System.err.println("\t--help (-h)\tfor seeing this usage message again");
+	System.err.println("\t-\tto have input text read from STDIN (as always, hit Ctrl-D to mark end of text stream)");
+	System.err.println("\t--file (-f)\tto provide an input file path");
+	System.err.println("\t--silent (-s): optional, to run silently and just exit with exit value.");
+	System.err.println("\t--min-confidence (-c): optional, to override the default minimum confidence value (" + DEFAULT_MINIMUM_CONFIDENCE + ")");
+	System.err.println("\t\tof the predicted language that will be considered acceptable.");
+	System.err.println();
+	System.err.println("This program terminates with exit value:");
+	System.err.println("\t0 if the input text is in Maori");
+	System.err.println("\t1 if input text is not in Maori");
+	System.err.println();
+	System.err.println("\t-1 if the input arguments were wrong");
+	System.err.println("\t255(!) if an Exception occurred in instantiating the MaoriTextDetector when attempting to detect the text's language");
+	System.err.println("\t2 if the user asked to run this program with --help/-h.");
+	System.err.println();
+    }
+
+    /**
+     * The main program exits with:
+     *    0 if text is in Maori;
+     *    1 if text is not in Maori;
+     *
+     *    -1 if the input arguments were wrong
+     *    255(!) if an Exception occurred in instantiating the MaoriTextDetector when attempting to detect the text's language
+     * QTODO: why does the program exit value end up as 255 and not -1 when returnVal remains at -1 on Exception?
+     *    2 if the user asked to run this program with --help/-h.
+     */
+    public static void main(String args[]) {
+	int returnVal = -1;
+
+	// 1. Check input arguments
+	boolean printUsage = false;
+	boolean readFromStdIn = false;
+	File inFile = null;
+	boolean runSilent = false;
+	double minConfidence = -1;
+
+	for (int i = 0; !printUsage && i < args.length; i++) {
+	    
+	    // check for help first and quit after printing usage
+	    if(args[i].equals("--help") || args[i].equals("-h")) {
+		printUsage = true;
+		returnVal = 2;		
+	    } else if(args[i].equals("--silent") || args[i].equals("-s")) {
+		runSilent = true;
+	    } else if(args[i].equals("--min-confidence") || args[i].equals("-c")) {
+		i++;
+		if(i >= args.length) {
+		    doPrintErr(runSilent, "ERROR: No minimum confidence value provided with --min-confidence|-c flag.\n");
+		    printUsage = true;
+		    returnVal = -1;
+		} else {
+		    try {
+			minConfidence = Double.parseDouble(args[i]);
+			if(minConfidence < 0 || minConfidence > 1) {
+			    throw new NumberFormatException("Number out of range, must be between 0-1");
+			}
+		    } catch(NumberFormatException nfe) {
+			doPrintErr(runSilent, "ERROR: value for min-confidence is the wrong format or out of range. It must be a (decimal point) number between 0-1.\n");
+			printUsage = true;
+			returnVal = -1;
+		    }
+		}
+	    } else if(args[i].equals("-")) {
+		readFromStdIn = true;
+		//break; // don't bother continuing to check input arguments for any --file flag if we're told to read from stdin
+	    } else if(args[i].equals("--file") || args[i].equals("-f")) {
+		i++;
+		if(i >= args.length) {
+		    doPrintErr(runSilent, "ERROR: No input file provided with --file|-f flag.\n");
+		    printUsage = true;
+		    returnVal = -1;
+		} else {
+		    String filePath = args[i];
+		    inFile = new File(filePath);
+		    if(!inFile.isFile()) {
+			doPrintErr(runSilent, "ERROR: Can't read text. Input file argument provided does not exist or is not a file.\n");
+			printUsage = true;
+			returnVal = -1;
+		    }
+		}
+	    } else { // unrecognised input argument
+		doPrintErr(runSilent, "ERROR: Unrecognised " + i + "th argument to this program.\n");
+		printUsage = true;
+		returnVal = -1;
+	    }
+	}
+
+	if(returnVal != 2) { // returnVal == 2 for help. Only if the user did not request --help/-h, do we continue to make sure the arguments provided are sane
+	    if(!readFromStdIn && inFile == null) { // at least one input source must be provided
+		doPrintErr(runSilent, "ERROR: must specify source to read text from, either STDIN (-) or input file (--file <file>).\n");
+		printUsage = true;
+		returnVal = -1;
+	    }
+	    
+	    if(readFromStdIn && inFile != null) { // this program can't be asked to read from stdin and from an input file
+		doPrintErr(runSilent, "ERROR: instructed to read from both STDIN and from an input file. Not possible.\n");
+		printUsage = true;
+		returnVal = -1;
+	    }
+	}
+	
+	if(printUsage) {
+	    // If not running silent print usage.
+	    // OR if expressly asked for help, then it doesn't matter if we're running silent: still print usage to stderr.
+	    if(returnVal == 2 || !runSilent) {			
+		printUsage();
+	    }
+	    System.exit(returnVal);
+	}	
+
+	
+	// 2. Finally, we can now do the actual language detection
+	try {
+	    MaoriTextDetector maoriTextDetector = null;
+	    if(minConfidence == -1) {
+		maoriTextDetector = new MaoriTextDetector(runSilent);
+	    } else {
+		maoriTextDetector = new MaoriTextDetector(runSilent, minConfidence);
+	    }
+
+	    
+	    //maoriTextDetector.getAllSentencesInMaori();
+
+	    
+	    //boolean textIsInMaori = maoriTextDetector.isTextInMaori(TEST_MRI_INPUT_TEXT); // test hardcoded string
+	    boolean textIsInMaori = false;
+	    
+	    // Using try with resources, https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html
+	    if(inFile != null) {
+		doPrint(runSilent, "Reading text from file " + inFile.getPath());
+		try (BufferedReader reader = new BufferedReader(new FileReader(inFile))) {
+		    textIsInMaori = maoriTextDetector.isTextInMaori(reader);
+		} // let outer try deal with any file/reading exceptions
+	    }
+	    else if (readFromStdIn) {
+		doPrint(runSilent, "Waiting to read text from STDIN... (press Ctrl-D when done entering text)>");
+		try (BufferedReader reader = new BufferedReader(new InputStreamReader(System.in))) {
+		    textIsInMaori = maoriTextDetector.isTextInMaori(reader);		    
+		} // let outer try deal with any file/reading exceptions
+	    }
+	
+	    if(textIsInMaori) {
+		returnVal = 0;
+	    } else {
+		returnVal = 1;
+	    }
+	    
+	} catch(Exception e) {
+	    e.printStackTrace();
+	    
+	} finally {
+	    doPrint(runSilent, "Exitting program with returnVal " + returnVal + "...\n");
+	    System.exit(returnVal);
+	}
+    }
+    
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/MongoDBAccess.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/MongoDBAccess.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/MongoDBAccess.java	(revision 33635)
@@ -0,0 +1,249 @@
+package org.greenstone.atea;
+
+
+import com.mongodb.client.MongoCollection; 
+import com.mongodb.client.MongoDatabase; 
+//import com.mongodb.client.MongoIterable;
+import com.mongodb.BasicDBObject;
+import com.mongodb.MongoClient; 
+import com.mongodb.MongoCredential;
+import com.mongodb.ServerAddress;
+import com.mongodb.MongoClientOptions;
+
+import org.bson.Document;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+
+import org.apache.log4j.Logger;
+
+
+/** 
+ * https://www.tutorialspoint.com/mongodb/mongodb_java.htm
+ *
+ * TO COMPILE:
+ *    maori-lang-detection/src$
+ *       javac -cp ".:../conf:../lib/*" org/greenstone/atea/MongoDBAccess.java
+ *
+ * TO RUN:
+ *       java -cp ".:../conf:../lib/*" org.greenstone.atea.MongoDBAccess
+ *
+ * Manually connecting to mongodb from client:
+ *    mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u USERNAME -p
+ * Then after connecting with pwd, type:
+ *    use DBNAME
+ *
+ * Or connect to mongodb and specify db in one statement:
+ *    mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017/DBNAME?authSource=admin' -u USERNAME -p
+ *
+ * Some links:
+ *   - https://stackoverflow.com/questions/19938153/do-i-need-to-explicitly-close-connection
+ *   - https://docs.mongodb.com/manual/reference/glossary/ (particularly "collection")
+ *   - https://tecadmin.net/tutorial/mongodb/drop-collection/
+ * IMPORTANT LINK:
+ *   - https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-1
+ *
+ */
+public class MongoDBAccess implements AutoCloseable {
+
+    private static Logger logger = Logger.getLogger(org.greenstone.atea.MongoDBAccess.class.getName());
+    
+    static final String PROPS_FILENAME = "config.properties";
+    public static final String WEBPAGES_COLLECTION = "webpages";
+    public static final String WEBSITES_COLLECTION = "websites";    
+
+    // configuration details, some with fallback values
+    private String HOST = "localhost";
+    private int PORT = 27017; // mongodb port
+    private String USERNAME;
+    private String PASSWORD;    
+    private String DB_NAME ="ateacrawldata";
+    
+    private MongoClient mongo = null;
+    private MongoDatabase database = null;
+    
+    
+    public MongoDBAccess() throws Exception {
+	boolean success = false;
+	
+	// Read in the username and password from our props file
+	Properties props = new Properties();
+
+	//File propsFile = new File(PROPS_FILENAME);
+	//logger.debug("*** Conf props filename: " + propsFile.getAbsolutePath());
+	try {
+	    props.load(getClass().getClassLoader().getResourceAsStream(PROPS_FILENAME));	    
+	} catch(Exception e) {
+	    logger.error(e);
+	}
+	
+	
+	USERNAME = props.getProperty("mongodb.user", "");
+	if(USERNAME.equals("")) {
+	    USERNAME = "root";
+	    logger.warn("WARNING: No sensible value for mongodb.user specified in " + PROPS_FILENAME + ". Attempting to use: " + USERNAME);
+			}
+	PASSWORD = props.getProperty("mongodb.pwd");
+
+	logger.debug("Got pwd: " + PASSWORD);
+	
+	if(PASSWORD != null && PASSWORD.equals("CHANGEME")) {
+	    
+	    success = false;
+	    throw new Exception("************ FATAL ERROR: Change DB password in properties file " + PROPS_FILENAME);	    
+	}
+
+	HOST = props.getProperty("mongodb.host", HOST);
+	String port = props.getProperty("mongodb.port", Integer.toString(PORT));
+	PORT = Integer.parseInt(port);
+	DB_NAME = props.getProperty("mongodb.dbname", DB_NAME);
+
+	logger.info("Connecting to mongodb with:");
+	logger.info(" - host:    " + HOST);
+	logger.info(" - port:    " + PORT);
+	logger.info(" - user:    " + USERNAME);
+	logger.info(" - db name: " + DB_NAME);	
+    }
+
+    /** 
+     * Since we have only a single MongoClient, don't need to call close/disconnect on it as per
+     * https://stackoverflow.com/questions/19938153/do-i-need-to-explicitly-close-connection
+     */
+    public void connectToDB() throws Exception {
+	
+	// Creating a Mongo client 
+	mongo = new MongoClient( HOST, PORT ); 
+	
+	// Creating Credentials 
+	MongoCredential credential; 
+	credential = MongoCredential.createCredential(USERNAME, DB_NAME, PASSWORD.toCharArray()); 
+	System.out.println("Connected to the database successfully");  
+	
+	// Accessing the database 
+	this.database = mongo.getDatabase(DB_NAME); 
+	logger.info("Credentials: "+ credential);
+
+	/*
+	MongoCredential credential; 
+	credential = MongoCredential.createCredential(USERNAME, DB_NAME, PASSWORD.toCharArray());
+	logger.info("Credentials: "+ credential);
+	
+	// Create our Mongo client
+	mongo = new MongoClient( new ServerAddress(HOST, PORT), credential, new MongoClientOptions.Builder().build());
+	System.out.println("Connected to the database successfully");
+
+	this.database = mongo.getDatabase(DB_NAME); 
+	*/
+	
+    }
+
+    // TODO: which fields should be indexed?
+
+    public void showCollections() {
+	//MongoIterable<String> colls = this.database.listCollectionNames();
+	for(String coll : this.database.listCollectionNames()) {
+	    System.err.println("coll: " + coll);
+	}
+    }
+    
+    
+    public void insertWebsiteInfo(WebsiteInfo website)
+    {
+	MongoCollection<Document> collection = this.database.getCollection(WEBSITES_COLLECTION);
+	Document document = new Document("_id", website.id)
+	    .append("siteFolderName", website.siteFolderName) 
+	    .append("domain", website.domain) 
+	    .append("totalPages", website.totalPages)
+	    .append("numPagesWithBodyText", website.countOfWebPagesWithBodyText)
+	    .append("numPagesInMRI", website.numPagesInMRI)
+	    .append("siteCrawledTimestamp", website.siteCrawledTimestamp)
+	    .append("siteCrawlUnfinished", website.siteCrawlUnfinished)
+	    .append("redoCrawl", website.redoCrawl);
+
+	document.put("urlContainsLangCodeInpath", website.urlContainsLangCodeInpath);
+	if(website.geoLocationCountryCode != null && !website.geoLocationCountryCode.equals("")) {
+	    document.put("countryCode", website.geoLocationCountryCode);
+	}
+
+	collection.insertOne(document); 
+	logger.debug("Website info for " + website.id + "(" + website.siteFolderName + ")"
+			   + " inserted successfully into " + WEBSITES_COLLECTION);  
+    }
+
+    /* TODO:
+    https://stackoverflow.com/questions/39433775/mongodb-java-inserting-throws-org-bson-codecs-configuration-codecconfigurationex
+    */
+    public void insertWebpageInfo(WebpageInfo webpage)
+    {
+	// load the webpages db 'table'
+	// in mongodb, the equivalent of db tables are called 'collections'
+	MongoCollection<Document> collection = this.database.getCollection(WEBPAGES_COLLECTION);
+
+	Document document = new Document("_id", webpage.webpageID)
+	    .append("siteid", webpage.websiteID)
+	    .append("url", webpage.URL)
+	    .append("isMRI", webpage.isMRI)
+	    .append("totalSentences", webpage.totalSentences)
+	    .append("charEncoding", webpage.charEncoding) 
+	    .append("modTime", webpage.modifiedTime)
+	    .append("fetchTime", webpage.fetchTime);
+
+	// DOESN'T WORK, AS EXPECTED, BUT DIDN'T KNOW HOW TO DO IT:
+	//document.put("singleSentences", webpage.singleSentences);
+	//document.put("overlappingSentences", webpage.overlappingSentences);
+
+	// INSTEAD, ARRAY OF OBJECTS TO BE INSERTED AS PER:
+	// https://stackoverflow.com/questions/15371839/how-to-add-an-array-to-a-mongodb-document-using-java
+	List<BasicDBObject> sentencesList = new ArrayList<>();
+	for(SentenceInfo sentence : webpage.singleSentences) {
+	    sentencesList.add(new BasicDBObject("langCode", sentence.langCode));
+	    sentencesList.add(new BasicDBObject("confidence", sentence.confidenceLevel));
+	    sentencesList.add(new BasicDBObject("sentence", sentence));
+	}	
+	document.put("singleSentences", sentencesList);
+	
+	List<BasicDBObject> overlappingSentencesList = new ArrayList<>();
+	for(SentenceInfo sentence : webpage.overlappingSentences) {
+	    sentencesList.add(new BasicDBObject("langCode", sentence.langCode));
+	    sentencesList.add(new BasicDBObject("confidence", sentence.confidenceLevel));
+	    sentencesList.add(new BasicDBObject("sentence", sentence));
+	}	
+	document.put("singleSentences", overlappingSentencesList);
+
+	// also put the full text in there
+	document.put("text", webpage.text);
+	
+	collection.insertOne(document);
+	logger.debug("\nwebpage info for " + webpage.webpageID + " inserted successfully into " + WEBPAGES_COLLECTION);  
+    }
+
+    /** https://stackoverflow.com/questions/19938153/do-i-need-to-explicitly-close-connection */
+    public void close() {}
+    
+
+    // TODO:
+    // In the database, need to ensure we have else
+    // create collection (table in RDBMS) websites, create collection webpages.
+    // The webpages collection will have sentences embedded based on my decisions from
+    // reading the series
+    // https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-1    
+    // Then need functions:
+    // insertWebsiteDocument()
+    // insertWebpageDocument()
+    
+    public static void main(String args[]) {
+	try {
+	    MongoDBAccess mongodbCon = new MongoDBAccess();
+	    mongodbCon.connectToDB();
+	    mongodbCon.showCollections();
+
+	} catch(Exception e) {
+	    e.printStackTrace();
+	}
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/NZTLDProcessor.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/NZTLDProcessor.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/NZTLDProcessor.java	(revision 33635)
@@ -0,0 +1,491 @@
+package org.greenstone.atea;
+
+import java.io.*;
+import java.util.Properties;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.log4j.Logger;
+import org.greenstone.util.SafeProcess; // from GS3's gutil.jar. For safely running wget/any process from Java.
+
+
+
+/**
+ * Ensure you have OPENNLP_HOME set to apache-opennlp's full path.
+ * Then go into the src folder of this extension before compiling or running.
+ * 
+ * Compile as:
+ *     maori-lang-detection/src$ javac -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar:../lib/*" org/greenstone/atea/NZTLDProcessor.java
+ * 
+ * Run as:
+ *     maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar:../conf:../lib/*" org.greenstone.atea.NZTLDProcessor ../uniq-tld-nz-WET-urls-2019-08-13.txt
+ */
+public class NZTLDProcessor
+{
+
+    private static Logger logger = Logger.getLogger(org.greenstone.atea.NZTLDProcessor.class.getName());
+    
+    private Properties configProps = null;
+    private boolean DEBUG_MODE = true;
+
+    private final ExtractableTextFileFilter extractableTxtFilesFilter = new ExtractableTextFileFilter();    
+    private final String SITES_DIR = "../dwn-sites";
+    private static final String COMMONCRAWL_DATA_PREFIX = "https://commoncrawl.s3.amazonaws.com/";
+    
+    
+    private void log(String msg) {
+	System.err.println(msg);
+	logger.info(msg);
+    }
+
+    private void debug(String msg) {
+	System.err.println(msg);
+	logger.debug(msg);
+    }
+    
+    private void error(String msg) {
+	System.err.println("### ERROR: " + msg);
+	logger.error(msg);
+    }
+    
+    private void warn(String msg) {
+	System.err.println("*** WARN: " + msg);
+	logger.warn(msg);
+    }
+    
+    public NZTLDProcessor(File seedURLsFile) throws Exception {	
+	log("In NZTLDProcessor constructor");
+    
+	if(DEBUG_MODE) {
+	    warn("Debugger is turned ON!!!");
+	}
+	
+	// TODO: why do I have to provide the path to config.props when this path is already on
+	// the classpath?
+	// In fact, I don't need it on the classpath for the following to work.
+	// How do I get it to work by specifying its path on the classpath and not here?
+	//try (FileInputStream infile = new FileInputStream("../lib/config.properties")) {
+	//try (InputStream infile = Class.forName("org.greenstone.atea.NZTLDProcessor").getClassLoader().getResourceAsStream("config.properties")) {
+	try (InputStream infile = this.getClass().getClassLoader().getResourceAsStream("config.properties")) {
+	    configProps = new Properties();
+	    configProps.load(infile);
+	    infile.close();
+	    
+	} catch(Exception e) {
+	    System.err.println("Exception attempting to read properties from config.properties.");
+	    //e.printStackTrace();
+	    throw e;
+	}
+
+	
+	try(BufferedReader br = new BufferedReader(new FileReader(seedURLsFile))) {
+	    String url = null;
+
+	    int maxURLs = 1;
+	    int urlCounter = 0;
+	    
+	    while((url = br.readLine()) != null) {
+		
+		log("Got URL: " + url);
+
+		// skip urls containing "/crawldiagnostics/" and "/robotstxt/"
+		if(url.indexOf("/robotstxt/") != -1) continue;
+		if(url.indexOf("/crawldiagnostics/") != -1) continue;
+
+		// convert the remaining WARC urls to WET urls
+		url = url.replace("/warc/CC-MAIN-", "/wet/CC-MAIN-");
+		url = url.replace(".warc.gz", ".warc.wet.gz");
+
+		// add the prefix
+		url = COMMONCRAWL_DATA_PREFIX + url;
+		
+		log("Final WET URL: " + url);
+		    
+		processURL(url);
+		
+		if(DEBUG_MODE) {
+		    urlCounter++;
+		    if(urlCounter == maxURLs) break;
+		}
+		
+	    }
+
+	    br.close();
+	    
+	} catch (IOException ioe) {
+	    System.err.println("Exception attempting to read from seedURLsFile.");
+	    //ioe.printStackTrace();
+	    throw ioe;
+	}
+    }
+
+    public boolean processURL(String ccWETfileURL) {
+	// launch wget on URL
+	// when download done, recurse through downloaded dir
+	// for each htm/html/no-extension file: parse out text from paragraphs (TODO: later Tika on non html/htm/no-extension files)
+	// For each para, run MaoriTextDetector and store results in db
+	// Remove download directory (or only remove if site doesn't contain Maori txt)?
+
+
+	if(DEBUG_MODE) {
+	    ccWETfileURL = "https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2019-30/segments/1563195523840.34/wet/CC-MAIN-20190715175205-20190715200204-00034.warc.wet.gz";
+	}
+	log("Processing WET file URL: " + ccWETfileURL);
+
+	/*
+	//if(wgetMirrorSite(ccWETfileURL)) {
+	if(!DEBUG_MODE) { wgetMirrorSite(ccWETfileURL); } // TODO: take out debug_mode condition
+	
+	processDownloadedDir(ccWETfileURL);
+	
+	//}
+	*/
+
+	
+	processWETfile(ccWETfileURL);
+	
+	
+	return true;
+    }
+
+    public boolean processWETfile(String ccWETfileURL) {
+	String zippedWETfileName = ccWETfileURL.substring(ccWETfileURL.lastIndexOf('/')+1);
+	String unzippedWETfileName = zippedWETfileName.substring(0, zippedWETfileName.lastIndexOf(".gz"));
+
+	File inZipFile = new File(SITES_DIR, zippedWETfileName);
+	File WETfile = new File(SITES_DIR, unzippedWETfileName);
+	
+	if(WETfile.exists()) {
+	    log("Unzipped WET file " + WETfile + " already exists");
+	} 
+	else {
+
+	    if(inZipFile.exists()) {
+		log("Not wgetting " + ccWETfileURL + " as " + inZipFile + " already exists");	
+	    }
+	    else if(!wgetWETfile(ccWETfileURL)) {
+		return false;
+	    }	
+
+	    // don't have the WET file yet. Get it from the zip file, which we know we should have by now
+	    
+	    boolean success = Utility.unzipFile(inZipFile, WETfile);
+	    log("Unzipped " + inZipFile + " to " + WETfile);
+	    
+	    // whether we succeeded or not, get rid of the zipped file:
+	    if(!inZipFile.delete()) {
+		warn("Unable to delete zipped WET file: " + zippedWETfileName);
+	    }
+	    
+	    if(!success) {
+		return false;
+	    }
+	}
+
+	// read in the giant WET file and 
+	
+	return true;
+    }
+
+    /*
+    // Run gunzip
+    // To avoid making this linux specific, use Java to unzip, instead of running gunzip as process
+    // https://www.mkyong.com/java/how-to-decompress-file-from-gzip-file/
+    public boolean unzipFile(File inZipFile, File outFile) {
+	
+	
+	byte[] buffer = new byte[1024];
+
+	// try-with-resources will safely close streams/dispose resources on success or error and exceptions
+	try (	    
+	    GZIPInputStream gzis = new GZIPInputStream(new FileInputStream(inZipFile));	    
+	    FileOutputStream out = new FileOutputStream(outFile);
+	) {
+	    int len;
+	    while ((len = gzis.read(buffer)) > 0) {
+        	out.write(buffer, 0, len);
+	    }
+	    
+	    //gzis.close();
+	    //out.close();
+	    
+	    log("Unzipped " + inZipFile + " to " + outFile);
+	    
+	} catch(IOException ex) {
+	    error("Failed to unzip " + inZipFile);
+	    ex.printStackTrace();
+	    return false;
+	}
+
+	return true;
+    }
+    */
+    
+    // wget will be launched from the specified directory, SITES_DIR
+    // Wget exit status/return codes: https://www.gnu.org/software/wget/manual/html_node/Exit-Status.html
+    public boolean wgetWETfile(String ccWETfileURL) {
+	// use SafeProcess and wget command in lib/config.properties
+	// DONE: set up logging	
+	
+	String wgetCmd = configProps.getProperty("wget.file.cmd");
+	if(wgetCmd == null || wgetCmd.equals("")) {
+	    System.err.println("Invalid or empty wget.file.cmd in config.properties");
+	    return false;
+	}
+
+	// replace the placeholder in the wget cmd for the seed url
+	wgetCmd = wgetCmd.replace("%%FILE_URL%%", ccWETfileURL);
+	
+	log("Will launch wget with the command: " + wgetCmd);
+
+	String[] wgetCommandArgs = wgetCmd.split(" ");
+	SafeProcess wgetProcess = new SafeProcess(wgetCommandArgs, null, new File(SITES_DIR));
+	
+	SafeProcess.LineByLineHandler outLineHandler = new ProcessLineHandler(logger, SafeProcess.STDOUT);
+	SafeProcess.LineByLineHandler errLineHandler = new ProcessLineHandler(logger, SafeProcess.STDERR);
+	    
+	int success = wgetProcess.runProcess(outLineHandler, errLineHandler);
+
+	if(success != 0) {
+	    System.err.println("Wget cmd \"" + wgetCmd  + "\" returned unsuccessfully with the value \"" + success + "\"");
+	    return false;
+	    	// TODO: why is return val = 6 ("Username/password authentication failure") for 3MB of downloads from www.waikato.ac.nz?
+	}
+	
+	return true;
+    }
+    
+    
+    // wget will be launched from the specified directory, SITES_DIR
+    // Wget exit status/return codes: https://www.gnu.org/software/wget/manual/html_node/Exit-Status.html
+    public boolean wgetMirrorSite(String nzDomainURL) {
+	// use SafeProcess and wget command in lib/config.properties
+	// DONE: set up logging
+
+	
+	String wgetCmd = configProps.getProperty("wget.mirror.cmd");
+	if(wgetCmd == null || wgetCmd.equals("")) {
+	    System.err.println("Invalid or empty wget.mirror.cmd in config.properties");
+	    return false;
+	}
+
+	// replace the placeholder in the wget cmd for the seed url
+	wgetCmd = wgetCmd.replace("%%BASE_URL%%", nzDomainURL);
+	
+	log("Will launch wget with the command: " + wgetCmd);
+
+	String[] wgetCommandArgs = wgetCmd.split(" ");
+	SafeProcess wgetProcess = new SafeProcess(wgetCommandArgs, null, new File(SITES_DIR));
+	
+	SafeProcess.LineByLineHandler outLineHandler = new ProcessLineHandler(logger, SafeProcess.STDOUT);
+	SafeProcess.LineByLineHandler errLineHandler = new ProcessLineHandler(logger, SafeProcess.STDERR);
+	    
+	int success = wgetProcess.runProcess(outLineHandler, errLineHandler);
+
+	if(success != 0) {
+	    System.err.println("Wget cmd \"" + wgetCmd  + "\" returned unsuccessfully with the value \"" + success + "\"");
+	    return false;
+	    	// TODO: why is return val = 6 ("Username/password authentication failure") for 3MB of downloads from www.waikato.ac.nz?
+	}
+	
+	return true;
+    }    
+
+    
+    // ----------- inner class for SafeProcess to Run Wget ----------
+    public class ProcessLineHandler extends SafeProcess.LineByLineHandler
+    {
+	//static Logger logger = Logger.getLogger(this.getClass().getName());;
+	Logger logger;
+	
+	public ProcessLineHandler(Logger logger, int src)
+	{
+	    super(src); // will set this.source to STDERR or STDOUT
+	    //logger = Logger.getLogger(this.getClass().getName());
+	    this.logger = logger;
+	}
+
+	public void gotLine(String line) { // first non-null line
+
+	    // String form of this.source will be "stderr" or "stdout"
+	    String msg = SafeProcess.streamToString(source) + ": " + line;
+	    System.err.println(msg);
+	    //NZTLDProcessor.this.logger.info(msg);
+	    logger.info(msg);
+	}
+	public void gotException(Exception e) {
+	    String msg = "Error in reading process' " + SafeProcess.streamToString(source);
+	    //NZTLDProcessor.this.logger.error(msg, e);
+	    logger.error(msg, e);
+	}
+
+    }
+
+
+    public boolean processDownloadedDir(String ccWETfileURL) {
+	// recurse through the downloaded directory, then process each file
+
+	File downloadedSiteDir = new File(SITES_DIR, ccWETfileURL);
+	log("*** Will process download dir " + downloadedSiteDir);
+	
+	if(!downloadedSiteDir.exists()) {
+	    error("Expected downloaded site " + downloadedSiteDir + ", but it did not exist!");
+	    return false;
+	}
+	/*
+	if(!downloadedSiteDir.isDirectory()) {
+	    error("Downloaded site " + downloadedSiteDir + " is not a directory!");
+	    return false; // TODO: or redo wget with "www." prefixed to URL???
+	}
+	*/
+	recursivelyProcessDir(ccWETfileURL, downloadedSiteDir);
+
+	debug("Have set filefilter regex to exclude: " + ExtractableTextFileFilter.unacceptableRegex);
+	
+	return true;
+    }
+
+    private void recursivelyProcessDir(String ccWETfileURL, File file) {
+
+	if(file.isDirectory()) { // recursive step
+	    // children array null iff IO Exception OR if file not a directory,
+	    // which it can't be since we tested isDirectory() on it just above
+	    File[] children = file.listFiles(extractableTxtFilesFilter);
+	    if(children == null) {
+		error("IO error occurred when trying to list children of " + file);
+	    } else {
+		// children array will be empty if 'file' dir was empty
+		for(int i = 0; i < children.length; i++) {
+		    recursivelyProcessDir(ccWETfileURL, children[i]);
+		}
+	    }
+	    
+	} else { // base step
+	    processFile(ccWETfileURL, file);	    
+	}
+    }	
+    
+    /*
+    public boolean processDownloadedDir(String ccWETfileURL) {
+	// recurse through the downloaded directory, then process each file
+
+	File downloadedSiteDir = new File(SITES_DIR, ccWETfileURL);
+	if(!downloadedSiteDir.exists()) {
+	    error("Expected downloaded site " + downloadedSiteDir + ", but it did not exist!");
+	    return false;
+	}
+
+	File[] files = downloadedSiteDir.listFiles(extractableTxtFilesFilter);
+	if(files == null) {
+	    error("Downloaded site dir " + downloadSiteDir + " was a file or an IO exception occurred");
+	    return false;
+	}
+	recursivelyProcessDir(ccWETfileURL, files);
+	
+	return true;
+    }
+
+    private void recursivelyProcessDir(String ccWETfileURL, File[] children) {
+	for(int i = 0; i < files.length; i++) {
+	    if(files[i].isDirectory()) { // recursive step
+
+		// children array will be empty if dir empty
+		// children null iff IOException OR if files[i] were a file, which it can't be since we tested isDirectory on it just above
+		File[] children = files[i].listFiles(extractableTxtFilesFilter);
+		if(children == null) {
+		    error("IO error trying to list children of " + files[i]);
+		} else {
+		    recursivelyProcessDir(ccWETfileURL, children);
+		}
+	    }
+	    else { // base step
+		processFile(ccWETfileURL, files[i]);
+	    }
+	}
+    }*/
+    
+    public boolean processFile(String ccWETfileURL, File file) {
+	// skip if js, css, robots.txt
+	// if no-extension or htm or html, call processHTMLFile()
+	// else: skip for now, TODO: later call Tika on other file types
+
+	// TODO: update db with url info, page location and para?
+
+
+	log("*** Processing file " + file);
+	
+	return true;
+    }
+
+    private class ExtractableTextFileFilter implements FilenameFilter {
+	// skip if js, css, robots.txt
+	
+	// For now also skip the image files. Later, with Tika, may be able to extract text from
+	// images though OCR of those imgs representing text? We accept files with no file extension,
+	// e.g. waikato uni has lots of files without extension that contain html.
+	// Examples: Files like *.png but also *.png?v=QE50XMk2oY should be rejected
+	public static final String unacceptableRegex = ".*\\.(json|js|css|svg|ico|jpe?g|png|gif|tif?)($|\\?.*)";
+	//Pattern unacceptableRegexPattern = Pattern.compile(unacceptableRegex);
+
+	
+	public boolean accept(File dir, String name) {
+	    
+	    return !(name.matches("robots.txt") || name.matches(unacceptableRegex));
+	    //return !(name.matches("robots.txt") || name.matches("/\\.png/")); // doesn't work
+	    //return name.matches(".*\\.html$"); // works
+	    //return name.matches(".*\\.png($|\\?.*)"); // works
+	}
+    }
+
+    
+    public boolean processHTMLFile() {
+	// Use JSoup to get paras
+
+	// call processParas(text);
+
+	return true;
+    }
+
+    public boolean processNonHTMLFile() {
+	// Use Tika to get text	
+
+	// call processParas(text)
+
+	return true;
+    }
+
+    public boolean processParas(String text) {
+
+	// Split on (double) new line for paras? If too short, then add text from next para unless there's no more. Else add to previous?
+	
+	// for each para, call MaoriTextDetector.java
+
+	return true;
+    }
+    // --------------------------- End inner class --------------------
+
+    public static void main(String[] args) {
+	if(args.length <= 0) {
+	    System.err.println("ERROR: Must provide input file of unique nz website addresses");
+	    System.exit(-1);
+	}
+	
+	String infilePath = args[0];
+
+	System.err.println("Requested to process URLS file: "  + infilePath);
+
+	File infile = new File(infilePath);
+	if(!infile.exists()) {
+	    System.err.println("ERROR: File " + infilePath + " did not exist");
+	    System.exit(-1);
+	}
+
+
+	try {
+	    NZTLDProcessor processor = new NZTLDProcessor(infile);
+	} catch(Exception e) {
+	    e.printStackTrace();
+	}
+	
+    }
+}
+
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToCSV.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToCSV.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToCSV.java	(revision 33635)
@@ -0,0 +1,452 @@
+package org.greenstone.atea;
+
+import java.io.*;
+import java.lang.ArrayIndexOutOfBoundsException;
+import java.time.LocalDateTime;
+import java.util.ArrayList;
+import java.util.Arrays;
+
+import org.apache.commons.csv.*;
+import org.apache.log4j.Logger;
+
+
+/**
+ * Class to process the dump text files produced FOR EACH SITE (e.g. site "00001") that
+ * Nutch has finished crawling and whose text has been dumped out to a file called dump.txt.
+ * This reads in the dump.txt file contained in each site folder within the input folder.
+ * (e.g. input folder "crawled" could contain folders 00001 to 01465. Each contains a dump.txt)
+ * Each dump.txt could contain the text contents for an entire site, or for individual pages.
+ * This class then uses class TextDumpPage to parse each webpage within a dump.txt,
+ * which parses out the actual text body content of each webpage's section within a dump.txt.
+ * Finally, MaoriTextDetector is run over that to determine whether the full body text is
+ * likely to be in Maori or not.
+ *
+ * Potential issues: since a web page's text is dumped out by nutch with neither paragraph
+ * nor even newline separator, it's hard to be sure that the entire page is in language.
+ * If it's in multiple languages, there's no way to be sure there aren't promising Maori language
+ * paragraphs contained in a page, if the majority/the remainder happen to be in English.
+ * 
+ * So if we're looking for any paragraphs in Maori to store in a DB, perhaps it's better to run
+ * the MaoriTextDetector.isTextInMaori(BufferedReader reader) over two "lines" at a time,
+ * instead of running it over the entire html body's text.
+ *
+ * TO COMPILE OR RUN, FIRST DO:
+ *    cd maori-lang-detection/apache-opennlp-1.9.1
+ *    export OPENNLP_HOME=`pwd`
+ *    cd maori-lang-detection/src
+ *
+ * TO COMPILE:
+ *    maori-lang-detection/src$
+ *       javac -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/NutchTextDumpToCSV.java
+ *
+ * TO RUN:
+ *    maori-lang-detection/src$
+ *       java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/NutchTextDumpToCSV ../crawled-small
+ *
+ * or:
+ *       java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/NutchTextDumpToCSV ../crawled-small > ../crawled-small/bla.txt 2>&1
+ *
+*/
+public class NutchTextDumpToCSV {
+    static Logger logger = Logger.getLogger(org.greenstone.atea.NutchTextDumpToCSV.class.getName());
+
+    static boolean DEBUG_MODE = true;
+
+    /** Counter for number of sites.
+     * Should be equal to number of times NutchTextDumpToCSV constructor
+     * is called: once per site.
+     */
+    static private int SITE_COUNTER = 0;
+    static private long WEBPAGE_COUNTER = 0;
+    static private long MRI_SENTENCE_COUNTER = 0;
+    
+    private final MaoriTextDetector maoriTxtDetector;
+    
+    public final String siteID;
+    public final boolean siteCrawlUnfinished;
+    public final long siteCrawledTimestamp; /** When the crawl of the site terminated */
+    
+    private String domainOfSite;
+    
+    /** keep a list to store the text of each page */
+    private ArrayList<TextDumpPage> pages;
+
+    /** list of pages in this site which were detected as being in MRI */
+    private ArrayList<MRIWebPageStats> pagesInMRI;
+    /** 
+     * list of pages in this site which were NOT detected as being in MRI but nevertheless
+     * contain one or more sentences in MRI 
+     */
+    private ArrayList<MRIWebPageStats> pagesContainingMRI;
+    
+    private boolean isStartOfNewWebPageRecord(String prevLine, String line) {
+	// The start of a new web page's record in nutch's text dump of an entire site
+	// is denoted by a newline followed by a URL (protocol)
+	// or the very start of the file with a URL (protocol)
+	return ((prevLine == null || prevLine.equals(""))
+		&& (line.startsWith("http://") || line.startsWith("https://")));
+    }
+
+    public void debugPageDump(StringBuilder pageDump) {
+	if(DEBUG_MODE) {
+	    // START DEBUG
+	    logger.debug("__________________________________________");
+	    logger.debug("@@@ Found page entry: ");
+	    logger.debug("__________________________________________");
+	    logger.debug(pageDump.toString());
+	    logger.debug("------------------------------------------");
+	    // END DEBUG
+	}
+    }
+
+    /** A NutchTextDumpToCSV processes the dump.txt for one site */
+    public NutchTextDumpToCSV(CSVPrinter webpagesCSVPrinter, CSVPrinter mriSentencesCSVPrinter,
+				  MaoriTextDetector maoriTxtDetector, String siteID,
+				  File txtDumpFile, long lastModified, boolean siteCrawlUnfinished)
+	throws IOException
+    {	
+	// increment static counter of sites processed by a NutchTextDumpToCSV instance
+	SITE_COUNTER++;
+	
+	// siteID is of the form %5d (e.g. 00020) and is just the name of a site folder
+	this.siteID = siteID;
+	this.siteCrawlUnfinished = siteCrawlUnfinished;
+	this.siteCrawledTimestamp = lastModified;
+	
+	this.maoriTxtDetector = maoriTxtDetector;
+	    
+	pages = new ArrayList<TextDumpPage>();
+
+	String line = null;
+	StringBuilder pageDump = null;
+	try (
+	     BufferedReader reader = new BufferedReader(new FileReader(txtDumpFile));
+	     ) {
+
+	    boolean readingText = false;
+	    String prevLine = null;
+	    
+	    while((line = reader.readLine()) != null) { // readLine removes newline separator
+		line = line.trim();
+		// iff outside of a page's body text, then an empty line marks the end of a page
+		// in nutch's text dump of a site.
+		// But note, there can be an empty line (or more?) between the start and end
+		// markers of a page's text, though.
+
+		if(isStartOfNewWebPageRecord(prevLine, line)) {
+
+		    if(pageDump != null) { // should also be the case then: if(prevLine != null)
+			// finish old pageDump and begin new one			
+
+			//debugPageDump(pageDump);
+			    
+			TextDumpPage page = new TextDumpPage(siteID, pageDump.toString());
+			// parses the fields and body text of a webpage in nutch's txt dump of entire site
+			//page.parseFields();
+			//page.getText();
+			pages.add(page);
+			pageDump = null;
+
+		    }		    
+
+		    // begin new webpage dump
+		    pageDump = new StringBuilder();
+		    pageDump.append(line);
+		    pageDump.append("\n");
+		    
+		}
+		else if(!line.equals("")) {
+		    pageDump.append(line);
+		    pageDump.append("\n");
+
+		}
+		// can throw away any newlines between text start and end markers.
+		
+		prevLine = line;		
+	    }
+
+	    // process final webpage record:
+	    //debugPageDump(pageDump);
+	    
+	    TextDumpPage page = new TextDumpPage(siteID, pageDump.toString());
+	    pages.add(page);
+	    pageDump = null;
+		
+	} catch (IOException ioe) {
+	    logger.error("@@@@@@@@@ Error reading in nutch txtdump file " + txtDumpFile, ioe);
+	}
+
+	// Just do this once: get and store domain of site.
+	// Passing true to get domain with protocol prefix
+	if(pages.size() > 0) {
+	    TextDumpPage firstPage = pages.get(0);
+	    String url = firstPage.getPageURL();
+	    this.domainOfSite = Utility.getDomainForURL(url, true);
+	}
+	else {
+	    this.domainOfSite = "UNKNOWN";
+	}
+	
+	prepareSiteStats(webpagesCSVPrinter, mriSentencesCSVPrinter);
+    }
+
+    /* UNUSED */
+    /** pageID: id into pages array */
+    /*
+    public boolean isPageInMaori(int pageID) throws ArrayIndexOutOfBoundsException {	
+
+	String text = getTextForPage(pageID);
+	
+	// QTODO: what to do when page body text is empty?
+	if(text.equals("")) return false;
+	return maoriTxtDetector.isTextInMaori(text);
+    }
+
+    private TextDumpPage getPage(int pageID) throws ArrayIndexOutOfBoundsException {
+	if(pageID < 0 || pageID >= pages.size()) {
+	    throw new ArrayIndexOutOfBoundsException();
+	}
+
+	TextDumpPage page = pages.get(pageID);
+	return page;
+    }
+
+    public String getTextForPage(int pageID) throws ArrayIndexOutOfBoundsException {
+	TextDumpPage page = getPage(pageID);
+	return page.getPageText();	
+    }
+    public String getURLForPage(int pageID) throws ArrayIndexOutOfBoundsException {
+	TextDumpPage page = getPage(pageID);
+	return page.getPageURL();
+    }   
+       
+    public int totalNumPages() {
+	return pages.size();
+    }
+    public int getNumPagesInMRI() {
+	return pagesInMRI.size();
+    }    
+    public int getNumPagesContainingMRI() {
+	return pagesContainingMRI.size();
+    }
+    */
+    
+    private void prepareSiteStats(CSVPrinter webpageCSVPrinter, CSVPrinter mriSentencesCSVPrinter) throws IOException {
+	pagesInMRI = new ArrayList<MRIWebPageStats>();
+	pagesContainingMRI = new ArrayList<MRIWebPageStats>();
+
+	TextDumpPage page = null;
+	for(int i = 0; i < pages.size(); i++) {
+	    
+	    page = pages.get(i);
+	    
+	    String text = page.getPageText();
+	    
+	    if(text.equals("")) {
+		//page.addMRILanguageStatus(false);
+		continue;
+	    }
+	    else {
+		boolean isMRI = maoriTxtDetector.isTextInMaori(text);
+		
+		//page.addMRILanguageStatus(isMRI);
+		
+    
+		// Even if the entire page is not found to be overall in MÄori,
+		// let's still inspect the sentences of the page and count how many (if any)
+		// are in te reo.
+		ArrayList<String> mriSentences = maoriTxtDetector.getAllSentencesInMaori(text);
+		// first element of ArrayList returned is always total num sentences on page
+		// remaining elements are the actual sentences that were detected as being MÄori
+		int totalSentences = Integer.parseInt(mriSentences.get(0));
+		int numSentencesInMRI = mriSentences.size() - 1;
+
+		// Add page to list of MRI pages if the page's body text overall was detected
+		// as MÄori
+		// Add page to list of pages containing MRI if >= 1 sentences in the page
+		// were detected as being in MRI
+		if(isMRI || numSentencesInMRI >= 1) {
+		    String url = page.getPageURL();
+		    MRIWebPageStats MRIpageStats = new MRIWebPageStats(this.siteID, url, i, isMRI,
+					       totalSentences, numSentencesInMRI);
+		    if(isMRI) {   
+			pagesInMRI.add(MRIpageStats);
+		    } else if(numSentencesInMRI >= 1) {
+			pagesContainingMRI.add(MRIpageStats);
+		    }
+
+		    // Only write to webpages csv file for those pages that had any MRI
+		    // language content.
+		    // column headers:
+		    //    pageID, siteID, URL, isMRI, numSentences, numSentencesInMRI
+		    //int pageID = i; // not primary key by itself,
+		            // must be combined with siteID to form primary key
+		    
+		    webpageCSVPrinter.printRecord(WEBPAGE_COUNTER++,
+						  SITE_COUNTER, /* alternative: this.siteID */
+						  url,
+						  //"origCharEncoding", "modifiedTime", "fetchTime",
+						  page.getOriginalCharEncoding(),
+						  page.getModifiedTime(),
+						  page.getFetchTime(),
+						  isMRI, totalSentences, numSentencesInMRI);
+
+		    // Write the sentences that are in te reo into the mri-sentences CSV file
+		    // whether from webpages that are MRI overall or only those that containing
+		    // any sentences in MRI
+		    for (int j = 1; j < mriSentences.size(); j++) { // 1st element not a sentence
+			//int sentenceID = j; // combine with siteID and pageID to form primary key
+			String mriSentence = mriSentences.get(j);
+			// sentenceID, pageID, sentence
+			//mriSentencesCSVPrinter.printRecord(sentenceID, pageID, mriSentence);
+			mriSentencesCSVPrinter.printRecord(MRI_SENTENCE_COUNTER++, WEBPAGE_COUNTER, mriSentence);
+		    }
+		}		
+
+	    }	    
+	}
+    }
+    
+    public void printSiteStats() {
+	
+	
+	logger.info("------------- " + this.siteID + " SITE STATS -----------");
+
+	logger.info("SITE DOMAIN: " + this.domainOfSite);
+	logger.info("Total number of web pages in site: " + pages.size());
+	logger.info("Of these, the number of pages in MÄori (mri) were: " + this.pagesInMRI.size());
+	
+	if(pagesInMRI.size() > 0) {
+	    logger.info("The following were the pages detected by OpenNLP as being in MÄori with " + maoriTxtDetector.MINIMUM_CONFIDENCE + " confidence");
+	    for(MRIWebPageStats mriWebPageInfo : pagesInMRI) {
+		logger.info(mriWebPageInfo.toString());
+	    }
+	}
+
+	logger.info("                      -----------                   ");
+	if(pagesContainingMRI.size() > 0) {	    
+	    logger.info("The following pages weren't detected as primarily being in MÄori");
+	    logger.info("But still contained sentences detected as MÄori");
+	    for(MRIWebPageStats mriWebPageInfo : pagesContainingMRI) {
+		logger.info(mriWebPageInfo.toString());
+	    }
+	    
+	} else {
+	    logger.info("No further pages detected as containing any sentences in MRI");	   
+	}
+	logger.info("                      -----------                   ");
+    }
+
+    
+    public void writeSiteRecordToCSV(CSVPrinter websitesCSVPrinter) throws IOException {
+
+	// https://stackoverflow.com/questions/35183146/how-can-i-create-a-java-8-localdate-from-a-long-epoch-time-in-milliseconds
+	// LocalDateTime date =
+	//     LocalDateTime.ofInstant(Instant.ofEpochMilli(this.siteCrawledTimestamp), ZoneId.systemDefault());
+	// String crawlTimestamp =
+	//     date.format(DateTimeFormatter.ofPattern("yyyy-MM-dd")) + " " + date.format(DateTimeFormatter.ofPattern("HH:mm:ss"));
+
+	boolean redoCrawl = false;
+	int numPagesInMRI = pagesInMRI.size();
+	int numPagesContainingMRI = pagesContainingMRI.size();
+	    
+	if(this.siteCrawlUnfinished) {
+	    // arbitrary decision, but need some indication that the MRI content was not close to one-off in the website
+	    if(numPagesInMRI > 2 || numPagesContainingMRI > 2) { 
+		redoCrawl = true;
+	    }
+	}
+	
+	// site.csv CSV file row:
+	// ID, siteID, domainURL, totalPagesInSite, numPagesInMRI, numPagesContainingMRI, crawlUnfinished, redoCrawl
+	websitesCSVPrinter.printRecord(SITE_COUNTER, this.siteID, this.domainOfSite,
+				       pages.size(), numPagesInMRI, numPagesContainingMRI,
+				   this.siteCrawledTimestamp, this.siteCrawlUnfinished, redoCrawl);
+    }
+
+    
+    // --------------- STATIC METHODS AND INNER CLASSED USED BY MAIN -------------- //
+   
+    public static void printUsage() {
+	System.err.println("Run this program as:");
+	System.err.println("\tNutchTextDumpToCSV <path to 'crawled' folder>");
+    }
+    
+    public static void main(String[] args) {
+	if(args.length != 1) {
+	    printUsage();
+	    return;
+	}
+
+	File sitesDir = new File(args[0]);
+	if(!sitesDir.exists() || !sitesDir.isDirectory()) {
+	    logger.error("Error: " + args[0] + " does not exist or is not a directory");
+	    return;
+	}
+
+	NutchTextDumpToCSV.DEBUG_MODE = false;
+	
+	File websitesCSVFile = new File(sitesDir, "websites.csv");
+	File webpagesCSVFile = new File(sitesDir, "webpages.csv");
+	File mriSentencesCSVFile = new File(sitesDir, "mri-sentences.csv");
+	
+	try (
+	     CSVPrinter websitesCSVPrinter = new CSVPrinter(new FileWriter(websitesCSVFile), CSVFormat.DEFAULT);
+	     CSVPrinter webpagesCSVPrinter = new CSVPrinter(new FileWriter(webpagesCSVFile), CSVFormat.DEFAULT);
+	     CSVPrinter mriSentencesCSVPrinter = new CSVPrinter(new FileWriter(mriSentencesCSVFile), CSVFormat.DEFAULT);
+	     ) {
+
+	    // print out the column headers for the websites csv file
+	    // https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVPrinter.html
+	    websitesCSVPrinter.printRecord("ID" /*websiteID*/, "siteID"/* site folder name*/,
+		   "domainURL","totalPagesInSite", "numPagesInMRI", "numOtherPagesContainingMRI",
+		   "nutchCrawlTimestamp", "crawlUnfinished", "redoCrawl");
+	    webpagesCSVPrinter.printRecord("webpageID", "websiteID", "URL",
+					   "origCharEncoding", "modifiedTime", "fetchTime",
+					   "isMRI", "numSentences", "numSentencesInMRI");
+	    mriSentencesCSVPrinter.printRecord("sentenceID", "webpageID", "sentence");
+	    
+	    MaoriTextDetector mriTxtDetector = new MaoriTextDetector(true); // true: run silent
+	    File[] sites = sitesDir.listFiles();
+
+	    // sort site folders in alphabetical order
+	    // https://stackoverflow.com/questions/7199911/how-to-file-listfiles-in-alphabetical-order
+	    Arrays.sort(sites);
+	    
+	    for(File siteDir : sites) { // e.g. 00001
+		if(siteDir.isDirectory()) {
+		    // look for dump.txt
+		    File txtDumpFile = new File(siteDir, "dump.txt");
+		    if(!txtDumpFile.exists()) {
+			logger.error("Text dump file " + txtDumpFile + " did not exist");
+			continue;
+		    }
+		    
+		    else {
+			File UNFINISHED_FILE = new File(siteDir, "UNFINISHED");			
+			
+			String siteID = siteDir.getName();
+			long lastModified = siteDir.lastModified();
+			logger.debug("Found siteID: " + siteID);		    
+			NutchTextDumpToCSV nutchTxtDump = new NutchTextDumpToCSV(
+				 webpagesCSVPrinter, mriSentencesCSVPrinter, mriTxtDetector,
+				 siteID, txtDumpFile, lastModified, UNFINISHED_FILE.exists());
+			// now it's parsed all the web pages in the site's text dump
+
+			// Let's print stats on each web page's detected language being MRI or not
+			// and how many pages there were in the site in total.
+			
+			nutchTxtDump.printSiteStats();
+			nutchTxtDump.writeSiteRecordToCSV(websitesCSVPrinter);
+		    }
+		}
+		
+	    }
+	    
+	} catch(Exception e) {
+	    // can get an exception when instantiating NutchTextDumpToCSV instance
+	    // or with CSV file
+	    logger.error(e.getMessage(), e);
+	}
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java	(revision 33635)
@@ -0,0 +1,385 @@
+package org.greenstone.atea;
+
+import java.io.*;
+import java.lang.ArrayIndexOutOfBoundsException;
+import java.time.LocalDateTime;
+import java.util.ArrayList;
+import java.util.Arrays;
+
+import org.apache.commons.csv.*;
+import org.apache.log4j.Logger;
+
+
+/**
+ * Class to process the dump text files produced FOR EACH SITE (e.g. site "00001") that
+ * Nutch has finished crawling and whose text has been dumped out to a file called dump.txt.
+ * This reads in the dump.txt file contained in each site folder within the input folder.
+ * (e.g. input folder "crawled" could contain folders 00001 to 01465. Each contains a dump.txt)
+ * Each dump.txt could contain the text contents for an entire site, or for individual pages.
+ * This class then uses class TextDumpPage to parse each webpage within a dump.txt,
+ * which parses out the actual text body content of each webpage's section within a dump.txt.
+ * Finally, MaoriTextDetector is run over that to determine whether the full body text is
+ * likely to be in Maori or not.
+ *
+ * Potential issues: since a web page's text is dumped out by nutch with neither paragraph
+ * nor even newline separator, it's hard to be sure that the entire page is in language.
+ * If it's in multiple languages, there's no way to be sure there aren't promising Maori language
+ * paragraphs contained in a page, if the majority/the remainder happen to be in English.
+ * 
+ * So if we're looking for any paragraphs in Maori to store in a DB, perhaps it's better to run
+ * the MaoriTextDetector.isTextInMaori(BufferedReader reader) over two "lines" at a time,
+ * instead of running it over the entire html body's text.
+ *
+ * TO COMPILE OR RUN, FIRST DO:
+ *    cd maori-lang-detection/apache-opennlp-1.9.1
+ *    export OPENNLP_HOME=`pwd`
+ *    cd maori-lang-detection/src
+ *
+ * TO COMPILE:
+ *    maori-lang-detection/src$
+ *       javac -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/NutchTextDumpToMongoDB.java
+ *
+ * TO RUN:
+ *    maori-lang-detection/src$
+ *       java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/NutchTextDumpToMongoDB ../crawled-small
+ *
+ * or:
+ *       java -cp ".:../conf:../lib/*:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/NutchTextDumpToMongoDB ../crawled-small > ../crawled-small/bla.txt 2>&1
+ *
+*/
+public class NutchTextDumpToMongoDB {
+    static Logger logger = Logger.getLogger(org.greenstone.atea.NutchTextDumpToMongoDB.class.getName());
+
+    static boolean DEBUG_MODE = true;
+
+    /** Counter for number of sites.
+     * Should be equal to number of times NutchTextDumpToMongoDB constructor
+     * is called: once per site.
+     */
+    static private int SITE_COUNTER = 0;
+    static private long WEBPAGE_COUNTER = 0;
+    
+    private final MaoriTextDetector maoriTxtDetector;
+    private final MongoDBAccess mongodbAccess;
+    
+    public final String siteID;
+    public final boolean siteCrawlUnfinished;
+    public final long siteCrawledTimestamp; /** When the crawl of the site terminated */
+    
+    private int countOfWebPagesWithBodyText = 0;
+    
+    private String geoLocationCountryCode = null; /** 2 letter country code */
+    private boolean urlContainsLangCodeInPath = false; /** If any URL on this site contains a /mi(/) in its URL */
+    
+    private String domainOfSite;
+    private int numPagesInMRI = 0;
+    
+    /** keep a list to store the text of each page */
+    private ArrayList<TextDumpPage> pages;
+    
+    private boolean isStartOfNewWebPageRecord(String prevLine, String line) {
+	// The start of a new web page's record in nutch's text dump of an entire site
+	// is denoted by a newline followed by a URL (protocol)
+	// or the very start of the file with a URL (protocol)
+	return ((prevLine == null || prevLine.equals(""))
+		&& (line.startsWith("http://") || line.startsWith("https://")));
+    }
+
+    public void debugPageDump(StringBuilder pageDump) {
+	if(DEBUG_MODE) {
+	    // START DEBUG
+	    logger.debug("__________________________________________");
+	    logger.debug("@@@ Found page entry: ");
+	    logger.debug("__________________________________________");
+	    logger.debug(pageDump.toString());
+	    logger.debug("------------------------------------------");
+	    // END DEBUG
+	}
+    }
+
+    /** A NutchTextDumpToMongoDB processes the dump.txt for one site */
+    public NutchTextDumpToMongoDB(MongoDBAccess mongodbAccess,
+				  MaoriTextDetector maoriTxtDetector, String siteID,
+				  File txtDumpFile, long lastModified, boolean siteCrawlUnfinished)
+	throws IOException
+    {	
+	// increment static counter of sites processed by a NutchTextDumpToMongoDB instance
+	SITE_COUNTER++;
+	
+	// siteID is of the form %5d (e.g. 00020) and is just the name of a site folder
+	this.siteID = siteID;
+	this.siteCrawlUnfinished = siteCrawlUnfinished;
+	this.siteCrawledTimestamp = lastModified;
+	
+	this.maoriTxtDetector = maoriTxtDetector;
+	this.mongodbAccess = mongodbAccess;
+	
+	pages = new ArrayList<TextDumpPage>();
+
+	String line = null;
+	StringBuilder pageDump = null;
+	try (
+	     BufferedReader reader = new BufferedReader(new FileReader(txtDumpFile));
+	     ) {
+
+	    boolean readingText = false;
+	    String prevLine = null;
+	    
+	    while((line = reader.readLine()) != null) { // readLine removes newline separator
+		line = line.trim();
+		// iff outside of a page's body text, then an empty line marks the end of a page
+		// in nutch's text dump of a site.
+		// But note, there can be an empty line (or more?) between the start and end
+		// markers of a page's text, though.
+
+		if(isStartOfNewWebPageRecord(prevLine, line)) {
+
+		    if(pageDump != null) { // should also be the case then: if(prevLine != null)
+			// finish old pageDump and begin new one			
+
+			//debugPageDump(pageDump);
+			    
+			TextDumpPage page = new TextDumpPage(siteID, pageDump.toString());
+			// parses the fields and body text of a webpage in nutch's txt dump of entire site
+			//page.parseFields();
+			//page.getText();
+			pages.add(page);
+			pageDump = null;
+
+		    }		    
+
+		    // begin new webpage dump
+		    pageDump = new StringBuilder();
+		    pageDump.append(line);
+		    pageDump.append("\n");
+		    
+		}
+		else if(!line.equals("")) {
+		    pageDump.append(line);
+		    pageDump.append("\n");
+
+		}
+		// can throw away any newlines between text start and end markers.
+		
+		prevLine = line;		
+	    }
+
+	    // process final webpage record:
+	    //debugPageDump(pageDump);
+	    
+	    TextDumpPage page = new TextDumpPage(siteID, pageDump.toString());
+	    pages.add(page);
+	    pageDump = null;
+
+	    // for every site, we just need to work out if any of its pages
+	    // contains /mi(/) in its URL
+	    String url = page.getPageURL();
+	    if(!this.urlContainsLangCodeInPath && (url.contains("/mi/") || url.endsWith("/mi"))) {
+		this.urlContainsLangCodeInPath = true;
+	    }
+		
+	} catch (IOException ioe) {
+	    logger.error("@@@@@@@@@ Error reading in nutch txtdump file " + txtDumpFile, ioe);
+	}
+
+	// Just do this once: get and store domain of site.
+	// Passing true to get domain with protocol prefix
+	if(pages.size() > 0) {
+	    TextDumpPage firstPage = pages.get(0);
+	    String url = firstPage.getPageURL();
+	    this.domainOfSite = Utility.getDomainForURL(url, true);
+	}
+	else {
+	    this.domainOfSite = "UNKNOWN";
+	}
+	
+	
+	prepareSiteStats(mongodbAccess);
+    }
+
+    
+    private void prepareSiteStats(MongoDBAccess mongodbAccess) throws IOException {
+    
+	TextDumpPage page = null;
+	for(int i = 0; i < pages.size(); i++) {
+	    
+	    page = pages.get(i);
+	    
+	    String text = page.getPageText();
+	    
+	    if(text.equals("")) {
+		// don't care about empty pages
+		continue;
+	    }
+	    else {
+		WEBPAGE_COUNTER++; // count of cumulative total of webpages for all sites
+		countOfWebPagesWithBodyText++; // of this site alone				
+		
+		boolean isMRI = maoriTxtDetector.isTextInMaori(text);
+		if(isMRI) {
+		    numPagesInMRI++;
+		}
+		
+		String[] sentences = maoriTxtDetector.getAllSentences(text);
+		int totalSentences = sentences.length;
+		ArrayList<SentenceInfo> singleSentences = maoriTxtDetector.getAllSentencesInfo(sentences);
+		ArrayList<SentenceInfo> overlappingSentences = maoriTxtDetector.getAllOverlappingSentencesInfo(sentences);
+		
+		WebpageInfo webpage = page.convertStoredDataToWebpageInfo(WEBPAGE_COUNTER,
+								     SITE_COUNTER,
+								     isMRI,
+								     totalSentences,
+								     singleSentences,
+								     overlappingSentences);
+
+		
+		mongodbAccess.insertWebpageInfo(webpage);
+	    }
+	}
+    }
+
+    /*
+    public void printSiteStats() {
+	
+	
+	logger.info("------------- " + this.siteID + " SITE STATS -----------");
+
+	logger.info("SITE DOMAIN: " + this.domainOfSite);
+	logger.info("Total number of web pages in site: " + pages.size());
+	logger.info("Of these, the number of pages in MÄori (mri) were: " + this.pagesInMRI.size());
+	
+	if(pagesInMRI.size() > 0) {
+	    logger.info("The following were the pages detected by OpenNLP as being in MÄori with " + maoriTxtDetector.MINIMUM_CONFIDENCE + " confidence");
+	    for(MRIWebPageStats mriWebPageInfo : pagesInMRI) {
+		logger.info(mriWebPageInfo.toString());
+	    }
+	}
+
+	logger.info("                      -----------                   ");
+	if(pagesContainingMRI.size() > 0) {	    
+	    logger.info("The following pages weren't detected as primarily being in MÄori");
+	    logger.info("But still contained sentences detected as MÄori");
+	    for(MRIWebPageStats mriWebPageInfo : pagesContainingMRI) {
+		logger.info(mriWebPageInfo.toString());
+	    }
+	    
+	} else {
+	    logger.info("No further pages detected as containing any sentences in MRI");	   
+	}
+	logger.info("                      -----------                   ");
+    }
+    */
+
+    
+
+    public void websiteDataToDB() {
+	
+	
+	// https://stackoverflow.com/questions/35183146/how-can-i-create-a-java-8-localdate-from-a-long-epoch-time-in-milliseconds
+	// LocalDateTime date =
+	//     LocalDateTime.ofInstant(Instant.ofEpochMilli(this.siteCrawledTimestamp), ZoneId.systemDefault());
+	// String crawlTimestamp =
+	//     date.format(DateTimeFormatter.ofPattern("yyyy-MM-dd")) + " " + date.format(DateTimeFormatter.ofPattern("HH:mm:ss"));
+
+	boolean redoCrawl = false;
+
+	if(this.siteCrawlUnfinished) {
+	    // arbitrary decision, but need some indication that the MRI content was not close to one-off in the website
+	    if(this.numPagesInMRI > 2) { 
+		redoCrawl = true;
+	    }
+	}
+	
+	//File geoLiteCityDatFile = new File(this.getClass().getResource("GeoLiteCity.dat").getFile());
+	//this.geoLocationCountryCode = getCountryCodeOfDomain(this.domainOfSite, geoLiteCityDatFile);
+
+	int totalPages = pages.size();	
+
+	WebsiteInfo website = new WebsiteInfo(SITE_COUNTER, this.siteID, this.domainOfSite,
+	      totalPages, this.countOfWebPagesWithBodyText, this.numPagesInMRI, 
+	      this.siteCrawledTimestamp, this.siteCrawlUnfinished, redoCrawl,
+	      this.geoLocationCountryCode, this.urlContainsLangCodeInPath);
+
+	mongodbAccess.insertWebsiteInfo(website);
+	
+    }
+
+    
+    // --------------- STATIC METHODS AND INNER CLASSED USED BY MAIN -------------- //
+   
+    public static void printUsage() {
+	System.err.println("Run this program as:");
+	System.err.println("\tNutchTextDumpToMongoDB <path to 'crawled' folder>");
+    }
+    
+    public static void main(String[] args) {
+	if(args.length != 1) {
+	    printUsage();
+	    return;
+	}
+
+	File sitesDir = new File(args[0]);
+	if(!sitesDir.exists() || !sitesDir.isDirectory()) {
+	    logger.error("Error: " + args[0] + " does not exist or is not a directory");
+	    return;
+	}
+
+	NutchTextDumpToMongoDB.DEBUG_MODE = false;
+
+	
+	try (
+	     MongoDBAccess mongodb = new MongoDBAccess();
+	     ) {
+	    
+	    mongodb.connectToDB();
+	    //mongodb.showCollections();
+	    
+	    // print out the column headers for the websites csv file
+	    // https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVPrinter.html
+	    // OPTIONAL TODO: creating collections can be done here if dropping and recreating
+	    
+	    MaoriTextDetector mriTxtDetector = new MaoriTextDetector(true); // true: run silent
+	    File[] sites = sitesDir.listFiles();
+
+	    // sort site folders in alphabetical order
+	    // https://stackoverflow.com/questions/7199911/how-to-file-listfiles-in-alphabetical-order
+	    Arrays.sort(sites);
+	    
+	    for(File siteDir : sites) { // e.g. 00001
+		if(siteDir.isDirectory()) {
+		    // look for dump.txt
+		    File txtDumpFile = new File(siteDir, "dump.txt");
+		    if(!txtDumpFile.exists()) {
+			logger.error("Text dump file " + txtDumpFile + " did not exist");
+			continue;
+		    }
+		    
+		    else {
+			File UNFINISHED_FILE = new File(siteDir, "UNFINISHED");			
+			
+			String siteID = siteDir.getName();
+			long lastModified = siteDir.lastModified();
+			logger.debug("Found siteID: " + siteID);		    
+			NutchTextDumpToMongoDB nutchTxtDump = new NutchTextDumpToMongoDB(
+				 mongodb, mriTxtDetector,
+				 siteID, txtDumpFile, lastModified, UNFINISHED_FILE.exists());
+			// now it's parsed all the web pages in the site's text dump
+
+			// Let's print stats on each web page's detected language being MRI or not
+			// and how many pages there were in the site in total.
+			
+			//nutchTxtDump.printSiteStats();
+			
+			nutchTxtDump.websiteDataToDB();
+		    }
+		}
+		
+	    }
+	    
+	} catch(Exception e) {
+	    // can get an exception when instantiating NutchTextDumpToMongoDB instance
+	    // or with CSV file
+	    logger.error(e.getMessage(), e);
+	}
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/SentenceInfo.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/SentenceInfo.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/SentenceInfo.java	(revision 33635)
@@ -0,0 +1,14 @@
+package org.greenstone.atea;
+
+public class SentenceInfo {
+    public final double confidenceLevel;
+    /** 3 letter lang code */
+    public final String langCode;
+    public final String sentence;
+    
+    public SentenceInfo(double confidence, String langCode, String sentence) {
+	this.confidenceLevel = confidence;
+	this.langCode = langCode;
+	this.sentence = sentence;
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/TextDumpPage.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/TextDumpPage.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/TextDumpPage.java	(revision 33635)
@@ -0,0 +1,225 @@
+package org.greenstone.atea;
+
+import java.io.*;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.log4j.Logger;
+
+
+public class TextDumpPage {
+    private static Logger logger = Logger.getLogger(org.greenstone.atea.TextDumpPage.class.getName());
+    
+    public static final String TEXT_START_MARKER="text:start:";
+    public static final String TEXT_END_MARKER="text:end:";
+    
+    private Map<String, String> tuples;
+
+    private boolean isMRI = false;
+
+    boolean DEBUG_MODE = false;
+    
+    public TextDumpPage(String siteID, String unparsedPageDump) {
+	tuples = new HashMap<String, String>();
+	
+	try (
+	     BufferedReader reader = new BufferedReader(new StringReader(unparsedPageDump));
+	     ) {
+
+	    String line = reader.readLine(); // should have at least first line
+
+	    // first line always has a "key:" somewhere after the pageURL
+	    int endIndex = line.indexOf("key:");
+
+	    String pageURL = line.substring(0, endIndex);
+	    //String pageURL = (endIndex == -1) ? line : line.substring(0, endIndex);	    
+	    
+	    
+	    tuples.put("pageURL", pageURL.trim());
+
+	    //if(endIndex != -1) {
+	    String key = line.substring(endIndex);
+	    tuples.put("key", key.trim());
+	    //} else {
+	    //logger.debug("@@@@ no key for pageURL: " + pageURL);
+	    //}
+	    /*
+	    if(pageURL.contains(TEXT_END_MARKER)) {
+		logger.debug("@@@@ TEXT_END_MARKER assigned to pageURL for page: ");
+		logger.debug("+++++++++");
+		logger.debug(unparsedPageDump);
+		logger.debug("+++++++++");
+	    }
+	    */
+
+	    boolean readingPageText = false;
+	    StringBuilder pageText = null;
+
+	    // continue reading all other tuples for this page, if any
+	    while((line = reader.readLine()) != null) {
+		line = line.trim();
+
+		// check if we're dealing with metadata or start/end of page's text body
+		// or actual text body
+		
+		if(line.equals(TEXT_START_MARKER)) { // dealing with the page body text
+		    pageText = new StringBuilder();
+		    readingPageText = true;
+		}
+		else if(line.equals(TEXT_END_MARKER)) {
+		    // finished with a page body		    
+		    // Remove any FINAL artificial newline we introduced to a page's body text
+		    tuples.put("pageText", pageText.toString().trim());
+		    readingPageText = false;
+		    pageText = null;
+		}
+		else {
+		    if(readingPageText) { // So we're reading in the page text
+			pageText.append(line);
+			pageText.append("\n"); // there are no newlines within pageText
+			// but if there were newlines, add them back here as readLine() removes them
+		    }
+		    else { // dealing with the rest of the page dump's metadata
+			// QTODO: nutch's text dump output is problematic
+			// strange characters are in the stream and end up here
+			// and can't detect end of metadata or even end of line.
+			endIndex = line.indexOf(":");
+			if(endIndex != -1) { 
+			    String k = line.substring(0, endIndex);
+			    String v = line.substring(endIndex+1);
+			    if(k.startsWith("metadata")) {
+				k = k.substring("metadata".length());
+			    }
+				
+			    tuples.put(k.trim(), v.trim());
+			} else {
+			    if(DEBUG_MODE) {
+				logger.error("No meta key for meta: " + line);
+				logger.error(unparsedPageDump);
+			    }
+			}
+		    }
+		}
+
+	    }
+
+	    // If the page had no pageText, add a "pageText" -> "" mapping
+	    if(!tuples.containsKey("pageText")) {
+		tuples.put("pageText", "");
+	    }
+
+	    
+	} catch (IOException ioe) {
+	    logger.error("@@@@@@@@@ Error reading in txtdump of a page.", ioe);
+	}
+
+	
+	// START DEBUG
+	debugTuples();
+	// END DEBUG
+	
+    }
+
+    public void debugTuples() {
+	if(DEBUG_MODE) {
+	    logger.debug("__________________________________________");
+	    for(Map.Entry<String, String> entry : tuples.entrySet()) {
+		String key = entry.getKey();
+		String value = entry.getValue();	    
+		logger.debug(key + " - " + value);
+	    }
+	    logger.debug("__________________________________________");
+	}
+    }
+    
+    
+    public String getPageURL() {
+	return tuples.get("pageURL");
+    }
+
+    public String getPageText() {
+	return tuples.get("pageText");
+    }
+
+    /* Dr Nichols suggested storing timestamp and char encoding. Not sure which timestamp
+       or encoding he meant, but storing 2 of several timestamps and selecting
+       original character encoding (presumably the char encoding of the page) out of 2
+       pieces of char encoding metadata to store. */
+    public String getModifiedTime() {	
+	// is this the webpage's last mod time?	
+	String time = tuples.get("modifiedTime");
+	time = time.equals("0") ? "" : time; // zero will be assumed to be epoch, rather than unset
+	return time;
+    }    
+    public String getFetchTime() {
+	// is this the nutch crawl time
+	String time = tuples.get("fetchTime");
+	time = time.equals("0") ? "" : time; // zero will be assumed to be epoch, rather than unset
+	return time;
+	
+    } 
+    public String getOriginalCharEncoding() {
+	// is this the web page's char-encoding?
+	return tuples.get("OriginalCharEncoding");
+    }
+    
+    public String get(String key) {
+	return tuples.get(key);
+    }
+
+    public void add(String key, String value) {
+	tuples.put(key, value);
+    }
+
+    /**
+     * IMPORTANT: This method deletes the data stored in this TextDumpPage object
+     * after converting relevant fields and parameters to a WebpageInfo object
+     */
+    public WebpageInfo convertStoredDataToWebpageInfo(
+      long webpageID, int websiteID, boolean isMRI, int totalSentences,
+      ArrayList<SentenceInfo> singleSentences, ArrayList<SentenceInfo> overlappingSentences)
+    {
+	// clear the map, after storing the important (meta)data
+	String pageText = getPageText();
+	String pageURL = getPageURL();
+	String charEncoding = getOriginalCharEncoding();
+	String modifiedTime = getModifiedTime();
+	String fetchTime = getFetchTime();
+
+	WebpageInfo webpage = new WebpageInfo(webpageID, websiteID,
+					      pageText, pageURL, isMRI, totalSentences,
+					      charEncoding, modifiedTime, fetchTime,
+					      singleSentences, overlappingSentences);
+	
+	tuples.clear();
+
+	return webpage;
+    }
+    
+
+
+    /*
+    public void addMRILanguageStatus(boolean status) {
+	if(status) {
+	    tuples.put("isMRI", "true");
+	} else {
+	    tuples.put("isMRI", "false");
+	}
+    }
+
+    public boolean getMRILanguageStatus() {
+	String value = tuples.get("isMRI");
+	if(value == null) {
+	    return false;
+	}
+	if(value.equals("true")) {
+	    return true;
+	}
+	else {
+	    return false;
+	}
+
+    }
+    */
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/TextLanguageDetector.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/TextLanguageDetector.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/TextLanguageDetector.java	(revision 33635)
@@ -0,0 +1,401 @@
+/**
+ * Class that uses OpenNLP with the Language Detection Model to determine, with a default
+ * or configurable level of confidence, whether text (from a file or stdin) is in a given
+ * language or not.
+ * Internal functions can be used for detecting any of the 103 languages currently supported by
+ * the OpenNLP Language Detection Model.
+ * 
+ * http://opennlp.apache.org/news/model-langdetect-183.html
+ * language detector model: http://opennlp.apache.org/models.html
+ *        Pre-trained models for OpenNLP 1.5: http://opennlp.sourceforge.net/models-1.5/
+ * Use of Apache OpenNLP in general:
+ *   http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#intro.api
+ * Use of OpenNLP for language detection:
+ * http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect
+ * 
+ * This code was based on the information and sample code at the above links and the links dispersed throughout this file.
+ * See also the accompanying README file.
+ *
+ * July 2019
+ */
+
+package org.greenstone.atea;
+
+import java.io.*;
+import opennlp.tools.langdetect.*;
+import opennlp.tools.sentdetect.*;
+import opennlp.tools.util.*;
+
+import java.util.ArrayList;
+
+/**
+ * EXPORT OPENNLP_HOME environment variable to be your apache OpenNLP installation.
+ * Create a folder called "models" within the $OPENNLP_HOME folder, and put the file "langdetect-183.bin" in there
+ *    (which is the language detection model zipped up and renamed to .bin extension).
+ *
+ * Then, to compile this program, do the following from the "src" folder (the folder containing this java file):
+ *    maori-lang-detection/src$ javac -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/TextLanguageDetector.java
+ *
+ * Only the subclass MaoriTextDetector.java has a main method at present that can be run.
+ *
+ */
+public class TextLanguageDetector {
+
+    public static final double DEFAULT_MINIMUM_CONFIDENCE = 0.50;
+
+    /**
+     * Configurable: cut off minimum confidence value,
+     * greater or equal to which determines that the best predicted language is
+     * acceptable to user of TextLanguageDetector.
+     */
+    public final double MINIMUM_CONFIDENCE;
+    
+    /** silentMode set to false means TextLanguageDetector won't print helpful messages while running. Set to true to run silently. */
+    public final boolean silentMode;
+
+    private final String OPENNLP_MODELS_RELATIVE_PATH = "models" + File.separator; 
+    
+    /** Language Detection Model file for OpenNLP is expected to be at $OPENNLP_HOME/models/langdetect-183.bin */
+    private final String LANG_DETECT_MODEL_RELATIVE_PATH = OPENNLP_MODELS_RELATIVE_PATH + "langdetect-183.bin";
+
+    /**
+     * The LanguageDetectorModel object that will do the actual language detection/prediction for us.
+     * Created once in the constructor, can be used as often as needed thereafter.
+    */
+    private LanguageDetector myCategorizer = null;
+
+    /** 
+     * The Sentence Detection object that does the sentence splitting for the language
+     * the sentece model was trained for.
+     */
+    private SentenceDetectorME sentenceDetector = null;
+
+
+    /** Constructor with default confidence for language detection.
+     * Does not create sentence model, just the language detection model.
+     */
+    public TextLanguageDetector(boolean silentMode) throws Exception {
+	this(silentMode, DEFAULT_MINIMUM_CONFIDENCE);
+    }
+    
+    /** Constructor with configurable min_confidence for language detection
+     * Does not create sentence model, just the language detection model.
+     */
+    public TextLanguageDetector(boolean silentMode, double min_confidence) throws Exception {
+	this.silentMode = silentMode;
+	this.MINIMUM_CONFIDENCE = min_confidence;
+
+	// 1. Check we can find the Language Detect Model file in the correct location (check that $OPENNLP_HOME/models/langdetect-183.bin exists);
+	String langDetectModelPath = System.getenv("OPENNLP_HOME");
+	if(System.getenv("OPENNLP_HOME") == null) {
+	    throw new Exception("\n\t*** Environment variable OPENNLP_HOME must be set to your Apache OpenNLP installation folder.");
+	}	
+	langDetectModelPath = langDetectModelPath + File.separator + LANG_DETECT_MODEL_RELATIVE_PATH;
+	File langDetectModelBinFile = new File(langDetectModelPath);
+	if(!langDetectModelBinFile.exists()) {
+	    throw new Exception("\n\t*** " + langDetectModelBinFile.getPath() + " doesn't exist."
+				+ "\n\t*** Ensure the $OPENNLP_HOME folder contains a 'models' folder"
+				+ "\n\t*** with the model file 'langdetect-183.bin' in it.");
+	}
+
+
+	// 2. Set up our language detector Model and the Categorizer for language predictions based on the Model.
+	// http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#intro.api
+	// https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html
+	try (InputStream modelIn = new FileInputStream(langDetectModelPath)) {
+
+	    LanguageDetectorModel model = new LanguageDetectorModel(modelIn);
+
+	    // http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect
+	    this.myCategorizer = new LanguageDetectorME(model);
+	}/*catch(Exception e) {
+	    e.printStackTrace();
+	    }*/
+	
+	// instantiating function should handle critical exceptions. Constructors shouldn't.
+	
+    }
+	
+    /** More general constructor that additionally can load up the sentence detector model
+     * for other languages, as long as the provided trained sentence model .bin file exists 
+     * in the OPENNLP_MODELS_RELATIVE_PATH folder. */
+    public TextLanguageDetector(boolean silentMode, double min_confidence,
+			     String sentenceModelFileName) throws Exception
+    {
+	this(silentMode, min_confidence);
+
+	// 3. Set up our sentence model and SentenceDetector object
+	String sentenceModelPath = System.getenv("OPENNLP_HOME") + File.separator
+	    + OPENNLP_MODELS_RELATIVE_PATH + sentenceModelFileName; // "mri-sent_trained.bin" default
+	File sentenceModelBinFile = new File(sentenceModelPath);
+	if(!sentenceModelBinFile.exists()) {	    
+	    throw new Exception("\n\t*** " + sentenceModelBinFile.getPath() + " doesn't exist."
+				+ "\n\t*** Ensure the $OPENNLP_HOME folder contains a 'models' folder"
+				+ "\n\t*** with the model file "+sentenceModelFileName+" in it.");
+	}
+	try (InputStream modelIn = new FileInputStream(sentenceModelPath)) {
+	    // https://www.tutorialspoint.com/opennlp/opennlp_sentence_detection.htm
+	    SentenceModel sentenceModel = new SentenceModel(modelIn);	    
+	    this.sentenceDetector = new SentenceDetectorME(sentenceModel);
+	    
+	} // instantiating function should handle this critical exception
+    }
+
+    /** TODO: Is it sensible to use the Maori Language Sentence Model to split the text
+     * into sentences? What if the text in any other language or a mix of languages?
+     * Doesn't this assume that all languages split sentences alike? */
+    public String[] getAllSentences(String text) {
+		
+	// This function doesn't work if the sentenceDetector object wasn't set up
+	if(sentenceDetector == null) return null;
+
+	String[] sentences = sentenceDetector.sentDetect(text);
+	return sentences;
+    }
+ 
+    public ArrayList<SentenceInfo> getAllSentencesInfo(String[] sentences) {
+
+	if(sentences == null) {
+	    return null;	    
+	}
+
+	ArrayList<SentenceInfo> sentencesList = new ArrayList<SentenceInfo>();
+	for(int i = 0; i < sentences.length; i++) {
+	    String sentence = sentences[i];	    
+	    
+	    //System.err.println(sentence);
+
+	    Language bestLanguage = myCategorizer.predictLanguage(sentence);
+	    double confidence = bestLanguage.getConfidence();
+
+	    sentencesList.add(new SentenceInfo(confidence, bestLanguage.getLang(), sentence));
+	}
+
+	return sentencesList;
+    }
+
+    public ArrayList<SentenceInfo> getAllOverlappingSentencesInfo(String[] sentences) {
+	
+	if(sentences == null) {
+	    return null;	    
+	}
+
+	ArrayList<SentenceInfo> sentencesList = new ArrayList<SentenceInfo>();
+	for(int i = 1; i < sentences.length; i++) {
+	    // glue every two adjacent sentences together
+	    String sentence = sentences[i-1];
+	    
+	    String separator = ". ";
+	    // if the sentence already ends with a terminating punctuation character,
+	    // then separator is just a space
+	    sentence = sentence.trim();
+	    if(sentence.endsWith(".") || sentence.endsWith("?") || sentence.endsWith("!")) {
+		separator = " ";
+	    }
+	    sentence = sentence + separator + sentences[i]; 
+	    
+	    //System.err.println(sentence);
+
+	    Language bestLanguage = myCategorizer.predictLanguage(sentence);
+	    double confidence = bestLanguage.getConfidence();
+
+	    sentencesList.add(new SentenceInfo(confidence, bestLanguage.getLang(), sentence));
+	}
+
+	return sentencesList;
+    }
+    
+    /** 
+     * In this class' constructor, need to have set up the Sentence Detection Model
+     * for the langCode passed in to this function in order for the output to make
+     * sense for that language.
+     * Function that takes a text and returns those sentences in the requested language.
+     * @param text: the string of text from which sentences in the requested
+     * language are to be identified and returned.
+     * @param langCode: 3 letter code of requested language
+     * @param confidenceCutoff: minimum confidence for a SINGLE sentence to be selected
+     * even if the language detector determined the requested language as the primary one
+     * for that sentence. The confidence cutoff provides an additional check.
+     * @return null if no Sentence Detection Model set up in constructor
+     * else returns an ArrayList where:
+     *   - the first element is the total number of sentences in the text parameter
+     *   - remaining elements are the sentences in the text parameter that were in the
+     *     requested language.
+     */
+    public ArrayList<String> getAllSentencesInLanguage(String langCode, String text, double confidenceCutoff)
+    {
+	// big assumption here: that we can split incoming text into sentences
+	// for any language using the sentence model trained for a given language (that of
+	// langCode), despite not knowing what language each sentence in the text param are in.
+	// Hinges on sentence detection in langCode being similar to all others?
+
+	
+	// This function doesn't work if the sentenceDetector object wasn't set up
+	if(sentenceDetector == null) return null;
+	
+	// we'll be storing just those sentences in text that are in the denoted language code
+	ArrayList<String> sentencesInLang = new ArrayList<String>();
+	// OpenNLP language detection works best with a minimum of 2 sentences
+	// See https://opennlp.apache.org/news/model-langdetect-183.html
+	// "It is important to note that this model is trained for and works well with
+	// longer texts that have at least 2 sentences or more from the same language."
+	
+	// For evaluating single languages, I used a very small data set and found that
+	// if the primary language detected is MRI AND if the confidence is >= 0.1, the
+	// results appear reasonably to be in te reo MÄori.
+	
+	String[] sentences = sentenceDetector.sentDetect(text);
+	if(sentences == null) {
+	    sentencesInLang.add("0"); // to indicate 0 sentences in requested language
+	    return sentencesInLang;
+	}
+
+	// add in first element: how many sentences there were in text.
+	sentencesInLang.add(Integer.toString(sentences.length));
+	
+	for(int i = 0; i < sentences.length; i++) {
+	    String sentence = sentences[i];	    
+	    
+	    //System.err.println(sentence);
+
+	    Language bestLanguage = myCategorizer.predictLanguage(sentence);
+	    double confidence = bestLanguage.getConfidence();
+	    
+	    if(bestLanguage.getLang().equals(langCode) && confidence >= confidenceCutoff) {
+		//System.err.println("Adding sentence: " + sentence + "\n");
+		sentencesInLang.add(sentence);		
+	    } //else {
+	    //System.err.println("SKIPPING sentence: " + sentence + "\n");
+	    //}
+	}
+	return sentencesInLang;
+    }
+
+
+    /** @param langCode is 3 letter language code, ISO 639-2/3 
+     * https://www.loc.gov/standards/iso639-2/php/code_list.php
+     * https://en.wikipedia.org/wiki/ISO_639-3
+     * @return true if the input text is Maori (mri) with MINIMUM_CONFIDENCE levels of confidence (if set,
+     * else DEFAULT_MINIMUM_CONFIDENCE levels of confidence).
+     */
+    public boolean isTextInLanguage(String langCode, String text) {
+	// Get the most probable language
+	Language bestLanguage = myCategorizer.predictLanguage(text);
+	doPrint("Best language: " + bestLanguage.getLang());
+	doPrint("Best language confidence: " + bestLanguage.getConfidence());
+
+	return (bestLanguage.getLang().equals(langCode) && bestLanguage.getConfidence() >= this.MINIMUM_CONFIDENCE);
+    }
+    
+    
+    /**
+     * Handle "smaller" textfiles/streams of text read in.
+     * Return value is the same as for isTextInLanguage(String langCode, String text);
+     */
+    public boolean isTextInLanguage(String langCode, BufferedReader reader) throws Exception {
+	// https://stackoverflow.com/questions/326390/how-do-i-create-a-java-string-from-the-contents-of-a-file
+	
+	StringBuilder text = new StringBuilder();
+	String line = null;
+
+	
+	while((line = reader.readLine()) != null) { // readLine removes newline separator
+	    text.append(line + "\n"); // add back (unix style) line ending
+	}
+	return isTextInLanguage(langCode, text.toString());
+    }
+    
+
+    /**
+     * Rudimentary attempt to deal with very large files.
+     * Return value is the same as for isTextInLanguage(String langCode, String text);
+     */    
+    public boolean isLargeTextInLanguage(String langCode, BufferedReader reader) throws Exception {
+	// https://stackoverflow.com/questions/326390/how-do-i-create-a-java-string-from-the-contents-of-a-file
+	
+	final int NUM_LINES = 100; // arbitrary 100 lines read, predict language, calculate confidence
+
+	StringBuilder text = new StringBuilder();
+	String line = null;
+	
+	double cumulativeConfidence = 0;
+	int numLoops = 0;
+	
+	int i = 0;
+	String language = null;
+	
+	while((line = reader.readLine()) != null) { // readLine removes newline separator
+	    text.append(line + "\n"); // add back (unix style) line ending
+	    
+	    i++; // read nth line of numLoop
+	    
+	    
+	    if(i == NUM_LINES) { // arbitrary 100 lines read, predict language, calculate confidence
+		
+		Language bestLanguage = myCategorizer.predictLanguage(text.toString());
+		if(language != null && !bestLanguage.getLang().equals(language)) { // predicted lang of current n lines not the same as predicted lang for prev n lines
+		    doPrintErr("**** WARNING: text seems to contain content in multiple languages or unable to consistently predict the same language.");			
+		}
+		language = bestLanguage.getLang();
+		cumulativeConfidence += bestLanguage.getConfidence();
+		
+		doPrintErr("Best predicted language for last " + NUM_LINES + " lines: " + language + "(confidence: " + bestLanguage.getConfidence() + ")"); 
+		
+		// finished analysing language of NUM_LINES of text
+		text = new StringBuilder();
+		i = 0;
+		numLoops++;
+	    }		
+	}
+	
+	// process any (remaining) text that was less than n NUM_LINES
+	if(!text.toString().equals("")) {
+	    text.append(line + "\n"); // add back (unix style) line ending		
+	    i++;
+	    
+	    Language bestLanguage = myCategorizer.predictLanguage(text.toString());
+	    
+	    if(language != null && !bestLanguage.getLang().equals(language)) { // predicted lang of current n lines not the same as predicted lang for prev n lines
+		doPrintErr("**** WARNING: text seems to contain content in multiple languages or unable to consistently predict the same language.");			
+	    }
+	    language = bestLanguage.getLang();
+	    cumulativeConfidence += bestLanguage.getConfidence();
+	    doPrintErr("Best predicted language for final " + NUM_LINES + " lines: " + language + "(confidence: " + bestLanguage.getConfidence() + ")");
+	}
+	
+	
+	int totalLinesRead = numLoops * NUM_LINES + i; // not used
+	double avgConfidence = cumulativeConfidence/(numLoops + 1); // not quite the average as the text processed outside the loop may have fewer lines than NUM_LINES
+	
+	
+	return (language.equals(langCode) && avgConfidence >= this.MINIMUM_CONFIDENCE);
+    }
+    
+
+    /**
+     * Prints to STDOUT the predicted languages of the input text in order of descending confidence.
+     * UNUSED.
+     */
+    public void predictedLanguages(String text) {
+	// Get an array with the most probable languages
+	
+	Language[] languages = myCategorizer.predictLanguages(text);
+	
+	if(languages == null || languages.length <= 0) {
+	    doPrintErr("No languages predicted for the input text");
+	} else {
+	    for(int i = 0; i < languages.length; i++) {
+		doPrint("Language prediction " + i + ": " + languages[i]);
+	    }
+	}
+	
+    }
+
+    public void doPrint(String msg) {
+	if(!this.silentMode) System.out.println(msg);
+    }
+    public void doPrintErr(String msg) {
+	if(!this.silentMode) System.err.println(msg);
+    }
+    
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/Utility.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/Utility.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/Utility.java	(revision 33635)
@@ -0,0 +1,142 @@
+package org.greenstone.atea;
+
+import java.io.*;
+import java.net.InetAddress;
+import java.util.zip.GZIPInputStream;
+
+import com.maxmind.geoip.*; // for LookupService and Location
+
+import org.apache.log4j.Logger;
+
+public class Utility {
+    private static Logger logger = Logger.getLogger(org.greenstone.atea.Utility.class.getName());
+    
+    // Run gunzip
+    // To avoid making this linux specific, use Java to unzip, instead of running gunzip as process
+    // https://www.mkyong.com/java/how-to-decompress-file-from-gzip-file/
+    public static boolean unzipFile(File inZipFile, File outFile) {	
+	
+	byte[] buffer = new byte[1024];
+
+	// try-with-resources will safely close streams/dispose resources on success or error and exceptions
+	try (	    
+	    GZIPInputStream gzis = new GZIPInputStream(new FileInputStream(inZipFile));	    
+	    FileOutputStream out = new FileOutputStream(outFile);
+	) {
+	    int len;
+	    while ((len = gzis.read(buffer)) > 0) {
+        	out.write(buffer, 0, len);
+	    }
+	    
+	    //gzis.close();
+	    //out.close();
+	    
+	    logger.debug("Unzipped " + inZipFile + " to " + outFile);
+	    
+	} catch(IOException ex) {
+	    //error("Failed to unzip " + inZipFile);
+	    System.err.println("Failed to unzip " + inZipFile);
+	    ex.printStackTrace();
+	    return false;
+	}
+
+	return true;
+    }
+
+    /**
+     * Attribution following below is as per https://dev.maxmind.com/geoip/geoip2/geolite2/
+     *
+     * This product includes GeoLite2 data created by MaxMind, available from
+     * <a href="https://www.maxmind.com">https://www.maxmind.com</a>.
+     *
+     * Usage:
+     * https://stackoverflow.com/questions/1415851/best-way-to-get-geo-location-in-java
+     * version I'm using: https://github.com/maxmind/geoip-api-java
+     * Newer version: https://maxmind.github.io/GeoIP2-java/
+     *
+     * @return 2 letter countrycode in uppercase or an exception
+     */
+    public static String getCountryCodeOfDomain(String domainWithProtocol, File geoLiteCityDatFile)
+	throws Exception
+    {	
+	int startIndex = domainWithProtocol.indexOf("//"); // http:// or https:// prefix
+	startIndex = (startIndex == -1) ? 0 : (startIndex+2); // skip past the protocol's // portion
+	String domain = domainWithProtocol.substring(startIndex);	
+	
+	// pass in the GeoLiteCity.dat file to be able to do the location lookup for domain's IP
+	LookupService cl = new LookupService(geoLiteCityDatFile, LookupService.GEOIP_MEMORY_CACHE);
+	    
+	// get IP for domain
+	InetAddress inetAddress = InetAddress.getByName(domain);
+	String ipAddress = inetAddress.getHostAddress();
+	
+	// get location object for IP
+	Location location = cl.getLocation(ipAddress);
+	
+	if(location == null) {
+	    throw new Exception("@@@@ No location info in DB for: " + domain);
+	} else {
+	    return location.countryCode;
+	}
+	
+    }
+    
+    /** Work out the 'domain' for a given url.
+     * This retains any www. or subdomain prefix.
+     */
+    public static String getDomainForURL(String url, boolean withProtocol) {
+	int startIndex = startIndex = url.indexOf("//"); // for http:// or https:// prefix
+	startIndex = (startIndex == -1) ? 0 : (startIndex+2); // skip past the protocol's // portion
+	// the keep the URL around in case param withProtocol=true
+	String protocol = (startIndex == -1) ? "" : url.substring(0, startIndex);
+	
+	String domain = url.substring(startIndex);
+	int endIndex = domain.indexOf("/");
+	if(endIndex == -1) endIndex = domain.length();
+	domain = domain.substring(0, endIndex);
+
+	if(withProtocol) {
+	    // now that we have the domain (everything to the first / when there is no protocol)
+	    // can glue the protocol back on
+	    domain = protocol + domain;
+	}
+	
+	return domain;
+    }
+    
+    public static boolean isDomainInCountry(String domainWithProtocol,
+					    String countryCode, File geoLiteCityDatFile)
+    {
+	countryCode = countryCode.toUpperCase();
+	
+	int startIndex = domainWithProtocol.indexOf("//"); // http:// or https:// prefix
+	startIndex = (startIndex == -1) ? 0 : (startIndex+2); // skip past the protocol's // portion
+	String domain = domainWithProtocol.substring(startIndex);	
+	
+	boolean result = false;
+	try {
+	    // pass in the GeoLiteCity.dat file to be able to do the location lookup for domain's IP
+	    LookupService cl = new LookupService(geoLiteCityDatFile, LookupService.GEOIP_MEMORY_CACHE);
+	    
+	    // get IP for domain
+	    InetAddress inetAddress = InetAddress.getByName(domain);
+	    String ipAddress = inetAddress.getHostAddress();
+	    
+	    // get location object for IP
+	    Location location = cl.getLocation(ipAddress);
+
+	    if(location != null) {
+		// compare country code with tld parameter
+		System.err.println("@@@@ Got country code: " + location.countryCode);
+		result = location.countryCode.equals(countryCode);
+	    } else {
+		System.err.println("@@@@ No location info in DB for: " + domainWithProtocol);
+	    }
+	} catch(Exception e) {
+	    e.printStackTrace();
+	    System.err.println("Could not check if domain " + domain + " was in country: " + countryCode);
+	} finally {
+	    return result;
+	}
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/WETProcessor.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/WETProcessor.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/WETProcessor.java	(revision 33635)
@@ -0,0 +1,326 @@
+package org.greenstone.atea;
+
+
+import java.io.*;
+import java.util.Properties;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.TreeSet;
+
+import org.apache.log4j.Logger;
+
+/**
+ * A single instance of the WETProcessor class can process a single unzipped warc.wet file.
+ * A WETProcessor take a warc.wet file and goes through all its WET records,
+ * putting each WET record into a file. Each file is put into a keep, discard or greylisted folder
+ * and its url listed written into a keep, discard or greylisted text file, based on:
+ *
+ * 1. whether it's whitelisted, else greylisted else blacklisted
+ * 2. and if explicitly whitelisted or else not greylisted or blacklisted and there's
+ * enough content. Formerly, content-length and number of lines were used to determine if
+ * the content was sufficient. Now it's just word count and number of MAX characters
+ * (not MINIMUM characters) that determine a string is a word.
+ * Explicit whitelisting has precedence over greylisting and which takes precedence
+ * over blacklisting in turn.
+ * However, even explicitly whitelisted urls still need to have sufficient content to end 
+ * up in keepURLs.txt.
+ *
+ * See CCWETProcessor.java for compile instructions and how to run.
+ *
+*/
+public class WETProcessor {
+    private static Logger logger = Logger.getLogger(org.greenstone.atea.WETProcessor.class.getName());
+
+    // WARC WET header lines and header line prefixes of interest
+    static final String WARC_RECORD_START = "WARC/1.0";
+    static final String WARC_INFO_HEADER = "WARC-Type: warcinfo";
+    static final String WARC_TARGET_URI_HEADER_PREFIX = "WARC-Target-URI:";
+    static final String WARC_CONTENT_LENGTH_HEADER_PREFIX = "Content-Length:";    
+
+    private final String crawlID;
+    private final int WETFileID;
+    private final File inFile;
+
+    private int recordCount = 0;
+
+    /** Handle to a CCWETProcessor that processes a set of WET files
+     * Whereas a WETProcessor instance only processes a single WET file
+     * containing multiple WET records.
+     */
+    private CCWETProcessor batchProcessor;
+    
+    /**
+     * WET processor processes a single warc.wet file containing multiple WET records
+     * containing text identified as primary langcode=mri. Each individual WET record is written
+     * out to a uniquely named file in either the keep or discard folder depending on the WET
+     * record's content length and number of lines of actual content (excluding WARC headers).
+     * @param inFile the warc.wet file whose WET records are to be processed
+     * @param crawlID is the ID of the commoncrawl containing this warc.wet file
+     * and is of the format YYYY-## (of full crawlID CC-MAIN-YYYY-##) which will be used
+     * as prefix to create unique filenames when storing each individual record).
+     */
+    public WETProcessor(File inFile, String crawlID, CCWETProcessor batchProcessor) {
+	this.batchProcessor = batchProcessor;
+	
+	this.inFile = inFile;
+	this.crawlID = crawlID;
+
+	// We just want a unique recordID prefix, which we get from concatenating
+	// the commoncrawl ID with the wet file name suffix and record count within the file:
+	// inFile name looks something like MAORI-CC-2019-30-20190902100139-000000.warc.wet
+	// the prefix will be everything after the last hyphen and without file extension,
+	// so "000000" in our example. Then converted into a number and padded to 2, e.g. 00.
+	// Then prefix the crawlID and suffix the 4-digit padded recordCount keeping track
+	// of the current WET record to get a unique filename to store each WET record into.
+	// e.g. 2019-30-00-0015 is the 15th WET record in the *00.warc.wet file of the
+	// common crawl CC-MAIN-2019-30 (15th WET record of CC-MAIN-2019-30-*-000000.warc.wet.gz)
+
+	String fileID = inFile.getName();
+	//logger.debug("*** Processing wetfile: " + fileID);		    
+	fileID = fileID.substring(fileID.lastIndexOf("0")+1);
+	if(fileID.startsWith(".")) { // took off too many zeroes, as happens with *000000.warc.wet
+	    this.WETFileID = 0;
+	} else {
+	    fileID = fileID.substring(0, fileID.indexOf("."));
+	    this.WETFileID = Integer.parseInt(fileID);
+	}
+    }
+
+    /**
+     * Processes all the WET records of a single warc.wet file
+     */
+    public int processWETFile() {
+	File keepURLsFile = this.batchProcessor.keepURLsFile;
+	File discardURLsFile = this.batchProcessor.discardURLsFile;
+	File greyListedFile = this.batchProcessor.greyListedFile;
+	
+	StringBuilder record = null;
+	String line = null;	
+	boolean readingRecord = false;
+
+	String WARCtargetURI = "";
+	
+	//int recordCount = 0;
+	
+	int contentLength = -1; // of record
+	int lineCount = -1; // actual number of non-empty lines in record body (i.e. excludes WET/WARC headers)
+	    
+	// read from WETfile
+	try (
+	     BufferedReader reader = new BufferedReader(new FileReader(this.inFile));
+	     BufferedWriter keepURLsWriter = new BufferedWriter(new FileWriter(keepURLsFile, true));
+	     BufferedWriter discardURLsWriter = new BufferedWriter(new FileWriter(discardURLsFile, true)); // true to append
+	     BufferedWriter greyListedURLsWriter = new BufferedWriter(new FileWriter(greyListedFile, true)); // true to append
+	     ) {
+	    
+	    while((line = reader.readLine()) != null) { // readLine removes newline separator
+
+		if(line.startsWith(WARC_INFO_HEADER)) { // "WARC-Type: warcinfo"
+		    readingRecord = false;
+		    record = null; // drop this record, which is just an info record not actual web page's text
+		    recordCount--;
+		    continue;
+		}
+		
+		if(line.startsWith(WARC_RECORD_START)) { // "WARC/1.0" means finished previous WET record
+		    // process any previous record
+		    if(record != null) {
+			processWETrecord(keepURLsWriter, discardURLsWriter, greyListedURLsWriter,
+					 recordCount, contentLength, lineCount,
+					 WARCtargetURI, record.toString());
+			record = null;
+			contentLength = -1;
+			lineCount = -1;
+		    }
+		    
+		    recordCount++;		    
+		    // get ready to start a new record
+		    readingRecord = true;
+		    record = new StringBuilder();
+		}
+
+		if(readingRecord) { // append current line to current record
+
+		    if(line.startsWith(WARC_TARGET_URI_HEADER_PREFIX)) { // "WARC-Target-URI:"
+			// get and store the value
+			WARCtargetURI = line.substring(WARC_TARGET_URI_HEADER_PREFIX.length()).trim();
+		    }
+		    
+		    record.append(line + "\n"); // add back (unix style) line ending
+
+		    // if the line is non-empty
+		    // AND if we've started counting lines, which happens only when the current
+		    // line is past WARC/WET headers and we're into the actual body portion
+		    // of the WET record,
+		    // start incrementing the line counter.
+		    if(lineCount >= 0 && !line.trim().equals("")) {
+			lineCount++;
+		    }
+		    else if(line.startsWith(WARC_CONTENT_LENGTH_HEADER_PREFIX)) { // "Content-Length:"
+			String contentLengthStr = line.substring(WARC_CONTENT_LENGTH_HEADER_PREFIX.length()).trim();
+			contentLength = Integer.parseInt(contentLengthStr);
+			lineCount = 0;
+		    }
+		    
+		}
+		
+	    }
+
+	    // flush the last record. If it was a warcinfo record, record would be null here
+	    if(record != null) {
+		processWETrecord(keepURLsWriter, discardURLsWriter, greyListedURLsWriter,
+				 recordCount, contentLength, lineCount,
+				 WARCtargetURI, record.toString());
+		record = null;
+	    }
+	    
+	} catch(IOException ioe) {
+	    ioe.printStackTrace();
+	}
+
+	return recordCount;
+    }
+
+    public int getRecordCount() { return this.recordCount; }
+
+    /**
+     * Determines if a WET record belongs in the keep or discard pile depending on if it
+     * contains enough text, based on contentLength and line count of the record body.
+     * Then writes out the WET record to a uniquely named file in the keep or discard folder,
+     * and writes out the WET record's URL to the keepURLs.txt file or discardURLs.txt file.
+     */
+    private void processWETrecord(BufferedWriter keepURLsWriter, BufferedWriter discardURLsWriter,
+				  BufferedWriter greyListedURLsWriter,
+				  int recordID, int contentLength, int lineCount,
+				  String recordURI, String record)
+    {
+	logger.info("CrawlID: CC-MAIN-" + this.crawlID
+			   + " WET #" + this.WETFileID
+			   + " record #" + recordID
+			   + " - contentLength: " + contentLength
+			   + " - lineCount: " + lineCount);
+	logger.info("URI: " + recordURI);
+	//logger.debug(record);
+	//logger.info("--------------------------");
+
+	File parentFolder = null;
+
+	if(batchProcessor.isBlacklisted(recordURI)) {
+	    
+	    // explicit whitelisting overrides blacklisting
+	    if(batchProcessor.isWhitelisted(recordURI)) {
+		parentFolder = batchProcessor.keepFolder; //tentative
+	    }
+	    // if not whitelisted, then greylisting still overrides blacklisting
+	    else if(batchProcessor.isGreylisted(recordURI)) {
+		parentFolder = batchProcessor.greyListedFolder;
+		logger.debug("@@@GREYLISTED");
+	    }
+	    else { // url was only blacklisted
+		parentFolder = batchProcessor.discardFolder;
+		logger.debug("@@@DISCARDING - blacklisted");
+	    }
+	}
+	else if(batchProcessor.isGreylisted(recordURI)) { // e.g. products sites
+	    // explicit whitelisting overrides greylisting
+	    if(batchProcessor.isWhitelisted(recordURI)) {
+		parentFolder = batchProcessor.keepFolder; // tentative
+	    }
+	    else {
+		parentFolder = batchProcessor.greyListedFolder;
+		logger.debug("@@@GREYLISTED");
+	    }
+	}
+
+	// If URL was not blacklisted/greylisted, or was even explicitly whitelisted,
+	// it still can't be in the keep list as it needs further inspection:
+	// it needs sufficient content for language analysis.
+	// We don't care about the combination of number of lines and content-length,
+	// we just care about the number of "valid words" as defined by us.
+	if(parentFolder != batchProcessor.greyListedFolder && parentFolder != batchProcessor.discardFolder) { // i.e. parentFolder == keepFolder if whiteListed || parentFolder == null
+	    
+	    // If a web page's WET record contains a certain minimum number of words,
+	    // we will think it's a meaningful web page and has sufficient content for text analysis
+	    // to have been successful. Cut off values at present are:
+	    // - a minimum of 20 words
+	    // - a word consists of 1 to 15 chars demarcated by spaces. Any more chars may point
+	    // to words having been glued together. This is used by irrelevant sites and moreover
+	    // can't be analysed for language, so may not be actually MRI.
+	    
+	    // Though StringTokenizer still in use, as seen in discussion at
+	    // https://stackoverflow.com/questions/6983856/why-is-stringtokenizer-deprecated
+	    // String.split(regex) seems better for splitting on general whitespace
+	    String[] allWords = record.split("\\s");
+	    int validWordCount = 0;
+	    int numCamelCaseWords = 0;
+	    for(int i = 0; i < allWords.length; i++) {
+		String word = allWords[i];
+
+		// throw away if n words contain camelcase, which is another case of words glued together
+		// For now, we'll only skip camelcased words in our count of valid words
+		if(word.matches(".*[a-z][A-Z].*") && word.length() >= 5) {
+		    numCamelCaseWords++;		    
+		}		
+		// In Maori, word length of 1 is not uncommon
+		// but let's skip camelcased words when counting valid words
+		else if(word.length() >= 1 && word.length() <= batchProcessor.MAX_WORD_LENGTH) {
+		    validWordCount++;
+		}
+	    }
+
+	    
+	    /*
+	    // dump if too many camelcase words (ideally keep no WET record of that kind?)
+	    if(numCamelCaseWords >= batchProcessor.MAX_WORDS_CAMELCASE) {
+		parentFolder = batchProcessor.discardFolder;
+		logger.debug("@@@DISCARDING - CAMELCASED CONTENTS");
+	    }
+	    else*/
+	    // For now, don't discount content with too many camelcased words
+	    // Just focus on whether there are a sufficient number of valid words
+	    // (camelcased words are however still ignored in our count of valid words)
+	    if(validWordCount >= batchProcessor.MIN_NUM_WORDS) { // otherwise, keep anything with a sufficient number of valid words
+		parentFolder = batchProcessor.keepFolder;
+		logger.debug("@@@KEEPING");
+	    }
+	}
+	// if parentFolder still not set, it means that the content length/num words or lines
+	// were insufficient, so meant to be discarded
+	if(parentFolder == null) {
+	    parentFolder = batchProcessor.discardFolder;
+	    logger.debug("@@@DISCARDING");
+	}
+
+	try {
+	    if (parentFolder == batchProcessor.keepFolder) {
+		keepURLsWriter.write(recordURI + "\n");
+	    } else if (parentFolder == batchProcessor.greyListedFolder) {
+		greyListedURLsWriter.write(recordURI + "\n");
+	    } else {
+		discardURLsWriter.write(recordURI + "\n");
+	    }
+	} catch(Exception e) {
+	    logger.debug("Unable to write URL");
+	    e.printStackTrace();
+	}
+   
+	logger.debug("--------------------------");
+
+	// outFilename will look something like YYYY-##-####
+	String outFilename = String.format("%s-%02d-%04d", this.crawlID, this.WETFileID, recordID);
+	    //= this.crawlID + "-" + String.format("%02d", this.WETFileID) + "-" + String.format("%04d.txt", recordID);
+	File outFile = new File(parentFolder, outFilename);
+	
+	try (BufferedWriter writer = new BufferedWriter(new FileWriter(outFile))) {
+	    writer.write(record);
+	    // Try-with-resources examples don't call close() explicitly:
+	    // https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html
+	    //writer.close();
+	} catch(IOException ioe) {
+	    ioe.printStackTrace();
+	    logger.error("@@@@@@@@@ Error writing to file " + outFile, ioe);
+	}
+    }
+
+
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/WebpageInfo.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/WebpageInfo.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/WebpageInfo.java	(revision 33635)
@@ -0,0 +1,47 @@
+package org.greenstone.atea;
+
+import java.util.ArrayList;
+
+public class WebpageInfo {
+
+    /** db table ids */
+    public final long webpageID;
+    public final int websiteID;
+
+    public final int totalSentences;
+    
+    public final String text;
+    public final String URL;
+    public final boolean isMRI;
+    
+    public final String charEncoding;
+    public final String modifiedTime;
+    public final String fetchTime;
+    public final ArrayList<SentenceInfo> singleSentences;
+    public final ArrayList<SentenceInfo> overlappingSentences;
+	
+    public WebpageInfo (long webpageID, int websiteID,
+			String pageText, String pageURL, boolean isMRI, int totalSentences,
+			String charEncoding, String modifiedTime, String fetchTime,
+			ArrayList<SentenceInfo> singleSentences,
+			ArrayList<SentenceInfo> overlappingSentences)
+    {
+
+	this.webpageID = webpageID;
+	this.websiteID = websiteID;
+
+	this.totalSentences = totalSentences;
+	
+	this.text = pageText;
+	this.URL = pageURL;
+	this.isMRI = isMRI;
+	
+	this.charEncoding = charEncoding;
+	this.modifiedTime = modifiedTime;
+	this.fetchTime = fetchTime;
+
+	this.singleSentences = singleSentences;
+	this.overlappingSentences = overlappingSentences;
+	
+    }
+}
Index: /other-projects/maori-lang-detection/src/org/greenstone/atea/WebsiteInfo.java
===================================================================
--- /other-projects/maori-lang-detection/src/org/greenstone/atea/WebsiteInfo.java	(revision 33635)
+++ /other-projects/maori-lang-detection/src/org/greenstone/atea/WebsiteInfo.java	(revision 33635)
@@ -0,0 +1,40 @@
+package org.greenstone.atea;
+
+public class WebsiteInfo {
+
+    public final int id;
+    public final String siteFolderName;
+    public final String domain;
+    
+    public final int totalPages;
+    public final int countOfWebPagesWithBodyText;
+    public final int numPagesInMRI;
+    
+    public final long siteCrawledTimestamp;
+    public final boolean siteCrawlUnfinished;
+    public final boolean redoCrawl;
+    
+    public final String geoLocationCountryCode;
+    public final boolean urlContainsLangCodeInpath;
+	
+    public WebsiteInfo(int siteCount, String siteFolderName, String domainOfSite,
+		       int totalPages, int countOfWebPagesWithBodyText, int numPagesInMRI,
+		       long siteCrawledTimestamp, boolean siteCrawlUnfinished, boolean redoCrawl,
+		       String geoLocationCountryCode, boolean urlContainsLangCodeInpath)
+    {
+	this.id = siteCount;
+	this.siteFolderName = siteFolderName;
+	this.domain = domainOfSite;
+	
+	this.totalPages = totalPages;
+	this.countOfWebPagesWithBodyText = countOfWebPagesWithBodyText;
+	this.numPagesInMRI = numPagesInMRI;
+	
+	this.siteCrawledTimestamp = siteCrawledTimestamp;
+	this.siteCrawlUnfinished = siteCrawlUnfinished;
+	this.redoCrawl = redoCrawl;
+	
+	this.geoLocationCountryCode = geoLocationCountryCode;
+	this.urlContainsLangCodeInpath = urlContainsLangCodeInpath;
+    }
+}
