Index: /gs3-extensions/maori-lang-detection/README.txt
===================================================================
--- /gs3-extensions/maori-lang-detection/README.txt	(revision 33335)
+++ /gs3-extensions/maori-lang-detection/README.txt	(revision 33335)
@@ -0,0 +1,146 @@
+http://opennlp.apache.org/news/model-langdetect-183.html
+Language Detector Model for Apache OpenNLP released
+
+0. Can we make it run
+Can we detect it with with Eng, Fr, NL docs
+
+1. openNLP - is Maori included,
+if not learn how to teach it to recognise Maori
+how you run the training bit of their software and add in Maori language training set
+
+ANSWER: Yes, Maori is included, see https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+
+2. Add in a small program to detect particularly Maori
+
+3. Macron and non-macron input text language recognition support?
+
+-----------------------------------------
+
+READING:
+	General:
+	* https://stackoverflow.com/questions/7670427/how-does-language-detection-work
+	* https://github.com/andreasjansson/language-detection.el	
+	* https://en.wikipedia.org/wiki/ISO_639-3
+
+	Specific:
+	* openNLP download: http://opennlp.apache.org/download.html
+		"Models
+		The models for Apache OpenNLP are found here.
+		The models can be used for testing or getting started, please train your own models for all other use cases."
+
+	* On LanguageDetectionModel for OpenNLP:
+		http://opennlp.apache.org/news/model-langdetect-183.html
+		https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect.classifying
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.langdetect
+	* Good precision etc on detecting Maori: https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/langdetect-183.bin.report.txt
+	* Maybe useful: http://opennlp.sourceforge.net/models-1.5/
+		"Use the links in the table below to download the pre-trained models for the OpenNLP 1.5 series."
+
+
+
+1. Download OpenNLP from https://opennlp.apache.org/download.html.
+I got the binary version. I unzipped it.
+
+2. Get the model, langdetect-183.bin, from: http://opennlp.apache.org/models.html
+	Direct download link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin
+The page says:
+	"All models are zip compressed (like a jar file), they must not be uncompressed."
+- So when langdetect-183.bin is renamed to langdetect-183.zip, can inspect the contents.
+- But for using with openNLP, don't rename or unzip it.
+
+3. UNNECESSARY:
+I started by following the instructions at the botton of: 
+https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+	Note that the svn checkout from the leipzig data link is too huge. So I restricted it to just Maori, English, Dutch and French text, see further below.
+
+These instructions proved ultimately unnecessary, as the downloaded openNLP model for language detection, langdetect-183.bin, already contains all that
+
+
+Individual downloading of Leipzig corpora for just Maori or specific languages can be found via:
+	http://wortschatz.uni-leipzig.de/en/download/
+
+[svn co https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig]
+https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig/
+Here's the svn web view for the leipzig/data language files: https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig/data/
+
+[Here's the svn web view of apache.org: http://svn.eu.apache.org/viewvc
+Unfortunately this is other apache stuff, including openNLP, but not the leipzig language data files I was after]
+
+
+	svn co https://svn.apache.org/repos/bigdata/opennlp/trunk --depth immediates
+	mv trunk opennlp-corpus
+	svn up --set-depth immediates
+	cd resources/
+	wharariki:[197]/Scratch/ak19/openNLP-lang-detect/opennlp-corpus/leipzig/resources>svn up --set-depth infinity
+	cd ../data
+	svn up mri_web_2011_100K-sentences.txt
+	svn up eng_wikipedia_2012_3M-sentences.txt
+	svn up nld_mixed_2012_1M-sentences.txt
+	svn up fra_mixed_2009_1M-sentences.txt
+
+	cd ..
+	# in opennlp-corpus/leipzig
+	chmod u+x create_langdetect_model.sh
+
+
+	cd /Scratch/ak19/openNLP-lang-detect/apache-opennlp-1.9.1
+	export OPENNLP_HOME=`pwd`
+	./create_langdetect_model.sh mri_web_2011_100K-sentences.txt
+	./create_langdetect_model.sh nld_mixed_2012_1M-sentences.txt
+
+
+4. Attempting running OpenNLP + LanguageDetectorModel's command line tools for language prediction
+
+I couldn't get prediction to work from the command line, so I ended up writing that part as a Java class. But I could run some part of the command line tools with the instructions on the following pages. 
+
+
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect.classifying
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.langdetect
+	"Usage: opennlp LanguageDetector model < documents"
+
+# must have exported OPENNLP_HOME (maybe add its bin to PATH?)
+# Following the Usage instruction just above:
+	$ cd /Scratch/ak19/openNLP-lang-detect
+	$ ./apache-opennlp-1.9.1/bin/opennlp LanguageDetector langdetect-183.bin < opennlp-corpus-downloaded/mri-nz_web_2017_100K/mri-nz_web_2017_100K-words.txt
+
+	[# Sending all output into a file for inspection:
+	$ ./apache-opennlp-1.9.1/bin/opennlp LanguageDetector langdetect-183.bin < opennlp-corpus-downloaded/mri-nz_web_2017_100K/mri-nz_web_2017_100K-words.txt > bla.txt 2>&1
+	]
+
+
+
+	[# Didn't work, even though the first opennlp manual link above said that the "opennlp LangDetector model" command should take input from stdin:
+
+	./apache-opennlp-1.9.1/bin/opennlp LanguageDetector langdetect-183.bin < "Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei."
+	]
+
+
+5. For writing Java code:
+To write the basic code, I followed the Java skeleton examples at
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html
+	* http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect.classifying
+
+
+To get the imports correct, I searched for example/tutorial java code on using openNLP with the LanguageDetectorModel 
+* Java code: Import files:
+https://www.tutorialkart.com/opennlp/language-detector-example-in-apache-opennlp/
+* The tutorial link above also covers Java code to train detecting a particular language.
+
+
+apache-opennlp-1.9.1/lib/opennlp-tools-1.9.1.jar contains the imports we want, in particular 
+	import opennlp.tools.langdetect.*;
+	import opennlp.tools.util.*;
+
+6. Wrote the very basic form of MaoriDetector.java class.
+
+To compile and run:
+
+a. export OPENNLP_HOME=/Scratch/ak19/openNLP-lang-detect/apache-opennlp-1.9.1
+b. wharariki:[115]/Scratch/ak19/openNLP-lang-detect/src>javac -cp ".:$OPENNLP_HOME/lib/*" MaoriDetector.java
+c. wharariki:[116]/Scratch/ak19/openNLP-lang-detect/src>java -cp ".:$OPENNLP_HOME/lib/*" MaoriDetector
+
+
+(Though possibly the only jar file needed in $OPENNLP_HOME/lib is opennlp-tools-1.9.1.jar) 
Index: /gs3-extensions/maori-lang-detection/src/MaoriDetector.java
===================================================================
--- /gs3-extensions/maori-lang-detection/src/MaoriDetector.java	(revision 33335)
+++ /gs3-extensions/maori-lang-detection/src/MaoriDetector.java	(revision 33335)
@@ -0,0 +1,66 @@
+/**
+ * http://opennlp.apache.org/news/model-langdetect-183.html
+ * language detector model: http://opennlp.apache.org/models.html
+ * Use of Apache OpenNLP in general:
+ *   http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#intro.api
+ * Use of OpenNLP for language detection:
+ * http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect
+ * 
+ * This code was based on the information and sample code at the above links and the links dispersed throughout this file.
+ */
+
+import java.io.*;
+import opennlp.tools.langdetect.*;
+import opennlp.tools.util.*;
+
+/**
+ * Run as:
+ *    wharariki:[115]/Scratch/ak19/openNLP-lang-detect/src>javac -cp ".:$OPENNLP_HOME/lib/*" MaoriDetector.java
+ *    wharariki:[116]/Scratch/ak19/openNLP-lang-detect/src>java -cp ".:$OPENNLP_HOME/lib/*" MaoriDetector
+ *
+ * https://stackoverflow.com/questions/219585/including-all-the-jars-in-a-directory-within-the-java-classpath
+ * Also has information on how to run this class if it's in a Java package.
+ */
+public class MaoriDetector {
+
+    /** 
+     * Taken from our university website
+     * https://www.waikato.ac.nz/maori/
+     */
+    public static final String TEST_INPUT_TEXT = "Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei.";
+    
+    public static void main(String args[]) {
+	// http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#intro.api
+	// https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html
+	try (InputStream modelIn = new FileInputStream("/Scratch/ak19/openNLP-lang-detect/langdetect-183.bin")) {
+
+	    LanguageDetectorModel model = new LanguageDetectorModel(modelIn);
+
+	    // http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.langdetect
+	    LanguageDetector myCategorizer = new LanguageDetectorME(model);
+
+	    // Get the most probable language
+	    Language bestLanguage = myCategorizer.predictLanguage(TEST_INPUT_TEXT);
+	    System.out.println("Best language: " + bestLanguage.getLang());
+	    System.out.println("Best language confidence: " + bestLanguage.getConfidence());
+
+	    
+	    // Get an array with the most probable languages
+	    Language[] languages = myCategorizer.predictLanguages(TEST_INPUT_TEXT);
+	    /*
+	    if(languages == null || languages.length <= 0) {
+		System.err.println("No languages predicted for the input text");
+	    } else {
+		for(int i = 0; i < languages.length; i++) {
+		    System.out.println("Language prediction " + i + ": " + languages[i]);
+		}
+	    }*/
+	    
+	} catch(Exception e) {
+	    e.printStackTrace();
+	}
+	
+	System.err.println("Exitting program...\n");
+	System.exit(0);
+    }
+}
