Index: gs3-extensions/maori-lang-detection/README.txt
===================================================================
--- gs3-extensions/maori-lang-detection/README.txt	(revision 33350)
+++ gs3-extensions/maori-lang-detection/README.txt	(revision 33355)
@@ -8,12 +8,28 @@
     svn co http://svn.greenstone.org/gs3-extensions/maori-lang-detection
 
-- It contains the OpenNLP that was current at the time of the commit, but you can also get it from its original site http://opennlp.apache.org/download.html
-- The LanguageDetectionModel to be used by OpenNLP is also included (again, current at the time of commit), but you can get it from http://opennlp.apache.org/models.html (direct link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin)
-
-1. Once you've svn checked it out, create a folder called "models" and put the langdetect-183.bin file into it.
-(This is just a zip file, but has to remain with the .bin extension in order for OpenNLP to use it. If you ever wish to see its contents, you can rename to .zip and use the Archive Manager or other Zip tool to inspect the contents.)
+This checkout contains:
+- tarball of the OpenNLP version that was current at the time of the commit, but you can also get it from its original site http://opennlp.apache.org/download.html
+- the folder "models-trainingdata-and-sampletxts", itself containing:
+	- langdetect-183.bin: The LanguageDetectionModel to be used by OpenNLP is also included (again, current at the time of commit), but you can get it from http://opennlp.apache.org/models.html (direct link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin)
+	- mri-sent_trained.bin: our own custom generated model for Maori Sentence Detection
+	- mri-sent.train: the training text file for generating the mri-sent_trained.bin Maori Sentence Detection model
+	- sample_mri_paragraphs.txt: contains some text to test the Maori Sentence Detection model on. Its content is from Niupepa collection page http://www.nzdl.org/cgi-bin/library?gg=text&e=p-00000-00---off-0niupepa--00-0----0-10-0---0---0direct-10---4-------0-1l--11-en-50---20-about---00-0-1-00-0-0-11-1-0utfZz-8-00&a=p&p=about&l=mi&nw=utf-8)
+	- sample_maori_shorttext.txt: to test the MaoriTextDetector.java with
+- "src" folder for Java classes, currently just MaoriTextDetector.java and its classfile. MaoriTextDetector.java uses the aforementioned LanguageDetectionModel, langdetect-183.bin, to detect whether input text from a file or stdin is in Maori or not
+- gen_SentenceDetection_model.sh, our custom script that generates both the mri-sent.train and model mri-sent_trained.bin files mentioned above
+	- the script works on opennlp's leipzig corpus of 100k Maori sentences from 2011 to get its sample sentences into the correct format in the mri-sent.train file
+	- from this file containing training sentences, it generates the Sentence Detector Model, mri-sent_trained.bin
+- mri-opennlp-corpus.tar.gz: a tarball containing the 100k Maori sentences opennlp corpus checked out with svn in its original directory structure from https://svn.apache.org/repos/bigdata/opennlp/trunk/mri_web_2011_100K-sentences.txt
+
+
+1. Once you've svn checked out the maori-lang-detection from gs3-extensions, create a folder called "models" and put the files langdetect-183.bin and mri-sent_trained.bin from the folder "models-trainingdata-and-sampletxts" into it.
+(These are just zip files, but have to remain with the .bin extension in order for OpenNLP to use them. If you ever wish to see the contents of such a .bin file, you can rename to .zip and use the Archive Manager or other Zip tool to inspect the contents.)
+
+You can optionally put the file models-trainingdata-and-sampletxts/
+
 
 2. Next extract the apache-opennlp-1.9.1-bin.tar.gz.
 This will create a folder called apache-opennlp-1.9.1. Move the "models" folder you created in step 1 into this folder.
+
 
 3. Before you can compile or run the MaoriTextDetector program, you always have to prepare a terminal by setting up the environment for OpenNLP as follows:
@@ -25,4 +41,5 @@
 Still in the SAME terminal as where you set up the OPENNLP_HOME environment in step 3, you can now run:
       maori-lang-detection/src$ javac -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" MaoriTextDetector.java
+
 
 5. To run the MaoriTextDetector program, you will need the JDK or JRE 7+ bin folder on your PATH and still in the SAME terminal as where you set up the OPENNLP_HOME environment in step 3,
@@ -109,4 +126,5 @@
 I got the binary version. I unzipped it.
 
+
 2. Get the model, langdetect-183.bin, from: http://opennlp.apache.org/models.html
 	Direct download link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin
@@ -116,4 +134,5 @@
 - But for using with openNLP, don't rename or unzip it.
 
+
 3. UNNECESSARY:
 I started by following the instructions at the botton of: 
@@ -135,14 +154,19 @@
 
 
-	svn co https://svn.apache.org/repos/bigdata/opennlp/trunk --depth immediates
-	mv trunk opennlp-corpus
-	svn up --set-depth immediates
+	svn co --depth immediates --trust-server-cert --non-interactive https://svn.apache.org/repos/bigdata/opennlp/trunk opennlp-corpus
+	cd opennlp-corpus
+	svn up --set-depth immediates --trust-server-cert --non-interactive
+	cd leipzig
+	svn up --set-depth immediates --trust-server-cert --non-interactive
 	cd resources/
-	wharariki:[197]/Scratch/ak19/openNLP-lang-detect/opennlp-corpus/leipzig/resources>svn up --set-depth infinity
+	wharariki:[197]/Scratch/ak19/openNLP-lang-detect/opennlp-corpus/leipzig/resources>svn up --set-depth infinity --trust-server-cert --non-interactive
 	cd ../data
-	svn up mri_web_2011_100K-sentences.txt
+	svn up --trust-server-cert --non-interactive mri_web_2011_100K-sentences.txt
+	
+	(# UNNECESSARY TO DOWNLOAD:
 	svn up eng_wikipedia_2012_3M-sentences.txt
 	svn up nld_mixed_2012_1M-sentences.txt
 	svn up fra_mixed_2009_1M-sentences.txt
+	)
 
 	cd ..
@@ -157,5 +181,5 @@
 
 
-4. Attempting running OpenNLP + LanguageDetectorModel's command line tools for language prediction
+4. Attempting to run OpenNLP + LanguageDetectorModel's command line tools for language prediction
 
 I couldn't get prediction to work from the command line, so I ended up writing that part as a Java class. But I could run some part of the command line tools with the instructions on the following pages. 
@@ -184,4 +208,14 @@
 
 
+UPDATE 24 July 2019: The following worked
+	>$OPENNLP_HOME/bin/opennlp LanguageDetector $OPENNLP_HOME/models/langdetect-183.bin < opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt 
+but it doesn't provide predictions. Not sure I understand what it did other than print the contents of the *sentences.txt file and end with:
+		...
+		Average: 0.1 doc/s 
+		Total: 1 doc
+		Runtime: 8.046s
+		Execution time: 8.719 seconds
+
+
 5. For writing Java code:
 To write the basic code, I followed the Java skeleton examples at
@@ -210,2 +244,16 @@
 
 (Though possibly the only jar file needed in $OPENNLP_HOME/lib is opennlp-tools-1.9.1.jar) 
+
+================
+
+UNORGANISED
+
+https://www.slideshare.net/shuyo/short-text-language-detection-with-infinitygram-12949447
+http://alexott.blogspot.com/2017/10/evaluating-fasttexts-models-for.html
+
+http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.sentdetect
+
+SentenceDetectorTrainer example
+https://www.tutorialkart.com/opennlp/train-model-sentence-detection-java/
+
+https://stackoverflow.com/questions/36516363/sentence-detection-with-opennlp
Index: gs3-extensions/maori-lang-detection/gen_SentenceDetection_model.sh
===================================================================
--- gs3-extensions/maori-lang-detection/gen_SentenceDetection_model.sh	(revision 33355)
+++ gs3-extensions/maori-lang-detection/gen_SentenceDetection_model.sh	(revision 33355)
@@ -0,0 +1,132 @@
+#!/bin/bash
+
+#####################################################################################################
+# Following instructions at
+# https://stackoverflow.com/questions/36516363/sentence-detection-with-opennlp
+# But contrary to their example, don't insert empty newlines separating the sentences input file
+# Each sentence just needs to be on its own line. But no empty lines between each sentence.
+#####################################################################################################
+
+# 1. Create training file of Maori sentences in the correct format
+# Train this on the 2011 set (rather than 2017) set of 100k Maori language sentences
+# because the 2011 one appears to have fewer accidentally incorporated English sentences
+
+
+if [ ! -z $1 ]; then
+    if [ "x$1" = "x--help" ]; then
+	echo ""
+	echo "Usage: $0 <mri sentences.txt>"
+	echo "   Try the one in ./opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt"
+	echo "   (Unzip mri-opennlp-corpus.tar.gz)"
+	echo "   It's better than the 100k sentences from 2017, as that contains a lot of English lines."
+	echo ""
+	exit
+    else
+	infile=$1
+    fi
+else
+    # use the file opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt as inputfile
+    # Check the file exists at the expected location. If not, check we have the tarball containing it and untar that.
+    # If the tarball doesn't exist, try to get what we want from svn if we have svn. If no svn either, bail.
+    if [ ! -f opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt ]; then
+	if [ -f mri-opennlp-corpus.tar.gz ]; then
+	    tar -xvzf mri-opennlp-corpus.tar.gz
+	else
+	    svn --help > /dev/null
+	    if [ "x$?" != "x0" ]; then
+		echo "Attempted to train the Sentence Detector on Maori sentences on missing mri_web_2011_100K-sentences.txt"
+		echo "The tarball mri-opennlp-corpus.tar.gz (containing mri_web_2011_100K-sentences.txt) didn't exist either."
+		echo "And couldn't get opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt using SVN: no SVN."
+		echo "Exitting..."
+		exit
+	    else
+		# Get just what we need from svn
+		# Based on https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt
+		svn co --depth immediates --trust-server-cert --non-interactive https://svn.apache.org/repos/bigdata/opennlp/trunk opennlp-corpus
+		cd opennlp-corpus
+		svn up --set-depth immediates --trust-server-cert --non-interactive
+		cd leipzig
+		svn up --set-depth immediates --trust-server-cert --non-interactive
+		cd resources/
+		svn up --set-depth infinity --trust-server-cert --non-interactive 
+		cd ../data
+		echo "About to download the sample Maori language 100k sentences file. This may take a while (<1 minute)."
+		svn up --trust-server-cert --non-interactive mri_web_2011_100K-sentences.txt
+		cd ..
+		# in opennlp-corpus/leipzig
+		chmod u+x create_langdetect_model.sh
+		cd ../..
+	    fi
+	fi	
+    fi
+    infile=./opennlp-corpus/leipzig/data/mri_web_2011_100K-sentences.txt
+fi
+
+echo $infile
+
+#tail -100 $infile
+
+
+mkdir -p $OPENNLP_HOME/training_data
+sentences_train_file=$OPENNLP_HOME/training_data/mri-sent.train
+if [ -f "$trainfile" ]; then
+    echo "Removing existing $sentences_train_file"
+    rm "$sentences_train_file"
+fi
+
+# Get just the sentences in the inputfile (remove the starting number followed by tab from each line)
+# see https://stackoverflow.com/questions/36516363/sentence-detection-with-opennlp
+# Don't add an extra newline at end of each sentence (don't insert an empty line between each sentence)
+
+
+# https://stackoverflow.com/questions/7619438/bash-read-a-file-line-by-line-and-process-each-segment-as-parameters-to-other-p
+# how come it removes the first number automatically?
+#while read file num sentence
+#do
+#    echo "$num $sentence" >> $sentences_train_file
+#done < "$infile"
+
+# Don't add the extra newline at the end, creating empty lines between sentences output,
+# or the SentenceDetector in step 3 will do a poor job.
+# (The example at the stackoverflow link may have included newlines separating sentences for clarity.)
+#cat $infile | awk -F '\t' '{ print $2 "\n" }' > $sentences_train_file
+cat $infile | awk -F '\t' '{ print $2 }' > $sentences_train_file
+
+# Why did my way below not work?
+#    openNLP-lang-detect/opennlp-corpus/leipzig/data>echo "100000\tYWCA Boarding house : ÄwhinaServices and support Kei te pÅ«manawa o TÄmaki Makaurau a YMCA." | awk -F "\t" '{ print $2 }'
+# $1 contains everything and $2 nothing. Why?
+# The problem appears to be that echo behaves differently from cat and less. Ask Dr Bainbridge what makes echo different.
+# Note that I tried manually inserting \t, after copying the original line with tabspacing had no effect. Still no difference.
+# Note 2: echo doesn't appear to preserve copied tab spaces.
+
+
+# 2. Create mri sentences model from training sentences file
+#$OPENNLP_HOME/bin/opennlp SentenceDetectorTrainer -model mri-sent_trained.bin -lang en -data mri-sent.train -encoding UTF-8
+
+if [ "x$OPENNLP_HOME" = "x" ]; then
+    echo "OPENNLP_HOME not set, attempting to set it to apache-opennlp-1.9.1 (ENSURE THIS EXISTS OR SET OPENNLP_HOME YOURSELF!)"
+    if [ -d apache-opennlp-* ]; then
+	cd apache-opennlp-*
+	export OPENNLP_HOME=`pwd`
+	cd ..
+    else
+	echo "No OPENNLP_HOME set and could not find a subfolder 'apache-opennlp-...' to set it to."
+	echo "Set OPENNLP_HOME yourself before running this script. Exitting..."
+    fi
+fi
+
+mkdir -p $OPENNLP_HOME/models
+
+$OPENNLP_HOME/bin/opennlp SentenceDetectorTrainer -model $OPENNLP_HOME/models/mri-sent_trained.bin -lang en -data $sentences_train_file -encoding UTF-8
+
+
+# 3. Let's try to split some Maori language text into sentences using our model created from training on sentences.
+# http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#tools.cli.sentdetect
+#    Usage: opennlp SentenceDetector model < sentences
+echo ""
+echo "Testing the model."
+echo "Running the SentenceDetector with the generated mri-sent_trained.bin model on sample_mri_paragraphs.txt..."
+echo "****************************"
+$OPENNLP_HOME/bin/opennlp SentenceDetector $OPENNLP_HOME/models/mri-sent_trained.bin < sample_mri_paragraphs.txt
+echo "****************************"
+echo ""
Index: gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/mri-sent.train
===================================================================
--- gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/mri-sent.train	(revision 33355)
+++ gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/mri-sent.train	(revision 33355)
@@ -0,0 +1,25 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Index: gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/sample_maori_shorttext.txt
===================================================================
--- gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/sample_maori_shorttext.txt	(revision 33355)
+++ gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/sample_maori_shorttext.txt	(revision 33355)
@@ -0,0 +1,5 @@
+
+Nau Mai Haere Mai
+
+Ko tÄnei te Whare WÄnanga o Waikato e whakatau nei i ngÄ iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngÄ maunga whakaruru e tau awhi nei.
+
Index: gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt
===================================================================
--- gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt	(revision 33355)
+++ gs3-extensions/maori-lang-detection/models-trainingdata-and-sampletxts/sample_mri_paragraphs.txt	(revision 33355)
@@ -0,0 +1,1 @@
+Kua whakaemitia mai ki konei he kohikohinga niupepa i tÄngia mÅ ngÄ kaipÄnui MÄori o ngÄ tau 1842-1932. E taea te pÄnui niupepa mÄ te rapu kupu, i te rÄrangi taitara, me te rÄrangi wÄtaka hoki. He mea i whakatÅ«ria e te kaupapa New Zealand Digital Library, i te Tari Rorohiko, o te Whare WÄnanga o Waikato. Kei raro nei he kupu whakamÄrama e pÄ ana ki te kohikohinga nei. Ka nui te mihi ki ngÄ rÅpÅ« tautoko i te kaupapa nei. NÄ tÄ rÄtou tautoko kua kore he utu, mÄ koutou, mÅ te titiro ki ngÄ niupepa nei. Ko te TÄhuhu o te MÄtauranga tÄrÄ, me Ätehi whare pukapuka e noho mai nei ki ngÄ whare wÄnanga o te motu. Tirohia ngÄ rÅpÅ« tautoko. 
Index: gs3-extensions/maori-lang-detection/src/MaoriTextDetector.java
===================================================================
--- gs3-extensions/maori-lang-detection/src/MaoriTextDetector.java	(revision 33350)
+++ gs3-extensions/maori-lang-detection/src/MaoriTextDetector.java	(revision 33355)
@@ -7,4 +7,5 @@
  * http://opennlp.apache.org/news/model-langdetect-183.html
  * language detector model: http://opennlp.apache.org/models.html
+ *        Pre-trained models for OpenNLP 1.5: http://opennlp.sourceforge.net/models-1.5/
  * Use of Apache OpenNLP in general:
  *   http://opennlp.apache.org/docs/1.9.1/manual/opennlp.html#intro.api
