Changeset 33350

Timestamp:

2019-07-23T17:29:18+12:00 (5 years ago)

Author:

ak19

Message:

Better comments. Tested macronised vs unmacronised Māori language test string and both are detected as mri, but the unmacronised is detected with lower confidence. Added a note on that in the README.

Location:

gs3-extensions/maori-lang-detection

Files:

• README.txt (modified) (1 diff)
• src/MaoriTextDetector.class (modified) ( previous)
• src/MaoriTextDetector.java (modified) (8 diffs)

gs3-extensions/maori-lang-detection/README.txt

@@ -38 +38 @@
 
 
-
-
-For reading materials, see the OLD README section below.
+For links to background reading materials, see the OLD README section further below.
+
+
+NOTE: The OpenNLP Language Detection Model can detect non-macronised Māori text too,
+but as anticipated, the same text produces a lower confidence level for the language prediction. Compare:
+
+$maori-lang-detection/src>java -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" MaoriTextDetector -
+   Waiting to read text from STDIN... (press Ctrl-D when done entering text)>
+   Ko tenei te Whare Wananga o Waikato e whakatau nei i nga iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o nga maunga whakaruru e tau awhi nei.
+   Best language: mri
+   Best language confidence: 0.5959533972070814
+   Exitting program with returnVal 0...
+
+$maori-lang-detection/src>java -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" MaoriTextDetector -
+   Waiting to read text from STDIN... (press Ctrl-D when done entering text)>
+   Ko tēnei te Whare Wānanga o Waikato e whakatau nei i ngā iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngā maunga whakaruru e tau awhi nei.
+   Best language: mri
+   Best language confidence: 0.6825737450092515
+   Exitting program with returnVal 0...
+
 
 -------------------------

gs3-extensions/maori-lang-detection/src/MaoriTextDetector.java

@@ -1 +1 @@
 /**
+ * Class that uses OpenNLP with the Language Detection Model to determine, with a default
+ * or configurable level of confidence, whether text (from a file or stdin) is in Māori or not.
+ * Internal functions can be used for detecting any of the 103 languages currently supported by
+ * the OpenNLP Language Detection Model.
+ * 
  * http://opennlp.apache.org/news/model-langdetect-183.html
  * language detector model: http://opennlp.apache.org/models.html
@@ -8 +13 @@
  * 
  * This code was based on the information and sample code at the above links and the links dispersed throughout this file.
+ * See also the accompanying README file.
+ *
+ * July 2019
  */
 
@@ -16 +24 @@
 /**
  * EXPORT OPENNLP_HOME environment variable to be your apache OpenNLP installation.
- * Then, to compile this program:
+ * Create a folder called "models" within the $OPENNLP_HOME folder, and put the file "langdetect-183.bin" in there
+ *    (which is the language detection model zipped up and renamed to .bin extension).
+ *
+ * Then, to compile this program, do the following from the "src" folder (the folder containing this java file):
  *    maori-lang-detection/src$ javac -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" MaoriTextDetector.java
- * To run this program, one of:
+ *
+ * To run this program, issue one of the following commands from the "src" folder (the folder containing this java file):
  *
  *    maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" MaoriTextDetector --help
@@ -25 +37 @@
  *
  *    maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" MaoriTextDetector -
- *       which expects text to stream in from standard input.
+ *       Press enter. This variant of the program expects text to stream in from standard input.
  *       If entering text manually, then remember to press Ctrl-D to indicate the usual end of StdIn.
  *
@@ -39 +51 @@
     greater or equal to which determines that the best predicted language is acceptable to user of MaoriTextDetector. */
     public final double MINIMUM_CONFIDENCE;
+    
     /** silentMode set to false means MaoriTextDetector won't print helpful messages while running. Set to true to run silently. */
     public final boolean silentMode;
@@ -44 +57 @@
     /** Language Detection Model file for OpenNLP is expected to be at $OPENNLP_HOME/models/langdetect-183.bin */
     private final String LANG_DETECT_MODEL_RELATIVE_PATH = "models" + File.separator + "langdetect-183.bin";
+
+    /**
+     * The LanguageDetectorModel object that will do the actual language detection/prediction for us.
+     * Created once in the constructor, can be used as often as needed thereafter.
+    */
     private LanguageDetector myCategorizer = null;
     
-    /** 
-     * String taken from our university website
-     * https://www.waikato.ac.nz/maori/
-     */
+    /** String taken from our university website, https://www.waikato.ac.nz/maori/ */
     public static final String TEST_MRI_INPUT_TEXT = "Ko tēnei te Whare Wānanga o Waikato e whakatau nei i ngā iwi o te ao, ki roto i te riu o te awa e rere nei, ki runga i te whenua e hora nei, ki raro i te taumaru o ngā maunga whakaruru e tau awhi nei.";
 
@@ -224 +239 @@
     /**
      * Prints to STDOUT the predicted languages of the input text in order of descending confidence.
-     * Unused.
+     * UNUSED.
      */
     public void predictedLanguages(String text) {
@@ -367 +382 @@
         System.exit(returnVal);
     }   
-    
+
+
+    // 2. Finally, we can now do the actual language detection
     try {
         MaoriTextDetector maoriTextDetector = null;