Changeset 33634 for gs3-extensions/maori-lang-detection/src/org/greenstone/atea/TextLanguageDetector.java

Timestamp:

2019-11-08T23:59:07+13:00 (4 years ago)

Author:

ak19

Message:

Rewrote NutchTextDumpProcessor as NutchTextDumpToMongoDB.java, which uses MongoDBAccess that now has insertWebpageInfo() and insertWebsiteInfo(). However, testing has been unsuccessful locally, despite the fact that authentication should be working, as I'm following the examples online to use the Credential object. It supposedly connects to the database, but database.listCollections() fails with an Unauthorized error. Nothing subsequent can be expected to work. I could do my preliminary testing against a small sample subset of crawled sites on vagrant where there is no authentication setup, but what if someone else wants to run this one day against a mongodb where they authentication is set up (the way TSG set it up for the mongodb they gave me access to). Then it still wouldn't work.

File:

• gs3-extensions/maori-lang-detection/src/org/greenstone/atea/TextLanguageDetector.java (modified) (4 diffs)

gs3-extensions/maori-lang-detection/src/org/greenstone/atea/TextLanguageDetector.java

@@ -142 +142 @@
     }
 
-    /** inner class */
-    public class SentenceInfo {
-    public final double confidenceLevel;
-    /** 3 letter lang code */
-    public final String langCode;
-    public final String sentence;
-
-    public SentenceInfo(double confidence, String langCode, String sentence) {
-        confidenceLevel = confidence;
-        this.langCode = langCode;
-        this.sentence = sentence;
-    }
-    }
-
     /** TODO: Is it sensible to use the Maori Language Sentence Model to split the text
      * into sentences? What if the text in any other language or a mix of languages?
@@ -183 +169 @@
         double confidence = bestLanguage.getConfidence();
 
-        sentencesList.add(new SentenceInfo(confidence, bestLanguage, sentence));
+        sentencesList.add(new SentenceInfo(confidence, bestLanguage.getLang(), sentence));
     }
 
@@ -207 +193 @@
         separator = " ";
         }
-        sentence = sentence + separator + sentence[i]; 
+        sentence = sentence + separator + sentences[i]; 
         
         //System.err.println(sentence);
@@ -214 +200 @@
         double confidence = bestLanguage.getConfidence();
 
-        sentencesList.add(new SentenceInfo(confidence, bestLanguage, sentence));
+        sentencesList.add(new SentenceInfo(confidence, bestLanguage.getLang(), sentence));
     }