Changeset 33634 for gs3-extensions/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToCSV.java

Timestamp:

2019-11-08T23:59:07+13:00 (4 years ago)

Author:

ak19

Message:

Rewrote NutchTextDumpProcessor as NutchTextDumpToMongoDB.java, which uses MongoDBAccess that now has insertWebpageInfo() and insertWebsiteInfo(). However, testing has been unsuccessful locally, despite the fact that authentication should be working, as I'm following the examples online to use the Credential object. It supposedly connects to the database, but database.listCollections() fails with an Unauthorized error. Nothing subsequent can be expected to work. I could do my preliminary testing against a small sample subset of crawled sites on vagrant where there is no authentication setup, but what if someone else wants to run this one day against a mongodb where they authentication is set up (the way TSG set it up for the mongodb they gave me access to). Then it still wouldn't work.

File:

• gs3-extensions/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToCSV.java (modified) (2 diffs)

gs3-extensions/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToCSV.java

@@ -244 +244 @@
         
         if(text.equals("")) {
-        page.addMRILanguageStatus(false);
+        //page.addMRILanguageStatus(false);
         continue;
         }
@@ -250 +250 @@
         boolean isMRI = maoriTxtDetector.isTextInMaori(text);
         
-        page.addMRILanguageStatus(isMRI);
+        //page.addMRILanguageStatus(isMRI);