Changeset 33988 for other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java

Timestamp:

2020-02-28T22:09:15+13:00 (4 years ago)

Author:

ak19

Message:

1. Print out which web pages of which web site's dump.txt were empty. Then can run NutchTextDumpToMongoDB > outfile.txt 2>&1. 2. More instructions for before running NutchTextDumpToMongoDB

File:

• other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java (modified) (2 diffs)

other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java

@@ -40 +40 @@
  *    cd maori-lang-detection/src
  *
+ * MORE IMPORTANT PRELIMINARIES:
+ * - Make sure the MongoDB is up and running and accessible.
+ * - If you want to keep any existing MongoDB collections called Websites and Webpages, then
+ * first renamed those collections in MongoDB (using Robo3T makes renaming easy) before
+ * running this program.
+ *
  * TO COMPILE:
  *    maori-lang-detection/src$
@@ -245 +251 @@
         
         if(text.equals("")) {
+        System.err.println("siteID: " + siteID + "- Empty page " + i + " - URL: "
+                   + page.getPageURL());
+        
         // don't care about empty pages
         continue;