Changeset 33906 for other-projects/maori-lang-detection/src/org/greenstone/atea/MongoDBAccess.java

Timestamp:

2020-02-05T23:36:37+13:00 (4 years ago)

Author:

ak19

Message:

Code is intermediate state. 1. Introduced basicDomain field to MongoDB and recreated the MongoDB tables/collections, this will help discount duplicated domains under http and https, with and without www. Though webpage URLs may potentially still be unique and not duplicated across all 4 possible variants, I want them counted under the same base domain name. 2. Another issue noticed now is that some of the sites appear to be hosted on multiple countries servers, and so slightly different country code counts and domainlistings are returned. 3. So added code modifications (untested) to sort the domains alphabetically after stripping protocol and www to allow comparing the old domainListing results of MongoDB's now renamed oldWebsites and oldWebpages collections to the new versions of these collections and to then update the differences in manual counts.

File:

• other-projects/maori-lang-detection/src/org/greenstone/atea/MongoDBAccess.java (modified) (11 diffs)

other-projects/maori-lang-detection/src/org/greenstone/atea/MongoDBAccess.java

@@ -26 +26 @@
 import com.mongodb.Block;
 
+import org.bson.BsonArray;
+import org.bson.BsonString;
 import org.bson.Document;
 import org.bson.conversions.Bson;
@@ -217 +219 @@
         .append("siteFolderName", website.siteFolderName) 
         .append("domain", website.domain) 
+        .append("basicDomain", website.basicDomain) 
         .append("totalPages", website.totalPages)
         .append("numPagesWithBodyText", website.countOfWebPagesWithBodyText)
@@ -389 +392 @@
      *  and https://stackoverflow.com/questions/47979978/ambiguous-reference-to-foreach-when-listing-mongodbs-database-in-java
      *
-     *
+     * Count by country code of non-NZ websites containing a positive number of sentences in MRI,
+     * listing all the base domain strings (no protocol or www) in ALPHABETICAL ORDER
+     * and total counts of numPagesInMRI and numPagesContainingMRI across all these
+     * matching sites.
+     * 
      * The mongodb aggregate() we want to run this time:
      *
@@ -405 +412 @@
           $group: {
             _id: "nz",
-            count: { $sum: 1 },
-            domain: { $addToSet: '$domain' }
+            count: { $sum: 1 },            
+        domain: { $addToSet: '$basicDomain' } // domain: {$push: "$basicDomain" }
           }
     },
@@ -431 +438 @@
          match(andQuery),
          unwind("$geoLocationCountryCode"),
-         group("NZ", Arrays.asList(sum("count", 1), addToSet("domain", "$domain"))),
+         group("NZ", Arrays.asList(sum("count", 1),
+                   addToSet("domain", "$basicDomain"))),
          sort(BasicDBObject.parse("{count : -1}"))
      )).forEach((Block<Document>)doc -> writeDoc(doc, writer));
@@ -441 +449 @@
     
     /**
+     * Count of NZ (incl .nz TLD)  websites containing a positive number of sentences in MRI, 
+     * listing all the base domain strings (no protocol or www) in ALPHABETICAL ORDER
+     * and total counts of numPagesInMRI and numPagesContainingMRI across all these
+     * matching sites.
+     *
      * The aggregate() we want to run this time:
      *
@@ -459 +472 @@
             _id: {$toLower: '$geoLocationCountryCode'},
             count: { $sum: 1 },
-            domain: { $addToSet: '$domain' }
+        domain: { $addToSet: '$basicDomain' } // domain: {$push: "$basicDomain" }
           }
      },
@@ -484 +497 @@
         orQuery);
 
-
     collection.aggregate(Arrays.asList(
          match(andQuery),  //match(BasicDBObject.parse(matchQuery))
          // match((List<DBObject>)JSON.parse(matchQuery)),
          unwind("$geoLocationCountryCode"),
-         group("$geoLocationCountryCode", Arrays.asList(sum("count", 1), addToSet("domain", "$domain"))),
+         group("$geoLocationCountryCode", Arrays.asList(sum("count", 1), 
+                        addToSet("domain", "$basicDomain"))),
          sort(BasicDBObject.parse("{count : -1}"))
        )).forEach((Block<Document>)doc -> writeDoc(doc, writer));
@@ -639 +652 @@
      */    
     public void writeDoc(Document doc, Writer writer) throws UncheckedIOException {
+
+    // If there's a domain field in the json Doc, sort this domain listing alphabetically
+    Object domainList = doc.remove("domain");
+    if(domainList != null) {
+        doc.put("domain", sortAlphabetically(domainList));
+    }
+    
     //OLD WAY: writer.write(doc.toJson(new JsonWriterSettings(JsonMode.STRICT, true)) + NEWLINE);
     // Can't control json output to add newlines after each array element,
@@ -655 +675 @@
     // Have to use gson's pretty print to produce a json string that contains
     // newlines after every array element in the json:
+    
     String jsonStr = prettyPrintJson(doc.toJson());
     //System.err.println(jsonStr);
@@ -663 +684 @@
         throw new UncheckedIOException(ex);
     }       
+    }
+
+    private List sortAlphabetically(Object list) {
+    BsonArray domainList = (BsonArray)list;
+    //for(String domain : domainList) {
+    for(int i = domainList.size() - 1; i >= 0; i--) {
+        BsonString domain = domainList.get(i).asString();
+        String domainStr = Utility.stripProtocolAndWWWFromURL(domain.toString());
+        domainList.set(i, new BsonString(domainStr));       
+    }
+
+    return domainList;
     }