Changeset 33906 for other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java

Timestamp:

2020-02-05T23:36:37+13:00 (4 years ago)

Author:

ak19

Message:

Code is intermediate state. 1. Introduced basicDomain field to MongoDB and recreated the MongoDB tables/collections, this will help discount duplicated domains under http and https, with and without www. Though webpage URLs may potentially still be unique and not duplicated across all 4 possible variants, I want them counted under the same base domain name. 2. Another issue noticed now is that some of the sites appear to be hosted on multiple countries servers, and so slightly different country code counts and domainlistings are returned. 3. So added code modifications (untested) to sort the domains alphabetically after stripping protocol and www to allow comparing the old domainListing results of MongoDB's now renamed oldWebsites and oldWebpages collections to the new versions of these collections and to then update the differences in manual counts.

File:

• other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java (modified) (3 diffs)

other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java

@@ -75 +75 @@
     private String geoLocationCountryCode = null; /** 2 letter country code */
     private boolean urlContainsLangCodeInPath = false; /** If any URL on this site contains a /mi(/) or http(s)://mi.* in its URL path */
-    
+
     private String domainOfSite;
+    private String baseSiteDomain; // domainOfSite stripped of any http(s)://www.
     private int numPagesInMRI = 0;
     private int numPagesContainingMRI = 0;
@@ -202 +203 @@
         String url = firstPage.getPageURL();
         this.domainOfSite = Utility.getDomainForURL(url, true);
+        this.baseSiteDomain = Utility.stripProtocolAndWWWFromURL(this.domainOfSite);
     }
     else {
         this.domainOfSite = "UNKNOWN";
+        this.baseSiteDomain = "UNKNOWN";
     }
     
@@ -339 +342 @@
     int totalPages = pages.size();  
 
-    WebsiteInfo website = new WebsiteInfo(/*SITE_COUNTER,*/ this.siteID, this.domainOfSite,
+    WebsiteInfo website = new WebsiteInfo(/*SITE_COUNTER,*/ this.siteID,
+          this.domainOfSite, this.baseSiteDomain,
           totalPages, this.countOfWebPagesWithBodyText,
           this.numPagesInMRI, this.numPagesContainingMRI,