Changeset 33906 for other-projects/maori-lang-detection/src/org/greenstone/atea/WebPageURLsListing.java

Timestamp:

2020-02-05T23:36:37+13:00 (4 years ago)

Author:

ak19

Message:

Code is intermediate state. 1. Introduced basicDomain field to MongoDB and recreated the MongoDB tables/collections, this will help discount duplicated domains under http and https, with and without www. Though webpage URLs may potentially still be unique and not duplicated across all 4 possible variants, I want them counted under the same base domain name. 2. Another issue noticed now is that some of the sites appear to be hosted on multiple countries servers, and so slightly different country code counts and domainlistings are returned. 3. So added code modifications (untested) to sort the domains alphabetically after stripping protocol and www to allow comparing the old domainListing results of MongoDB's now renamed oldWebsites and oldWebpages collections to the new versions of these collections and to then update the differences in manual counts.

File:

• other-projects/maori-lang-detection/src/org/greenstone/atea/WebPageURLsListing.java (modified) (5 diffs)

other-projects/maori-lang-detection/src/org/greenstone/atea/WebPageURLsListing.java

@@ -57 +57 @@
     
     public void produceURLsForPagesInMRI(File domainsFile) {
-    ArrayList<Tuple> urlsList = getURLsForWebPages(MongoDBAccess.IS_MRI, domainsFile);
+    ArrayList<Tuple> urlsList = getURLsForAllWebPagesInSiteListing(MongoDBAccess.IS_MRI, domainsFile);
     File outFile = new File(outFolder, "isMRI_"+domainsFile.getName());
     writeURLsToFile(urlsList, outFile, urlsList.size());
@@ -66 +66 @@
     
     public void produceURLsForPagesContainingMRI(File domainsFile) {
-    ArrayList<Tuple> urlsList = getURLsForWebPages(MongoDBAccess.CONTAINS_MRI, domainsFile);    
+    ArrayList<Tuple> urlsList = getURLsForAllWebPagesInSiteListing(MongoDBAccess.CONTAINS_MRI, domainsFile);    
     File outFile = new File(outFolder, "containsMRI_"+domainsFile.getName());
     writeURLsToFile(urlsList, outFile, urlsList.size());
@@ -74 +74 @@
     }
     
-    private ArrayList<Tuple> getURLsForWebPages(int filterType, File domainsFile) {
+    private ArrayList<Tuple> getURLsForAllWebPagesInSiteListing(int filterType, File domainsFile) {
     ArrayList<Tuple> urlsList = new ArrayList<Tuple>();
     
@@ -120 +120 @@
     }
     
-    /** Given a hand curated list of NZ sites with positive numPagesContainingMRI,
-     * get a listing of all their web pages IN_MRI (or CONTAINS_MRI?). 
-     * Total all these pages in MRI (N), then work out the correct sample size (n)
+    /** Given a hand curated list of all sites with positive numPagesContainingMRI
+     * determined by manual inspection, get a listing of all their web pages that
+     * are IN_MRI (or CONTAINS_MRI?). 
+     * Total all these pages that are inMRI (N), then work out the correct sample size (n)
      * at 90% confidence with 5% margin of error. Then generate a random listing
      * of n of these pages in MRI of these trusted sites and output to a file
-     * for manual inspection. */
+     * for manual inspection of the sample webpage URLs at page-level. */
     /* OLD: Given a hand curated list of non-NZ sites that CONTAINS_MRI, get a listing
      * of all their web pages IN_MRI (or CONTAINS_MRI).
@@ -138 +139 @@
 
     // 0. get a list of all the web pages in the given domain listing where isMRI = true
-    ArrayList<Tuple> urlsList = getURLsForWebPages(MongoDBAccess.IS_MRI, domainsFile);
+    ArrayList<Tuple> urlsList = getURLsForAllWebPagesInSiteListing(MongoDBAccess.IS_MRI, domainsFile);
         // produceURLsForPagesInMRI(domainsFile);