Changeset 33800 for other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java

Timestamp:

2019-12-12T18:04:10+13:00 (4 years ago)

Author:

ak19

Message:

Removed an adult site from crawled contents and added its url to blacklist conf file (for if ever anyone crawls our MRI set of common crawl sites again)

File:

• other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java (modified) (2 diffs)

other-projects/maori-lang-detection/src/org/greenstone/atea/NutchTextDumpToMongoDB.java

@@ -291 +291 @@
     File geoLiteCityDatFile = new File(this.getClass().getClassLoader().getResource("GeoLiteCity.dat").getFile());
     try {
-        if(this.domainOfSite.equals("UNKNOWN")) {
+        if(this.domainOfSite.equals("UNKNOWN")) { // for sites that had 0 webpages downloaded, we have no domain
         this.geoLocationCountryCode = "UNKNOWN";
         } else {
@@ -298 +298 @@
     } catch(Exception e) {      
         logger.error("*** For SiteID " + siteID + ", got exception: "  + e.getMessage(), e);
-        this.geoLocationCountryCode = null;
+        this.geoLocationCountryCode = "UNKNOWN"; // couldn't get the country code, so should also be UNKNOWN not null
     }