Changeset 33623 for gs3-extensions/maori-lang-detection/src/org/greenstone/atea/Utility.java

Timestamp:

2019-11-05T21:04:09+13:00 (4 years ago)

Author:

ak19

Message:

1. Incorporated Dr Nichols earlier suggestion of storing page modified time and char-encoding metadata if present in the crawl dump output. Have done so, but neither modifiedTime nor fetchTime metadata of the dump file appear to be a webpage's actual modified time, as they're from 2019 and set around the period we've been crawling. 2. Moved getDomainFromURL() function from CCWETProcessor.java to Utility.java since it's been reused. 3. MongoDBAccess class successfully connects (at least, no exceptions) and uses the newly added properties in config.properties to make the connection.

File:

• gs3-extensions/maori-lang-detection/src/org/greenstone/atea/Utility.java (modified) (1 diff)

gs3-extensions/maori-lang-detection/src/org/greenstone/atea/Utility.java

@@ -82 +82 @@
     }
     
+    /** Work out the 'domain' for a given url.
+     * This retains any www. or subdomain prefix.
+     */
+    public static String getDomainForURL(String url, boolean withProtocol) {
+    int startIndex = startIndex = url.indexOf("//"); // for http:// or https:// prefix
+    startIndex = (startIndex == -1) ? 0 : (startIndex+2); // skip past the protocol's // portion
+    // the keep the URL around in case param withProtocol=true
+    String protocol = (startIndex == -1) ? "" : url.substring(0, startIndex);
+    
+    String domain = url.substring(startIndex);
+    int endIndex = domain.indexOf("/");
+    if(endIndex == -1) endIndex = domain.length();
+    domain = domain.substring(0, endIndex);
+
+    if(withProtocol) {
+        // now that we have the domain (everything to the first / when there is no protocol)
+        // can glue the protocol back on
+        domain = protocol + domain;
+    }
+    
+    return domain;
+    }
+    
     public static boolean isDomainInCountry(String domainWithProtocol,
                         String countryCode, File geoLiteCityDatFile)