Changeset 33623 for gs3-extensions/maori-lang-detection/src/org/greenstone/atea/TextDumpPage.java

Timestamp:

2019-11-05T21:04:09+13:00 (4 years ago)

Author:

ak19

Message:

1. Incorporated Dr Nichols earlier suggestion of storing page modified time and char-encoding metadata if present in the crawl dump output. Have done so, but neither modifiedTime nor fetchTime metadata of the dump file appear to be a webpage's actual modified time, as they're from 2019 and set around the period we've been crawling. 2. Moved getDomainFromURL() function from CCWETProcessor.java to Utility.java since it's been reused. 3. MongoDBAccess class successfully connects (at least, no exceptions) and uses the newly added properties in config.properties to make the connection.

File:

• gs3-extensions/maori-lang-detection/src/org/greenstone/atea/TextDumpPage.java (modified) (2 diffs)

gs3-extensions/maori-lang-detection/src/org/greenstone/atea/TextDumpPage.java

@@ -84 +84 @@
                 String k = line.substring(0, endIndex);
                 String v = line.substring(endIndex+1);
+                if(k.startsWith("metadata")) {
+                k = k.substring("metadata".length());
+                }
+                
                 tuples.put(k.trim(), v.trim());
             } else {
@@ -134 +138 @@
     }
 
+    /* Dr Nichols suggested storing timestamp and char encoding. Not sure which timestamp
+       or encoding he meant, but storing 2 of several timestamps and selecting
+       original character encoding (presumably the char encoding of the page) out of 2
+       pieces of char encoding metadata to store. */
+    public String getModifiedTime() {   
+    // is this the webpage's last mod time? 
+    String time = tuples.get("modifiedTime");
+    time = time.equals("0") ? "" : time; // zero will be assumed to be epoch, rather than unset
+    return time;
+    }    
+    public String getFetchTime() {
+    // is this the nutch crawl time
+    String time = tuples.get("fetchTime");
+    time = time.equals("0") ? "" : time; // zero will be assumed to be epoch, rather than unset
+    return time;
+    
+    } 
+    public String getOriginalCharEncoding() {
+    // is this the web page's char-encoding?
+    return tuples.get("OriginalCharEncoding");
+    }
+    
     public String get(String key) {
     return tuples.get(key);