Changeset 34121 for main

Timestamp:

2020-05-25T23:53:29+12:00 (4 years ago)

Author:

ak19

Message:

1. Introducing NutchTextDumpPlugin to process the records (representing web pages' text content) of the dump.txt files produced for each website crawled by Nutch. Created for handling the commoncrawl URLs of interest that we recrawled with Nutch. This first version does everything, but the code requires more cleaning up. 2. Also added a useful util::trim() function as I kept reusing the same code several times.

Location:

main/trunk/greenstone2/perllib

Files:

• plugins/NutchTextDumpPlugin.pm (added)
• strings.properties (modified) (1 diff)
• util.pm (modified) (1 diff)

main/trunk/greenstone2/perllib/strings.properties

@@ -1141 +1141 @@
 MetadataXMLPlugin.desc:Plugin that processes metadata.xml files.
 
+NutchTextDumpMARCXMLPlugin.keep_urls_file:File path or name of optional whitelist file containing one URL per line, whose records are to be retained when processing each url's record in the dump.txt files produced by nutch per website. Those records whose URLs are not listed in the file will be discarded. For relative paths, the plugin will look for the file in the collection's etc directory.
+
 GreenstoneMETSPlugin.desc:Process Greenstone-style METS documents
 

main/trunk/greenstone2/perllib/util.pm

@@ -1959 +1959 @@
 
 
+# Useful String utility functions
+# trim(str) removes whitespace at start and end of string parameter
+sub trim {
+    my ($str) = @_;
+    
+    # trim whitespace at start and end of str
+    # https://perlmaven.com/trim
+    $str =~ s/^\s+|\s+$//g;
+
+    return $str;
+}
+
 1;