Changeset 34131 for main

Timestamp:

2020-05-30T15:18:25+12:00 (4 years ago)

Author:

ak19

Message:

Allowing input keep-urls-file to contain a comma followed by country code at end, as that's the sort of URLs file I want for the newest commoncrawl collection. The URLs file is the one at http://trac.greenstone.org/browser/other-projects/maori-lang-detection/mongodb-data-auto/isMRI_full_manualList_globalDomains_whereAPageContainsMRI.txt

File:

• main/trunk/greenstone2/perllib/plugins/NutchTextDumpPlugin.pm (modified) (1 diff)

main/trunk/greenstone2/perllib/plugins/NutchTextDumpPlugin.pm

@@ -329 +329 @@
     my $fh;
     if (open($fh,'<:encoding(UTF-8)', $urls_file)) {
-    while (defined (my $line = <$fh>)) {            
+    while (defined (my $line = <$fh>)) {
         $line = &util::trim($line); #$line =~ s/^\s+|\s+$//g; # trim whitespace
-        if($line =~ m@^https?://@) { # add only URLs        
+
+        if($line =~ m@^https?://@) { # add only URLs
+        # remove any ",COUNTRYCODE" at end
+        # country code can be NZ but also UNKNOWN, so not 2 chars
+        $line =~ s/,[A-Z]+$//; 
+        #print STDERR "LINE: |$line|\n";
         $self->{'keep_urls'}->{$line} = 1; # add the url to our perl hash
         }