Changeset 34126

Timestamp:

2020-05-27T19:10:44+12:00 (4 years ago)

Author:

ak19

Message:

When I'd modified the code to make the keep_urls_file non-compulsory, introduced a mistake detected as a warning: I was wrongly dereferencing a ref to a perl hash. I found a long-winded way that works at present, and the code is presently running with it. Have also added a comment with the better way of dereferencing a hash, but that's untested at the moment.

File:

• main/trunk/greenstone2/perllib/plugins/NutchTextDumpPlugin.pm (modified) (2 diffs)

main/trunk/greenstone2/perllib/plugins/NutchTextDumpPlugin.pm

@@ -189 +189 @@
 
 
+# To get all the isMRI results, I ran Robo-3T against our mongodb as
+# in the instructions at http://trac.greenstone.org/browser/other-projects/maori-lang-detection/MoreReading/mongodb.txt
+# Then I launched Robo-3T and connected to the mongodb
+#
+# Then in the "ateacrawldata" database, I ran the following queries
+# to get a URL listing of all the Webpages where isMRI = true as determined
+# by apache openNLP.
+#
+#db.getCollection('Webpages').find({isMRI:true}).count();
+#7830
+#
+#db.getCollection('Webpages').find({isMRI:true},{URL: 1, _id: 0});
+#
+#Then I set robo-3T's output display to display 8000 results on a page, then copied the results into this file below.
+#
+# I cleaned out all the JSON from the results using regex in Notepad++.
+# This then becomes our urls.txt file, which I put into the cc nutch crawl
+# GS3 collection's etc folder under the name isMRI_urls.txt,
+# to consider processing only webpages apache Open-NLP detected as isMRI
+# into our collection.
+# Remember to configure the NutchTextDumpPlugin with option "keep_urls_file" = isMRI_urls.txt to make use of this.
+    
 sub BEGIN {
     @NutchTextDumpPlugin::ISA = ('SplitTextFile'); 
@@ -387 +409 @@
     }
     
-    # if keep_urls hash is empty, ensure it is undefined from this point onward
+    # If keep_urls hash is empty, ensure it is undefined from this point onward
+    # Use if(!keys %hash) to SECURELY test for an empty hash
     # https://stackoverflow.com/questions/9444915/how-to-check-if-a-hash-is-empty-in-perl
-    my %urls_map = $self->{'keep_urls'};
+    #
+    # But may not do: keys $hashref, only: keys %hash.
+    # Unable to work out how to dereference the hashref that is $self->{'keep_urls'},
+    # in order for me to then finally get the keys of the hashmap it refers to
+    # Googled: perl convert reference to hashmap
+    # The way to dereference hashref and get the keys is at https://www.thegeekstuff.com/2010/06/perl-hash-reference/
+    #      keys % { $hash_ref };
+    my $hashmap_ref = $self->{'keep_urls'};
+    my %urls_map = %$hashmap_ref;
     if(!keys %urls_map) {
     $self->{'keep_urls'} = undef;