Changeset 32640 for main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm

Timestamp:

2018-11-29T21:23:49+13:00 (5 years ago)

Author:

ak19

Message:

Important changes (and commented out debugging statements) to get charset encodings (utf8 or specifically utf8mb4 in perl mysql case) to work with GS SQL Plugs on Windows. First, Strawberry Perl 5.22 was required and hence committed before this since Strawberry Perl 5.18 came with older mysql DBD/DBI packages that didn't decode utf8 content in the database when content was retrieved. Strawberry Perl 5.22 came with newer versions of DBD and DBI that do this automatically, as has been the case in Linux testing where Ubuntu had Perl 5.22 with sufficiently new versions of the DBI/DBD mysql perl packages. The newer Perl and specifically the newer MySQL DBD/DBI packages required some important changes to the gsmysql.pm code in the way charset encoding stuff is configured. This should work on Linux too, as that already allowed 2 ways to configure DB encoding stuff. I chose the single-line version on Linux, no longer supported with the DBI/DBD upgrade that comes with our new Strawberry Perl 5.22, so opting for the 2 line version to setup the DB encoding stuff which works on Windows and should continue to work on Linux too (where I had tested it before settling on the single-line variant).

File:

• main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm (modified) (5 diffs)

main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm

@@ -37 +37 @@
 use gsmysql;
 
-
+#use unicode;
+#use Encode;
 
 ########################################################################################
@@ -370 +371 @@
     # browse the rebuilt collection by files/titles. So unless you set the OID here, the deleted
     # doc oids will still be listed in the index.
-    $self->{'doc_obj'}->set_OID($oid);
     
     print STDERR "   GreenstoneSQLPlugin processing doc $oid (reading into docobj from SQL db)\n"
@@ -385 +385 @@
         my ($primary_key, $did, $sid, $metaname, $metaval) = @$row;
         
+        # don't allow duplicate toplevel OID, as can happen when a doc has been renamed and requires reindexing
+        # TODO: any other meta that should not be duplicated, but can have been changed between rebuilds so that we need to call set_meta instead of add_meta?
+        # e.g. FileSize, SourceFile. But Language can be multiple, so gs meta is not guaranteed to be unique either. Whereas of dc metadata we know
+        # that some if not all should allow multiple entires for the same meta name, e.g. dc.Author/dc.Creator
+        if($sid =~ m@^root@ && $metaname eq "Identifier") {
+            # doc_obj's toplevel Identifier metadata is a special case:
+            # it should have only one value, so instead of add_meta() that will allow old Identifier meta to linger
+            # Need to do set_meta(). We then break out of the loop, to prevent duplicates (older values from DB) to be inserted for Identifier into doc_obj
+            # Handles the case where a doc was renamed and rebuilding triggers re-indexing case: old identifier is now overwritten with new one
+            $self->{'doc_obj'}->set_OID($oid); # calls doc_obj->set_metadata_element(top_section, Identifier, $oid). Sets OID if one doesn't exist.
+            next; # ensures Identifier set only once, and ensure Identifier is set to current docOID for the doc, a.o.t. allowing it to be set to any expired docOID from before a doc got renamed.         
+        }
+        
+        # process all other metadata the normal way:
+        
         # get rid of the artificial "root" introduced in section id when saving to sql db
         $sid =~ s@^root@@;
@@ -395 +410 @@
         $doc_obj->add_utf8_metadata($sid, $metaname, $metaval);
     }
+    
+    # UTF8 debugging, e.g. if we have a with macron in dc.Title
+    ##binmode(STDERR, ":utf8"); # not necessary when calling utf8decomp to print wide characters as unicode codept, avoiding 'wide character' warnings.
+    #my $dcTitle = $doc_obj->get_metadata_element($doc_obj->get_top_section(), "dc.Title"); 
+    #print STDERR "###### dc.Title: ".&unicode::utf8decomp($dcTitle)."\n";
+    #print STDERR "###### DECODED dc.Title: ".&unicode::utf8decomp(&Encode::decode_utf8($dcTitle))."\n"; # decoding was needed for perl 5.18
+    
     print $outhandle "----------FIN READING DOC's META FROM SQL DB------------\n"
         if $self->{'verbosity'} > 2;
     }
-    
+    
     if($proc_mode eq "all" || $proc_mode eq "text_only") {
     # read in fulltxt for the collection (i.e. select * from <col>_fulltxt table
@@ -429 +451 @@
     
     # done reading into docobj from SQL db
-    
+    
     # don't forget to clean up on close() in superclass
     # It will get the doc_obj indexed then make it undef