Changeset 32643 for main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm

Timestamp:

2018-11-30T22:02:15+13:00 (5 years ago)

Author:

ak19

Message:

1. Previous commit (r32640) reintroduced an earlier bug in attempting to fix another. It reintroduced the bug whereby a reindex operation upon a doc rename (implemented as delete and and) would result in the deleted doc turning up on browsing. Despite being deleted from the SQL db, a ref to its oid remained in the index. This bug had been fixed by a call to doc_obj.set_OID() which presumably helped to identify the OID of any doc marked for deletion after which the indexing part of the process would proceed to delete it. The recent commit had tried to prevent the assignment of 2 OIDs for renamed documents (the deleted oid and the new oid) by selectively calling set_OID(). But this reintroduced the older bug. The solution was to call set_OID at the end, AFTER reading into the doc_obj from the SQL db, which both prevents 2 OIDs for a renamed doc and properly gets the doc deleted. 2. A further modification is reintroducing an improvement that existed in earlier uncommitted attempts of the GS SQL Plugin. When a doc is marked for deletion, its oid didn't exist in the MySQL db after import.pl, yet during buildcol.pl the code still attempted to read all the records for any and all docoids, including those marked for deletion, back in from the MySQL db. Now the code just does doc_obj.set_OID() if an oid is for a doc marked for deletion (as required to get the doc actually deleted from the index) and then calls the super class method to let the indexing part process the doc to get it deleted and then returns, skipping attempting to read in info on that oid from the SQL db when nothing exists for it. For any non-deleted oid, the code of course continues to read in the entries from the MySQL db for that oid to reconstruct the doc_object.

File:

• main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm (modified) (3 diffs)

main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm

@@ -373 +373 @@
     
     print STDERR "   GreenstoneSQLPlugin processing doc $oid (reading into docobj from SQL db)\n"
-    if $self->{'verbosity'};
+        if $self->{'verbosity'};
+    
+    my $build_proc_mode = $self->{'processor'}->get_mode(); # can be "text" as per basebuildproc or infodb
+    if($build_proc_mode =~ m/(delete)$/) {
+    # build_proc_mode could be "(infodb|text)(delete|reindex)"
+    # "...delete" or "...reindex" as per ArchivesInfPlugin
+    # But reindex is implemented as delete for GreenstoneSQLPlugs, so that's all we see here?
+    print STDERR "   DOC $oid WAS MARKED FOR DELETION. Won't attempt to retrieve from SQL db.\n" if $self->{'verbosity'};
+    $self->{'doc_obj'}->set_OID($oid); # oid is all we care about for a doc marked for deletion
+    $self->SUPER::close_document(@_); # at the end of this method, doc will have been deleted
+    return;  # oid of doc marked for deletion is not in the SQL db, don't bother looking it up
+    }
+
+    # else, doc denoted by oid was not marked for deletion, look up its oid in db and read it into doc obj
     
     if($proc_mode eq "all" || $proc_mode eq "meta_only") {
@@ -384 +397 @@
     foreach my $row (@$records) {
         my ($primary_key, $did, $sid, $metaname, $metaval) = @$row;
-        
-        # don't allow duplicate toplevel OID, as can happen when a doc has been renamed and requires reindexing
-        # TODO: any other meta that should not be duplicated, but can have been changed between rebuilds so that we need to call set_meta instead of add_meta?
-        # e.g. FileSize, SourceFile. But Language can be multiple, so gs meta is not guaranteed to be unique either. Whereas of dc metadata we know
-        # that some if not all should allow multiple entires for the same meta name, e.g. dc.Author/dc.Creator
-        if($sid =~ m@^root@ && $metaname eq "Identifier") {
-            # doc_obj's toplevel Identifier metadata is a special case:
-            # it should have only one value, so instead of add_meta() that will allow old Identifier meta to linger
-            # Need to do set_meta(). We then break out of the loop, to prevent duplicates (older values from DB) to be inserted for Identifier into doc_obj
-            # Handles the case where a doc was renamed and rebuilding triggers re-indexing case: old identifier is now overwritten with new one
-            $self->{'doc_obj'}->set_OID($oid); # calls doc_obj->set_metadata_element(top_section, Identifier, $oid). Sets OID if one doesn't exist.
-            next; # ensures Identifier set only once, and ensure Identifier is set to current docOID for the doc, a.o.t. allowing it to be set to any expired docOID from before a doc got renamed.         
-        }
-        
-        # process all other metadata the normal way:
         
         # get rid of the artificial "root" introduced in section id when saving to sql db
@@ -450 +448 @@
     }
     
+
+    # setting OID here instead of before reading from SQL db into docobj, will prevent duplicate values for Identifier
+    # since doc::set_OID() calls doc::set_metadata_element() for metadata that can't occur more than once
+    $self->{'doc_obj'}->set_OID($oid); # may only be necessary if doc was marked for deletion so that SUPER::close_document knows
+                                       # the oid of marked doc to remove from index
+    
     # done reading into docobj from SQL db