Changeset 33392

Timestamp:

2019-08-08T15:15:11+12:00 (5 years ago)

Author:

ak19

Message:

Kathy found a problem whereby she wanted to run consecutive buildcols without activate on a solr collection. She experienced file locking issues on Windows, which the original solr related building code would inevitably cause without activate. Dr Bainbridge's solution was to change our way of thinking about what activate and buildcol should now be doing as regards solr collections. The solution was to unload building-cores for indexes at the end of buildcol, instead of only doing this during activate.pl. I've tried to be conservative with the changes made to the existing code, so that activate still attempts to also unload building-cores, but first pings them (and any other cores it attempts to unload) to ensure the cores exist. During buildcol too, the building-cores are pinged to check they exist before we attempt to unload them.

Files:

• gs3-extensions/solr/trunk/src/perllib/solrbuilder.pm (modified) (4 diffs)
• gs3-extensions/solr/trunk/src/perllib/solrserver.pm (modified) (1 diff)
• main/trunk/greenstone2/bin/script/activate.pl (modified) (4 diffs)

gs3-extensions/solr/trunk/src/perllib/solrbuilder.pm

@@ -532 +532 @@
     }
     else {
-        # if collect==core already in solr.xml (check with STATUS)
-        # => use RELOAD call to refresh fields now expressed in schema.xml
-        #
+        # if collect==core is already in solr.xml (check with STATUS)
+        # => use RELOAD* call to refresh fields now expressed in schema.xml
+        # 
         # else 
         # => use CREATE API to add to solr.xml
+        #
+        # No longer calling RELOAD, because Georgy documented a memory leak with it (svn r32178)
+        # Using unload + create to get the same effect as RELOAD without its side-effects.
+        # 
         
         my $check_core_exists = $solr_server->admin_ping_core($core);
@@ -545 +549 @@
         }
         
-        print $outhandle "Creating Solr core: $core\n";
-        $solr_server->admin_create_core($core);
+        print $outhandle "Creating Solr core: $core\n";
+        $solr_server->admin_create_core($core);
         
     }
@@ -690 +694 @@
     print STDERR "</Stage>\n" if $self->{'gli'};
 
+
+    print STDERR "@@@@@ FINISHED PROCESSING INDEX: indexlevel $self->{'index_mapping'}->{$index}\n\n";
+
 }
 
@@ -700 +707 @@
     # as this has been done in our pre_build_indexes() phase for solr
 
-
+      
+    my $solr_server = $self->{'solr_server'};
+
+    # 1 Aug 2019: now we unload (remove) building-cores for each index during buildcol itself
+    # instead of during activate.
+    # Kathy described a problem that when calling buildcol.pl successively without succeeding each
+    # with a call to activate, there were Win file lock issues when attempting to manually remove
+    # the building folder. This was what activate was to solve, however, there's no reason to
+    # call activate after buildcol in such cases where it is known the buildcol failed in some way.
+    # (In such cases, the user building the collection would have to manually unload the building-
+    # cores through the solr servlet interface).
+    # Dr Bainbridge instructed that the building- cores should be unloaded again at the end
+    # of buildcol. And any symmetrical step during pre-building, if any is found necessary.
+    # I'm still not sure this won't break activate in some way, for some combination,
+    # as that is meant to ensure building-cores exist whenever the building folder exists...
+    # But I was asked not to take to long on this, so I can't test all the different combinations
+    # (removeold/incremental/..., or even remote GS situation) in which building can happen and in
+    # which buildcol can be combined or not with activate, or be sequenced with further calls to
+    # buildcol, with or without -activate.
+    # So to compensate, I've tried to keep the code changes as conservative as possible to keep the chances
+    # for things going wrong to a minimum, by pinging for building-* cores before unloading them here
+    # in solrbuilder.pm (note that unload doesn't delete the index directory associated with the core)
+    # and then in activate.pl the building-* cores get pinged again to determine whether they exist
+    # before attempting to unload them there as well, since I can no longer assume the corse exist
+    # and can be unloaded. There is now the additional overhead of all the extra pinging going on,
+    # but it helps ensure we only unload building-* cores when they exist.
+
+    # Note that pre-build-indexes() was already creating the building- cores, so don't need to
+    # worry about the needed symmetry at start and end of buildcol to create building- cores
+    # in symmetry with unloading them here.
+    
+    my $site        = $self->{'site'};
+    my $collect     = $self->{'collection'};
+    my $core_prefix = (defined $site) ? "$site-$collect" : $collect;    
+    my $build_dir = $self->{'build_dir'};
+    my $idx = "idx";
+    
+    foreach my $level (keys %{$self->{'levels'}}) {
+    
+    my ($pindex) = $level =~ /^(.)/;
+
+    my $index_dir = $pindex.$idx;
+    my $corename = "building-$core_prefix-$index_dir";
+
+    # unload the core if it exists. Since I can't think of exactly in which cases
+    # the building-cores exist and won't exist (e.g. removeold), I'll just always
+    # first check if the building-core exists and then unload it.
+    if ($solr_server->admin_ping_core($corename)) {
+        print STDERR "@@@@ solrbuilder::post_build_indexes(): Now unloading this index's building core: $corename\n\n";
+        $solr_server->admin_unload_core_explicitly_retaining_index($corename)
+    }
+
+    }
+    
     # Also need to stop the Solr server (be it tomcat or jetty) if it was explicitly started
     # in pre_build_indexes()
-    
-    my $solr_server = $self->{'solr_server'};
 
     if ($solr_server->explicitly_started()) {

gs3-extensions/solr/trunk/src/perllib/solrserver.pm

@@ -407 +407 @@
 }
 
+sub admin_unload_core_explicitly_retaining_index
+{
+    # For UNLOAD core params, see page 315 of
+    #    https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.7.pdf
+    
+    my $self = shift @_;
+    my ($core) = @_;
+    
+    # Don't delete index (sidx/didx folder) along with unloading core, so force 0 as parameter
+    # (though not deleting the index is the default behaviour of admin_unload_core() anyway,
+    # since activate is meant to manually take care of deleting the index folder and moving the
+    # building folder to replace index, instead of activate asking unload_core to delete the
+    # index folder).
+    # But this function's very particular behaviour may be crucial for other instances such as
+    # its use in solrbuilder::post_build_indexes(), so even if admin_unload_core() could
+    # conceivably be changed to delete the index by default, this method would still do the
+    # right thing when called by solrbuilder::post_build_indexes().
+    $self->admin_unload_core($core, 0);
+}
+
+
 sub start
 {

main/trunk/greenstone2/bin/script/activate.pl

@@ -277 +277 @@
     
     # If the Solr/Jetty server is not already running, the following starts
-    # it up, and only returns when the server is "reading and listening"    
+    # it up, and only returns when the server is "ready and listening"  
     $solr_server = new solrserver($build_dir);
     $solr_server->start();
     
-    # We'll be moving building to index. For solr collection, there's further 
+    # We'll be moving building to index. For a solr collection, there's further 
     # special processing to make a corresponding change to the solr.xml
     # by removing the temporary building cores and (re)creating the index cores
@@ -307 +307 @@
         # if solr, remove any cores that are using the index_dir before deleting this dir
         foreach my $corename (@corenames) {
-            $solr_server->admin_unload_core($corename);
+            $solr_server->admin_unload_core($corename) if ($solr_server->admin_ping_core($corename));
         }
         }   
@@ -327 +327 @@
         # if solr, remove any cores that are using the building_dir before moving this dir onto index
         foreach my $corename (@corenames) {
-        $solr_server->admin_unload_core("building-$corename");
+        $solr_server->admin_unload_core("building-$corename") if ($solr_server->admin_ping_core("building-$corename"));
         }
     }
@@ -376 +376 @@
             $solr_server->admin_unload_core($corename); 
         } 
-            $solr_server->admin_create_core($corename, $index_dir);
+        $solr_server->admin_create_core($corename, $index_dir);
         
         }