Changeset 32088 for gs3-extensions/solr/trunk/src/perllib

Timestamp:

2017-12-08T17:58:07+13:00 (6 years ago)

Author:

ak19

Message:

Martin (mwilliman email id) on the mailing list found that solr got SIGPIPE errors when he built his 3020 doc sorl collection. The problem occurred when the docs were sent in a single stream for solr ingestion using the SimplePostTool (post.jar/solr-post.jar). The problem is that the data stream becomes to large, since SimplePostTool doesn't cause a commit until after the pipe to it is closed. Initially other methods were attempted: increasing the Java VM mem size from 512 to 2048, which only helped process a certain additional number of docs before resulting in a SIGPIPE again. We tried changing the solr update url to have ?commit=true and ?commitWithin=15000 (ms) suffixed to it, but as the commit isn't done until after the pipe to SimplePostTool is closed, the url change had no effect with SimplePostTool. Though we retained an increase to 1024 of the Java VM when launching SimplePostTool, the actual present solution was to close and reopen the pipe to the post tool jar file executable after every x number of docs. Currently this batch size is set to 20. However, if any file is gigantic, we could get to see this problem again: it has to do with the overall size of the data stream rather than number of docs. The actual problem lies in HttpURLConnection that SimplePostTool opens, rather than how often we open/close the open to the post tool. This commit contains 3 changes: 1. changed Java VM memory to 1024 when launching SimplePostTool (solr-post.jar); 2. code changes to solrutil.pm and solr_passes.pl to close and reopen the pipe to flush the data after every 20 docs to force a commit to solr; 3. the existing code changes work with the old solr-post.jar (version 1.3) but committing version 1.5 since it has a larger buffer and is found to be better by Dr Bainbridge. The new, v1.5 solr-post.jar is from solr-4.7.2's example/examples/post.jar, renamed to the usual solr-post.jar.

File:

• gs3-extensions/solr/trunk/src/perllib/solrutil.pm (modified) (2 diffs)

gs3-extensions/solr/trunk/src/perllib/solrutil.pm

@@ -112 +112 @@
 }
 
-
-sub open_post_pipe
+sub get_post_pipe_cmd
 {
     my ($core, $solr_base_url) = @_;
@@ -125 +124 @@
     
     # Now run solr-post command
+    # See https://wiki.apache.org/solr/UpdateXmlMessages
+    # also https://lucene.apache.org/solr/4_2_1/tutorial.html
+        # suffixing commit=true/commitWithin=10000 to solr's /update servlet didn't work, because
+        # when using SimplePostTool, the commit only happens after the pipe to the tool is closed
     my $post_props = "-Durl=$solr_base_url/$core/update"; # robustness of protocol is taken care of too
 
     $post_props .= " -Ddata=stdin";
     $post_props .= " -Dcommit=yes";
+
+    # increased VM mem from 512 to 1024, but increasing to 2048M didn't help either when too much
+    # data streamed to SimplePostTool before commit. Nothing works short of committing before the
+    # data streamed gets too large. The solution is to close and reopen the pipe to force commits.
+    my $post_java_cmd = "java -Xmx1024M $post_props -jar \"$full_post_jar\"";
     
-    my $post_java_cmd = "java -Xmx512M $post_props -jar \"$full_post_jar\"";
+       ##print STDERR "**** post cmd = $post_java_cmd\n";
     
-    ##print STDERR "**** post cmd = $post_java_cmd\n";
+    return $post_java_cmd;
+}
+
+sub open_post_pipe
+{
+    my ($core, $solr_base_url) = @_;
+    my $post_java_cmd = &get_post_pipe_cmd($core, $solr_base_url);
+
+    open (PIPEOUT, "| $post_java_cmd") 
+    || die "Error in solr_passes.pl: Failed to run $post_java_cmd\n!$\n";
+
+    return $post_java_cmd; # return the post_java_cmd so caller can store it and reopen_post_pipe()
+}
+
+sub reopen_post_pipe
+{
+    my $post_java_cmd = shift(@_);
     
     open (PIPEOUT, "| $post_java_cmd") 
-    || die "Error in solr_passes.pl: Failed to run $post_java_cmd\n!$\n";
+    || die "Error in solrutil::reopen_post_pipe: Failed to run $post_java_cmd\n!$\n";
     
 }