Changeset 32088 for gs3-extensions/solr/trunk/src/bin/script/solr_passes.pl

Timestamp:

2017-12-08T17:58:07+13:00 (6 years ago)

Author:

ak19

Message:

Martin (mwilliman email id) on the mailing list found that solr got SIGPIPE errors when he built his 3020 doc sorl collection. The problem occurred when the docs were sent in a single stream for solr ingestion using the SimplePostTool (post.jar/solr-post.jar). The problem is that the data stream becomes to large, since SimplePostTool doesn't cause a commit until after the pipe to it is closed. Initially other methods were attempted: increasing the Java VM mem size from 512 to 2048, which only helped process a certain additional number of docs before resulting in a SIGPIPE again. We tried changing the solr update url to have ?commit=true and ?commitWithin=15000 (ms) suffixed to it, but as the commit isn't done until after the pipe to SimplePostTool is closed, the url change had no effect with SimplePostTool. Though we retained an increase to 1024 of the Java VM when launching SimplePostTool, the actual present solution was to close and reopen the pipe to the post tool jar file executable after every x number of docs. Currently this batch size is set to 20. However, if any file is gigantic, we could get to see this problem again: it has to do with the overall size of the data stream rather than number of docs. The actual problem lies in HttpURLConnection that SimplePostTool opens, rather than how often we open/close the open to the post tool. This commit contains 3 changes: 1. changed Java VM memory to 1024 when launching SimplePostTool (solr-post.jar); 2. code changes to solrutil.pm and solr_passes.pl to close and reopen the pipe to flush the data after every 20 docs to force a commit to solr; 3. the existing code changes work with the old solr-post.jar (version 1.3) but committing version 1.5 since it has a larger buffer and is found to be better by Dr Bainbridge. The new, v1.5 solr-post.jar is from solr-4.7.2's example/examples/post.jar, renamed to the usual solr-post.jar.

File:

• gs3-extensions/solr/trunk/src/bin/script/solr_passes.pl (modified) (3 diffs)

gs3-extensions/solr/trunk/src/bin/script/solr_passes.pl

@@ -50 +50 @@
 use solrutil;
 use solrserver;
+
+my $DOC_BATCH_SIZE = 20;
 
 # Not quite OO, but close enough for now
@@ -66 +68 @@
   $self->{'solr_server'} = $solr_server;
 
-  # Now start up the solr-post command
-  &solrutil::open_post_pipe($core, $solr_server->get_solr_base_url());
+  # Now start up the solr-post command and store the open cmd that is returned
+  $self->{'post_java_cmd'} = &solrutil::open_post_pipe($core, $solr_server->get_solr_base_url());
+}
+
+# To commit any stream of data to solr that's amassed so far on the pipe to SimplePostTool,
+# close the pipe. Then reopen it to continue streaming data to it.
+sub flush_and_reopen_java_solr
+{
+    &solrutil::close_post_pipe();
+    &solrutil::reopen_post_pipe($self->{'post_java_cmd'});
 }
 
@@ -160 +170 @@
 
 
+# Called when mode = index,
+# This is the function that passes the contents of the docs for ingesting into solr
+# as one long stream.
+# Since if we have many docs in a collection, there will be one long stream of bytes
+# sent to the SimplePostTool/(solr-)post.jar, which remain uncommitted until
+# the pipe is closed. And for a long bytestream, this will result in an out of heap memory
+# https://stackoverflow.com/questions/2082057/outputstream-outofmemoryerror-when-sending-http
+# So we need to close and then reopen solr post pipe to force commit after some $DOC_BATCH_SIZE
+# We could still have the same issue if any one document is very large (long stream of bytes),
+# but then we'd need to take care of the problem at the root, see the StackOverFlow page
+# and SimplePostTool.java, as the problem lies in HttpURLConnection.
 sub pass_on_xml_stream
 {
+    # the xml_stream sent to solr looks like:
+    # <update>
+    #   <add>
+    #     <doc>
+    #       ....
+    #     </doc>
+    #   </add>
+    #     <doc>
+    #       ....
+    #     </doc>
+    #   </add>
+    #   ...
+    # </update>    
+    
+    my $doc_count = 0;
+    
     my $line;
     while (defined ($line = <STDIN>)) {
+    # monitor for end of each document      
+    if ($line =~ m/^\s*<\/add>$/) {
+        $doc_count++;
+    }
+    else {
+        if ($doc_count == $DOC_BATCH_SIZE) {
+        # evidence that there is still more documents to process
+        # => but reached batch size, so flush and reopen
+        # to force a commit of past docs to solr
+        # before sending this current line on its way
+        
+        if ($line =~ m/^\s*<add>$/) {
+            &solrutil::print_to_post_pipe("</update>\n");
+            flush_and_reopen_java_solr();
+        
+            # start next doc
+            &solrutil::print_to_post_pipe("<update>\n");
+            $doc_count = 0;
+        }
+        }
+    }
+            
     &solrutil::print_to_post_pipe($line);
-    }
-}
-
-
+
+    }
+}