Changeset 32321

2018-08-02T21:50:16+12:00 (4 years ago)

Describing new problem when the openoffice extension breaks SafeProcess: readLine and all bufferedreader/inputstreamreader read() calls block and not even interruptedexceptions can unblock them.

1 edited


  • main/trunk/gli/src/org/greenstone/gatherer/util/Readme_Using_SafeProcess.txt

    r31720 r32321  
     931In the InputStreamGobbler Thread classes for handling the proc.errstream and proc.outstream, the run() methods have a loop on readLine():
     933   while (!this.interrupted() && (line = br.readLine()) != null) { ... }
     935This so far worked beautifully, since if a process was cancelled, SafeProcess would call gobbler.interrupt() and the above loop would terminate. If a process had ended and the process err and output streams therefore run out, line == null and the loop would terminate. In both cases, we're out of the while loop and the process' stream would have been proeprly handed and the clean up would have taken place.
     937However, and consequently its readLine() are actually blocking calls. This was discovered when the OpenOfficeExtension was included in a GS installation and didn't work properly: if it fails to launch soffice --headless to accept on some port, then soffice is not running headless in the background. At this point, the stderr stream of the perl process that launched the OpenOfficePlugin (and OpenOfficeConverter) worked fine and output the pluginfo for the plugin and finished, terminating the while loop for the err stream gobbler. However, the process' output stream gobbler never got any data, including eof(). No cancel button was pressed, so this.interrupted() wasn't true. And br.readLine() wouldn't return but blocked: it waited for data that never came. This blocked SafeProcess' call to join() on that gobbler thread, as its run() method never terminated.
     939The first solution is checking bufferedReader.ready() BEFORE attempting to any read() operation on the BufferedReader:
     941     // readLine() can block if nothing is forthcoming, including eof marker.
     942     // So must check BufferedReader.ready() before attempting readLine() on it
     943     // See
     944     while ( !this.isInterrupted() && br.ready() && (line = br.readLine()) != null ) { ...
     947HOWEVER, there are 2 problems with this:
     9491. As per the API, ready() returns "True if the next read() is guaranteed not to block for input, false otherwise. Note that returning false does not guarantee that the next read will block."
     951   NOTE: As per Andrew Mackintosh the above does not mean what I understood it to mean. I thought it means that if a read() blocks at this moment and therefore ready() returns false now, it need not meant it will block 5 mins from now if there was finally data sent to the process' stdout 5 mins from now. Andrew says that what the above means is that if ready() returns false a read() operation may OR may not block at present, whereas if ready() returns true, doing read() is guaranteed to NOT block.
     9532. In the example of the OpenOfficeExtension, what happened is that the process' exit value was always 141, non-zero hence not a success. 141 signifies a SIGPIPE error:
     955        // NOTE: exitValue can be 141 when we got nothing not even eof from any of the process'
     956        // output streams. According to
     957        // an exit value of 128+n means Fatal error signal "n". 128 + 13 = 141, so n = 13.
     958        // And "Signal 13 is SIGPIPE" see
     959        // 'SIGPIPE is the "broken pipe" signal, which is sent to a process when it attempts
     960        // to write to a pipe whose read end has closed (or when it attempts to write to a
     961        // socket that is no longer open for reading), but not vice versa. The default action
     962        // is to terminate the process.' as per   
     965Having realised that itself can block and how to get such blocking gobbler threads to terminate, I discussed the problem with Andrew. He explained the solution:
     967- blocking IO calls in loops is FINE as long as they are in a separate thread and not the main thread: the OS, or the JVM in this case rather, will know to use timeslicing and set the processor on other tasks until blocking IO calls are ready again.
     969- to "wake" up blocking calls on process termination, the "Scheduler" parent class that launched the child process and the StreamGobbler Threads should cause InterruptedExceptions on its StreamGobbler threads once the child process has exited. The Scheduler should internally induce InterruptedExceptions on the StreamGobblers regardless of whether the process terminated naturally or was cancelled by the user. The Exception would then end the blocking br.readLine().
     971In my case, the Scheduler is and is the interrupting Thread that is to send interrupted exceptions. So after doing
     972  exitVal = proc.waitFor()
     973the scheduler class SafeProcess should somehow cause InterruptedExceptions on its gobbler threads.
     975I found these pages, which seem to imply indicate that the interrupting thread (SafeProcess in my case) should close the streams of the child process and that this will generate the necessary InterruptedExceptions stopping the read()/readLine() operations:
     977- (example speaks of bufferedreaders reading from sockets rather than from a process' IO streams)
     980I can't get it to work yet though, as at 1 and 2 Aug 2018.
     984Andrew Mackintosh further said that the read() block calls in worker Threads are far better in terms of not wasting processor power than a constantly active while loop in the worker Thread.
     9851. So doing this:
     987    while(!thisthread.isInterrupted() && ) // possibly blocks
     989is good since if the read() the blocks, the OS or JVM knows to reuse the processor for other tasks. (I think of it as the OS/JVM considers blocking calls as being idle processes and therefore puts other tasks onto processor in the meantime.)
     992   while(!thisthread.isInterrupted()) {
     993      if(!br.ready) continue; // will keep testing interrupted and ready() and so keeps using the processor.
     994      else;
     995   }
     996will keep the processor/core wastefully occupied even though there may be long stretches when there's no data to read.
     1001says that BufferedReader.readLine() can't even be interrupted. And their solution doesn't work either.
     1003Other links:
     1004- EOF/EOS (End of Stream) is indicated by a return value of -1 for read and null for readLine().
Note: See TracChangeset for help on using the changeset viewer.