Changeset 32321


Ignore:
Timestamp:
2018-08-02T21:50:16+12:00 (3 years ago)
Author:
ak19
Message:

Describing new problem when the openoffice extension breaks SafeProcess: readLine and all bufferedreader/inputstreamreader read() calls block and not even interruptedexceptions can unblock them.

File:
1 edited

Legend:

Unmodified
Added
Removed
  • main/trunk/gli/src/org/greenstone/gatherer/util/Readme_Using_SafeProcess.txt

    r31720 r32321  
    924924
    925925
    926 
     926_____________________________________________________________________________________________________________________
     927
     928P. STREAM GOBBLER **THREADS** ARE ALLOWED TO BLOCK (JVM WILL TIMESLICE). CAUSE INTERRUPTEDEXCEPTIONS TO END BLOCKS
     929_____________________________________________________________________________________________________________________
     930
     931In the InputStreamGobbler Thread classes for handling the proc.errstream and proc.outstream, the run() methods have a loop on readLine():
     932
     933   while (!this.interrupted() && (line = br.readLine()) != null) { ... }
     934
     935This so far worked beautifully, since if a process was cancelled, SafeProcess would call gobbler.interrupt() and the above loop would terminate. If a process had ended and the process err and output streams therefore run out, line == null and the loop would terminate. In both cases, we're out of the while loop and the process' stream would have been proeprly handed and the clean up would have taken place.
     936
     937However, BufferedReader.read() and consequently its readLine() are actually blocking calls. This was discovered when the OpenOfficeExtension was included in a GS installation and didn't work properly: if it fails to launch soffice --headless to accept on some port, then soffice is not running headless in the background. At this point, the stderr stream of the perl process that launched the OpenOfficePlugin (and OpenOfficeConverter) worked fine and output the pluginfo for the plugin and finished, terminating the while loop for the err stream gobbler. However, the process' output stream gobbler never got any data, including eof(). No cancel button was pressed, so this.interrupted() wasn't true. And br.readLine() wouldn't return but blocked: it waited for data that never came. This blocked SafeProcess' call to join() on that gobbler thread, as its run() method never terminated.
     938
     939The first solution is checking bufferedReader.ready() BEFORE attempting to any read() operation on the BufferedReader:
     940
     941     // readLine() can block if nothing is forthcoming, including eof marker.
     942     // So must check BufferedReader.ready() before attempting readLine() on it
     943     // See https://stackoverflow.com/questions/15521352/bufferedreader-readline-blocks
     944     while ( !this.isInterrupted() && br.ready() && (line = br.readLine()) != null ) { ...
     945
     946
     947HOWEVER, there are 2 problems with this:
     948
     9491. As per the API, ready() returns "True if the next read() is guaranteed not to block for input, false otherwise. Note that returning false does not guarantee that the next read will block."
     950
     951   NOTE: As per Andrew Mackintosh the above does not mean what I understood it to mean. I thought it means that if a read() blocks at this moment and therefore ready() returns false now, it need not meant it will block 5 mins from now if there was finally data sent to the process' stdout 5 mins from now. Andrew says that what the above means is that if ready() returns false a read() operation may OR may not block at present, whereas if ready() returns true, doing read() is guaranteed to NOT block.
     952
     9532. In the example of the OpenOfficeExtension, what happened is that the process' exit value was always 141, non-zero hence not a success. 141 signifies a SIGPIPE error:
     954
     955        // NOTE: exitValue can be 141 when we got nothing not even eof from any of the process'
     956        // output streams. According to http://tldp.org/LDP/abs/html/exitcodes.html
     957        // an exit value of 128+n means Fatal error signal "n". 128 + 13 = 141, so n = 13.
     958        // And "Signal 13 is SIGPIPE" see https://github.com/Tunnelblick/Tunnelblick/issues/272
     959        // 'SIGPIPE is the "broken pipe" signal, which is sent to a process when it attempts
     960        // to write to a pipe whose read end has closed (or when it attempts to write to a
     961        // socket that is no longer open for reading), but not vice versa. The default action
     962        // is to terminate the process.' as per https://www.quora.com/What-are-SIGPIPEs   
     963       
     964
     965Having realised that br.read() itself can block and how to get such blocking gobbler threads to terminate, I discussed the problem with Andrew. He explained the solution:
     966
     967- blocking IO calls in loops is FINE as long as they are in a separate thread and not the main thread: the OS, or the JVM in this case rather, will know to use timeslicing and set the processor on other tasks until blocking IO calls are ready again.
     968
     969- to "wake" up blocking calls on process termination, the "Scheduler" parent class that launched the child process and the StreamGobbler Threads should cause InterruptedExceptions on its StreamGobbler threads once the child process has exited. The Scheduler should internally induce InterruptedExceptions on the StreamGobblers regardless of whether the process terminated naturally or was cancelled by the user. The Exception would then end the blocking br.readLine().
     970
     971In my case, the Scheduler is SafeProcess.java and is the interrupting Thread that is to send interrupted exceptions. So after doing
     972  exitVal = proc.waitFor()
     973the scheduler class SafeProcess should somehow cause InterruptedExceptions on its gobbler threads.
     974
     975I found these pages, which seem to imply indicate that the interrupting thread (SafeProcess in my case) should close the streams of the child process and that this will generate the necessary InterruptedExceptions stopping the read()/readLine() operations:
     976
     977- https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline (example speaks of bufferedreaders reading from sockets rather than from a process' IO streams)
     978- https://arstechnica.com/civis/viewtopic.php?t=259678
     979
     980I can't get it to work yet though, as at 1 and 2 Aug 2018.
     981
     982
     983ANOTHER IMPORTANT LESSON FROM ANDREW:
     984Andrew Mackintosh further said that the read() block calls in worker Threads are far better in terms of not wasting processor power than a constantly active while loop in the worker Thread.
     9851. So doing this:
     986
     987    while(!thisthread.isInterrupted() && br.read() ) // br.read() possibly blocks
     988
     989is good since if the read() the blocks, the OS or JVM knows to reuse the processor for other tasks. (I think of it as the OS/JVM considers blocking calls as being idle processes and therefore puts other tasks onto processor in the meantime.)
     990
     991Whereas:
     992   while(!thisthread.isInterrupted()) {
     993      if(!br.ready) continue; // will keep testing interrupted and ready() and so keeps using the processor.
     994      else br.read();
     995   }
     996will keep the processor/core wastefully occupied even though there may be long stretches when there's no data to read.
     997
     998
     999UNFORTUNATELY,
     1000https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline
     1001says that BufferedReader.readLine() can't even be interrupted. And their solution doesn't work either.
     1002
     1003Other links:
     1004- EOF/EOS (End of Stream) is indicated by a return value of -1 for read and null for readLine().
     1005https://stackoverflow.com/questions/36569875/how-does-bufferedreader-readline-handle-eof-or-slow-input
     1006https://stackoverflow.com/questions/3714090/how-to-see-if-a-reader-is-at-eof
Note: See TracChangeset for help on using the changeset viewer.