Changeset 32321

Show
Ignore:
Timestamp:
02.08.2018 21:50:16 (13 months ago)
Author:
ak19
Message:

Describing new problem when the openoffice extension breaks SafeProcess?: readLine and all bufferedreader/inputstreamreader read() calls block and not even interruptedexceptions can unblock them.

Files:
1 modified

Legend:

Unmodified
Added
Removed
  • main/trunk/gli/src/org/greenstone/gatherer/util/Readme_Using_SafeProcess.txt

    r31720 r32321  
    924924 
    925925 
    926  
     926_____________________________________________________________________________________________________________________ 
     927 
     928P. STREAM GOBBLER **THREADS** ARE ALLOWED TO BLOCK (JVM WILL TIMESLICE). CAUSE INTERRUPTEDEXCEPTIONS TO END BLOCKS 
     929_____________________________________________________________________________________________________________________ 
     930 
     931In the InputStreamGobbler Thread classes for handling the proc.errstream and proc.outstream, the run() methods have a loop on readLine(): 
     932 
     933   while (!this.interrupted() && (line = br.readLine()) != null) { ... } 
     934 
     935This so far worked beautifully, since if a process was cancelled, SafeProcess would call gobbler.interrupt() and the above loop would terminate. If a process had ended and the process err and output streams therefore run out, line == null and the loop would terminate. In both cases, we're out of the while loop and the process' stream would have been proeprly handed and the clean up would have taken place. 
     936 
     937However, BufferedReader.read() and consequently its readLine() are actually blocking calls. This was discovered when the OpenOfficeExtension was included in a GS installation and didn't work properly: if it fails to launch soffice --headless to accept on some port, then soffice is not running headless in the background. At this point, the stderr stream of the perl process that launched the OpenOfficePlugin (and OpenOfficeConverter) worked fine and output the pluginfo for the plugin and finished, terminating the while loop for the err stream gobbler. However, the process' output stream gobbler never got any data, including eof(). No cancel button was pressed, so this.interrupted() wasn't true. And br.readLine() wouldn't return but blocked: it waited for data that never came. This blocked SafeProcess' call to join() on that gobbler thread, as its run() method never terminated. 
     938 
     939The first solution is checking bufferedReader.ready() BEFORE attempting to any read() operation on the BufferedReader: 
     940 
     941     // readLine() can block if nothing is forthcoming, including eof marker. 
     942     // So must check BufferedReader.ready() before attempting readLine() on it 
     943     // See https://stackoverflow.com/questions/15521352/bufferedreader-readline-blocks 
     944     while ( !this.isInterrupted() && br.ready() && (line = br.readLine()) != null ) { ... 
     945 
     946 
     947HOWEVER, there are 2 problems with this: 
     948 
     9491. As per the API, ready() returns "True if the next read() is guaranteed not to block for input, false otherwise. Note that returning false does not guarantee that the next read will block." 
     950 
     951   NOTE: As per Andrew Mackintosh the above does not mean what I understood it to mean. I thought it means that if a read() blocks at this moment and therefore ready() returns false now, it need not meant it will block 5 mins from now if there was finally data sent to the process' stdout 5 mins from now. Andrew says that what the above means is that if ready() returns false a read() operation may OR may not block at present, whereas if ready() returns true, doing read() is guaranteed to NOT block. 
     952 
     9532. In the example of the OpenOfficeExtension, what happened is that the process' exit value was always 141, non-zero hence not a success. 141 signifies a SIGPIPE error: 
     954 
     955        // NOTE: exitValue can be 141 when we got nothing not even eof from any of the process' 
     956        // output streams. According to http://tldp.org/LDP/abs/html/exitcodes.html 
     957        // an exit value of 128+n means Fatal error signal "n". 128 + 13 = 141, so n = 13. 
     958        // And "Signal 13 is SIGPIPE" see https://github.com/Tunnelblick/Tunnelblick/issues/272 
     959        // 'SIGPIPE is the "broken pipe" signal, which is sent to a process when it attempts 
     960        // to write to a pipe whose read end has closed (or when it attempts to write to a 
     961        // socket that is no longer open for reading), but not vice versa. The default action 
     962        // is to terminate the process.' as per https://www.quora.com/What-are-SIGPIPEs     
     963         
     964 
     965Having realised that br.read() itself can block and how to get such blocking gobbler threads to terminate, I discussed the problem with Andrew. He explained the solution: 
     966 
     967- blocking IO calls in loops is FINE as long as they are in a separate thread and not the main thread: the OS, or the JVM in this case rather, will know to use timeslicing and set the processor on other tasks until blocking IO calls are ready again. 
     968 
     969- to "wake" up blocking calls on process termination, the "Scheduler" parent class that launched the child process and the StreamGobbler Threads should cause InterruptedExceptions on its StreamGobbler threads once the child process has exited. The Scheduler should internally induce InterruptedExceptions on the StreamGobblers regardless of whether the process terminated naturally or was cancelled by the user. The Exception would then end the blocking br.readLine(). 
     970 
     971In my case, the Scheduler is SafeProcess.java and is the interrupting Thread that is to send interrupted exceptions. So after doing 
     972  exitVal = proc.waitFor() 
     973the scheduler class SafeProcess should somehow cause InterruptedExceptions on its gobbler threads. 
     974 
     975I found these pages, which seem to imply indicate that the interrupting thread (SafeProcess in my case) should close the streams of the child process and that this will generate the necessary InterruptedExceptions stopping the read()/readLine() operations: 
     976 
     977- https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline (example speaks of bufferedreaders reading from sockets rather than from a process' IO streams) 
     978- https://arstechnica.com/civis/viewtopic.php?t=259678 
     979 
     980I can't get it to work yet though, as at 1 and 2 Aug 2018. 
     981 
     982 
     983ANOTHER IMPORTANT LESSON FROM ANDREW: 
     984Andrew Mackintosh further said that the read() block calls in worker Threads are far better in terms of not wasting processor power than a constantly active while loop in the worker Thread. 
     9851. So doing this: 
     986 
     987    while(!thisthread.isInterrupted() && br.read() ) // br.read() possibly blocks 
     988 
     989is good since if the read() the blocks, the OS or JVM knows to reuse the processor for other tasks. (I think of it as the OS/JVM considers blocking calls as being idle processes and therefore puts other tasks onto processor in the meantime.) 
     990 
     991Whereas: 
     992   while(!thisthread.isInterrupted()) { 
     993      if(!br.ready) continue; // will keep testing interrupted and ready() and so keeps using the processor. 
     994      else br.read(); 
     995   } 
     996will keep the processor/core wastefully occupied even though there may be long stretches when there's no data to read. 
     997 
     998 
     999UNFORTUNATELY, 
     1000https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline 
     1001says that BufferedReader.readLine() can't even be interrupted. And their solution doesn't work either. 
     1002 
     1003Other links: 
     1004- EOF/EOS (End of Stream) is indicated by a return value of -1 for read and null for readLine(). 
     1005https://stackoverflow.com/questions/36569875/how-does-bufferedreader-readline-handle-eof-or-slow-input 
     1006https://stackoverflow.com/questions/3714090/how-to-see-if-a-reader-is-at-eof