root/main/trunk/gli/src/org/greenstone/gatherer/util/Readme_Using_SafeProcess.txt @ 32322

Revision 32322, 85.6 KB (checked in by ak19, 13 months ago)

Furter updates to SafeProcess? Readme, even though Dr Bainbridge has fixed the OpenOffice? extension that led to all these Readme updates.

Line 
1First drafts of this document
2ak19, 30 May 2017
3
4=========================================
5    SAFEPROCESS.JAVA README
6=========================================
7
8GS Java developers should use SafeProcess.java in place of directly using Java's Process class, unless any issues are discovered with SafeProcess.java hereafter.
9
10
11A tailored version of SafeProcess is found both in GLI and GS3's Java src code:
12- gli/src/org/greenstone/gatherer/util/SafeProcess.java
13- greenstone3/src/java/org/greenstone/util/SafeProcess.java
14
15
16This document contains the following sections
17    A. WHY WE SHOULD GO THROUGH SAFEPROCESS INSTEAD OF USING JAVA'S PROCESS DIRECTLY
18    B. MODEL OF SAFEPROCESS THREADS AND INTERRUPT BEHAVIOUR
19    C. USAGE: DEBUGGING
20    D. USAGE: INSTANTIATION & DEFAULT RUNPROCESS VARIANTS
21    E. INTERNAL IMPLEMENTATION DETAILS - USEFUL IF CUSTOMISING SAFEPROCESS' BEHAVIOUR OR CALLING CANCEL ON SAFEPROCESS
22    F. USAGE: CUSTOMISING
23    G. USAGE: CANCELLING
24    H. USAGE: IMPLEMENTING HOOKS AROUND CANCELLING OR PROCESS' END
25    I. USEFUL STATIC METHODS AND STATIC INNER CLASSES
26    J. OTHER USEFUL INSTANCE METHODS
27    K. LINKS AND NOTES ON PROCESSES, PROCESS STREAMS, INTERRUPTING AND DESTROYING PROCESSES
28    L. USEFUL SYNCHRONISATION NOTES AND SUGGESTED READING ON THREADS AND THREAD SAFETY
29    M. TICKETS, COMMITS, TAGS
30    N. GLI vs GS3 CODE's SAFEPROCESS.JAVA - THE DIFFERENCES
31    O. FUTURE WORK, IMPROVEMENTS
32    P. ALTERNATIVE FUTURE WORK: A case for rewriting SafeProcess to use IO::Select (Selector in Java) instead of multi-threading
33    Q. PUBLIC API OF SAFEPROCESS, ITS SUB CLASSES AND INTERFACES
34
35
36_______________________________________________________________________________
37A. WHY WE SHOULD GO THROUGH SAFEPROCESS INSTEAD OF USING JAVA'S PROCESS DIRECTLY
38_______________________________________________________________________________
39
40It's easy to misuse Java's Process class, and this can result in unexpected behaviour and random errors that are hard to track down. Further, the older GLI and GS3 Java src code used to use Java's Process class in different and inconsistent ways, some worse than others. Going through one class, SafeProcess, provides consistency. So if it's buggy, you can fix it in one place.
41
42SafeProcess handles the internal Process' iostreams (input, output and error) properly using separate worker Threads, following http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html?page=2.
43
44SafeProcess handles this itself so that the code that uses SafeProcess doesn't need to deal with it, unless customising the processing of the Process' iostreams.
45
46
47
48_________________________________________________________
49B. MODEL OF SAFEPROCESS THREADS AND INTERRUPT BEHAVIOUR
50_________________________________________________________
51
52This section is especially IMPORTANT TO READ IF you're thinking of customising SafeProcess' handling of the internal Process' iostreams, since each of them are always handled in their own distinct worker threads, and you need to remember to deal with ThreadSafety issues.
53
54The primary process Thread:
55The main process internal to a SafeProcess instance is run in whatever thread the SafeProcess instance's runProcess() (or runBasicProcess()) methods are called.
56The SafeProcess instance internally keeps track of this thread, so that any cancel operation can send the InterruptedException to this primary thread.
57
58SafeProcess does NOT inherit from Thread, it is therefore not a Thread. It just runs its internal process in whatever Thread called SafeProcess's runProcess/runBasicProcess methods.
59
60A SafeProcess thread may further launch 0 or 3 additional worker threads, depending on whether runBasicProcess() or a runProcess() variant was called.
61
62In total, using SafeProcess involves
63- 1 thread (the primary thread) when runBasicProcess() is called
64- or 4 threads (the primary thread + 3 worker threads) when any variant of runBasicProcess() is called. If a variant of runProcess() was called, then 3 additional worker threads are spawned by SafeProcess. One for each iostream of the Process: inputstream, outputstream and errorstream. Note that the SafeProcess class will read from the internal Process' outputStream and ErrorStream, and can send/write a string to the Process' inputstream.
65
66
67            SafeProcess thread
68                |
69                |
70        ________________|_______________________________
71        |       |       |       |
72        |       |       |       |
73        |       |       |       |
74    Worker Thread     SafeProcess     Worker Thread   Worker Thread
75    for Process'        Thread     for Process'    for Process'
76    InputStream    (primary)       OutputStream    ErrorStream
77
78
79__________________________________________________
80C. USAGE: DEBUGGING
81__________________________________________________
82
83There's a lot of info that a SafeProcess instance will log during its execution and termination phases, including Exceptions but also very basic things, such as the command that the SafeProcess instance is running and other messages.
84
85Exceptions are always logged, but debugging related messages can be turned off or on. By default, it's turned off. You can turn it on by adjusting the static SafeProcess.DEBUG = 1 and recompiling.
86
87For GS3, the log output uses log4j.
88For GLI, it goes to System.err at present. But if you want it to go to GLI's DebugStream, uncomment the relevant lines in the 3 variants of the static log() functions in SafeProcess.java. Then recompile.
89
90If you're using SafeProcess yourself and wish to log debug statements that are related to how well you're using SafeProcess, you can call one of the static SafeProcess.log() functions yourself. Remember to set the static SafeProcess.DEBUG = 1.
91
92
93There are 4 variants of the log() function:
94
95    - public static void log(String msg)
96    logs the message if debugging is on, DEBUG = 1
97
98    - public static void log(Exception e)
99    logs the stacktrace for the exception, regardless of if debugging is on
100
101    - public static void log(String msg, Exception e)
102    logs the message and logs the stacktrace for the exception, regardless of if debugging is on
103
104    - public static void log(String msg, Exception e, boolean printStackTrace)
105    if debugging is on or if the printStackTrace parameter is true, both the message and the stacktrace for the exception are logged.
106    if debugging is off and the printStackTrace parameter is false, nothing is logged. Call this variant if an exception is expected and you only want to log it in debugging mode. An example of an expected exception are some InterruptedExceptions that may occur over the course of SafeProcess
107
108_______________________________________________________
109D. USAGE: INSTANTIATION & DEFAULT RUNPROCESS VARIANTS
110_______________________________________________________
111
112Usage of the SafeProcess class generally follows the following sequence:
113
1141. instantiate a SafeProcess object using one of the constructors
115    - public SafeProcess(String[] cmd_args)
116    - public SafeProcess(String cmdStr)
117    - public SafeProcess(String[] cmd_args, String[] envparams, File launchDir)
118    Either or both of envparams and launchdir can be null.
119
120
1212. optionally configure your SafeProcess instance.
122Use an appropriate setter method to set some additional fields:
123
124    - public void setInputString(String sendStr)
125    Call this if you wish to write any string to the process' inputstream
126
127    - public void setSplitStdOutputNewLines(boolean split)
128    - public void setSplitStdErrorNewLines(boolean split)
129    Pass in true to either of the above methods if you want to preserve individual lines in the content retrieved from the internal Process' stderr and stdoutput.
130    By default, lines are not split, so you get a String of one single line for the entire output from the Process' stdoutput, and a String of one single line from the Process' stderr output.
131   
132
133    - public void setExceptionHandler(SafeProcess.ExceptionHandler exception_handler)
134    Use this to register a SafeProcess ExceptionHandler whose gotException() method will get called for each exception encountered during the *primary* thread's execution: the thread that runs the Process. If you wish to handle exceptions that could happen when reading from the Process' outputStream/errorStream, you will need to do that separately. See the section CUSTOMISING. By default, exceptions when reading from Process' output and errorStreams are logged.
135   
136    - public void setMainHandler(SafeProcess.MainProcessHandler handler)
137    To set a handler that will handle the primary (SafeProcess) thread, your handler will need to implement the hooks that will get called during the internal Process' life cycle, such as before and after process.destroy() is called upon a cancel operation.
138
139
1403. call one of the following runProcess() variants on your SafeProcess instance:
141
142    a. public int runBasicProcess()
143
144This variant will not handle the Process' iostreams at all, so it won't launch any worker threads. It merely runs the internal process in the thread from which it is called (the primary thread) and waits until it's done.
145Use this variant if you know that the external process you wish SafeProcess to run will NOT be expecting any input nor be producing any output (in stdout or stderror).
146
147NOTE: Do not call runBasicProcess() if you merely wish to ignore the process' iostreams. Because, in Java 6 and earlier, this can block indefinitely if any of the Process' iostreams contain anything or expect anything. If you wish to ignore a process' iostreams, or if you're not sure if they may need handling, call the zero-argument runProcess() variant described below.
148
149    b. public int runProcess()
150
151This zero argument variant will handle all the Process' iostreams in the DEFAULT manner.
152
153- If you wanted to send something to the Process' inputstream, you ought to have configured this by calling setInputString(sendStr) before calling runProcess(). The DEFAULT behaviour is for sendStr to be written to the internal Process' inputstream as soon as the process runs (happens in its own thread).
154
155- once runProcess() finishes, you can inspect what had come out of the Process' output and error streams by calling, see point 4 below.
156The DEFAULT behaviour of SafeProcess' processing of the stdout and stderr streams is for both streams to be read one line at a time until they have nothing left. Note that both streams are always dealt with in their separate worker threads.
157
158Resources are safely allocated at the end of the worker threads dealing with each of the internal Process' iostream, regardless of whether the Process is allowed to terminate naturally or whether SafeProcess is instructed to prematurely terminate it.
159
160    c. public int runProcess(SafeProcess.LineByLineHandler outLineByLineHandler, SafeProcess.LineByLineHandler errLineByLineHandler)
161Use this variant if you want to do anything specific when each line comes in from the internal Process' stderr and stdout streams. Passing in null for either parameter will return to the default behaviour for that stream.
162You'll want to read further details in the section on CUSTOMISING.
163
164    d. public int runProcess(CustomProcessHandler procInHandler, CustomProcessHandler procOutHandler, CustomProcessHandler procErrHandler)
165Use this variant if you want to completely override the way the internal Process' iostreams are handled. Passing in null for any of the parameters will return to the default behaviour for that stream. You'll want to read further details in the section on CUSTOMISING.
166
167
168All the runProcess() variants do a Process.waitFor(). This means all runProcess() variants block the primary thread, which is the thread in which they're executing, until the internal Process has completed.
169
170If you want to completely override the default behaviour of any of SafeProcess' iostream related worker threads (such as if you want to read a char at a time from the stderr stream and do something, instead of the default behaviour of reading a line at a time from it), then call this method.
171
172You need to pass in an *instance* of SafeProcess.CustomProcessHandler for *each* of the 3 iostreams, since they're running in separate threads and so can't share the same instance.
173
174You can pass in null for any of these, if you want the default SafeProcess handling to apply for that stream's worker thread. For any iostream's default handling that you want to override, however, you'd extend SafeProcess.CustomProcessHandler.
175
176You would need to take care of ThreadSafety yourself if you're extending a SafeProcess.CustomProcessHandler. You can maintain concurrency by using synchronization, for instance. See the section USEFUL SYNCHRONISATION NOTES.
177
178For EXAMPLES OF USE, see GS3's GS2PerlConstructor.java and GLI's DownloadJob.java.
179
180
181Further NOTES:
182All of the variants of the runProcess methods return the internal Process' exit value *after* the internal Process has completed. The exit value will be -1 if the process was prematurely terminated such as by a cancel operation, which would have interrupted the primary and subsidiary threads.
183
184Running a process blocks the primary thread until complete. So all the runProcess variants above *block*. (Unless and until an external thread calls the cancelRunningProcess() method on a reference to the SafeProcess instance, discussed in the section on CANCELLING).
185
186
1874. if necessary, once the process has *terminated*,
188
189- you can read whatever came out from the Process' stderr or stdout by calling whichever is required of:
190public String getStdOutput()
191public String getStdError()
192 
193If you want to do anything special when handling any of the Process' iostreams, see the CUSTOMISING section.
194
195- You can also inspect the exit value of running the process by calling
196public int getExitValue()
197
198The exit value will be -1 if the process was prematurely terminated such as by a cancel operation, which would have interrupted the primary and subsidiary threads.
199The exit value will also be -1 if you call getExitValue() before setting off the SafeProcess via any of the runProcess() variants.
200
201_____________________________________________________________________________________________________________________
202E. INTERNAL IMPLEMENTATION DETAILS - USEFUL IF CUSTOMISING SAFEPROCESS' BEHAVIOUR OR CALLING CANCEL ON SAFEPROCESS
203_____________________________________________________________________________________________________________________
204
205This section is for understanding how SafeProcess runs a Java Process internally. These methods are not public and their details are hidden away in the code.
206
207Calling any of the runProcess() variants does the following internally:
208
2091. calls method doRuntimeExec()
210
211    based on how you configured your SafeProcess instance, this method instantiates and returns a Process object by calling Runtime.exec().
212    A possible improvement may be to consider using the ProcessBuilder class for creating a Process object.
213
2142. calls method waitForStreams()
215
216    Called by all but the runBasicProcess() variant, since the runBasicProcess() variant does not do anything with the Process' iostreams and therefore doesn't use worker threads .
217    This method first starts off all the worker threads, to handle each input or output stream of the Process.
218    Then, it calls waitFor() on the Process object. This is the first of 2 calls that block the primary thread (the Thread the Safeprocess runs its Process object in) until finished.
219
220    One of two things happen,
221    - either the internal Process is allowed to terminate naturally, reaching the end.
222    - Or the Process is interrupted by a cancel action, which will result in an InterruptedException. When an InterruptedException is caught, each of the worker threads is interrupted. The worker threads check for the interrupt flag during any loop, so that they may exit their loop early. If an InterruptException happened, the exitValue after the Process' waitFor() call remains at -1 and the SafeProcess instance sets its boolean forciblyTerminateProcess flag to true, to indicate that it may have to call Process.destroy() and ensure any subprocesses are terminated.
223   
224    Regardless of whether the Process naturally terminates or is prematurely terminated by an InterruptedException upn a cancel,
225    - all the resources such as streams are closed.
226    - the join() method is called on all worker threads. As per the Java API, callin join() on a Thread "Waits for this thread to die." This means the primary thread now waits for the worker threads to all come to a halt. So the section where join() is called on each worker thread also blocks the primary thread until the joins are complete.
227
228Since both the Process.waitFor() phase and the join() phase block the primary thread until finished, an InterruptedException could theoretically have occurred during either phase. However, we don't want an InterruptedException occurring during the join() phase, as an interrupt at this point will break us out of the join (going into an exception handler for the Interruption, prematurely ending the join()) and thus the worker threads would have been prevented from neatly cleaning up after themselves. To prevent InterruptedExceptions from occurring during any of the join() calls on the worker threads, the join() calls are embedded in a section that is marked as not interruptible. Once the join calls are finished, the code in the waitForStreans() method becomes interruptible again and a notify() is sent to anyone that was waiting for the thread to become interruptible. (Technically, the remainder of the code, the rest of the primary thread, does not respond to InterruptedExceptions hereafter, as there are no more waiting/blocking calls after the join()s are over.)
229
230Marking the join() phase as an uninterruptible section of code helps when the cancelRunProcess() is called on the SafeProcess instance by a separate thread, such as a GUI thread with a cancel button that triggered the call to cancelRunProcess(). The cancelRunProcess() method only interrupts the primary SafeProcess thread if it is in an interruptible section of code. If the primary thread is uninterruptible when cancelRunProcess() was called, then the method either
231- returns immediately if called by a GUI thread (EventDispatchThread),
232- or if cancelRunProcess(boolean forceWaitUntilInterruptible) is called with forceWaitUntilInterruptible, even if the caller is the EventDispatchThread, it will wait until the primary thread is interruptible before it returns from the cancelRunProcess() method. The wait() call blocks until notify()ed that the primary thread has become interruptible again.
233
234The cancelRunningProcess() method, which is called by a Thread external to the primary SafeProcess thread, therefore only causes an InterruptedException on the primary SafeProcess thread. It never sends an interruption during or after the primary thread is in an uninterruptible section, since only the waitFor() phase is ever to be interrupted, as waitFor() is the only phase when the external process is actually running, while the subsequent phases are already in closing down naturally and only need to do cleanup thereafter. If cancelRunningProcess() is called when the primary thread is in the uninterruptible phase after waitFor(), the cancelRunningProcess() method returns straight away or can be told to wait until the uninterupptible phase is done. Either way, it won't cause an interrupt any more, as the process is already naturally closing down.
235
236
2373. If and only if SafeProcess was cancelled, so that its Process is to be prematurely terminated, does SafeProcess call destroyProcess(). (It is not called on natural termination.) destroyProcess() ensures both the external process that is run, denoted by the Process object, and also any subprocesses that are launched by the external process are all terminated.
238
239- On Linux (only tested on Ubuntu), it was found that the simple sequence of events from interrupting the worker threads to the join()s called on them (which closed off al resources used by the worker threads) was sufficient to cleanly end the Process and any subprocesses.
240
241- On Windows, it needs to call Process.destroy() to terminate the Process and further needs to obtain the childids of any subprocesses this launched and terminate them recursively in a Windows-specific manner by calling the Windows specific kill command on each process id.
242The method "public static long getProcessID(Process p)" uses the newly included Java Native Access library to obtain the processID of any Process. Then the childpids of any process denoted by pid are obtained with the command:
243
244    "wmic process where (parentprocessid=<pid>) get processid"
245
246Having the pids of all processes to be terminated, one of "wmic process <pid> delete", "taskkill /f /t /PID <pid>" or "tskill pid" commands is invoked, whichever is available, to recursively and forcibly terminate any process and its subprocesses, by their pid. The taskkill command works like kill -9 (kill -KILL) rather than like kill -TERM.
247
248The wmic prompt-like tool, which has been present since Windows XP, not only recongises the command to delete a process by pid, but it also has commands to get the childids of a process denoted by pid, and vice-versa (to find the parent pid if you know a childpid). Much more is possible with wmic. The tskill command has also been available for a long time, taskkill is newer and may have appeared with Windows Vista or later.
249
250- On Mac, pressing cancel when building a collection in GLI launches the perl processes full-import.pl followed by full-buildcol.pl. The buildcol phase terminates even without a process.destroy() being necessary, which is as on Linux. However, for full-import.pl, process.destroy() firstly is necessary: as always, process.destroy() is able to destroy the Process that was launched (full-import.pl), but not any subprocesses that this launched in turn (such as import.pl). So on Mac, during destroyProcess(), an operating system command is issued to terminate the process and its subprocesses. See the NOTE at the end of the section.
251
252On Linux, SafeProcess() does not issue operating system commands to terminate the internal Process upon cancel, since on Linux both a process and its subprocesses seem to come to an end on their own in such a case. Nevertheless, SafeProcess now also includes code to send an OS level command to terminate a process with its subprocesses on Linux too. The way SafeProcess works on premature termination, with interrupts and closing of resources, doesn't yet require an OS level command. However, there's now a variant of destroyProcess() which takes a boolean parameter which, if false, will force an OS level command to be issued on Linux too if an entire process group (process and subprocesses) is to be destroyed.
253
254On Mac, the command is "pkill -TERM -P <pid>". This is to kill a process and subprocesses. It was tested successfully on Mac Snow Leopard.
255
256However, on Linux, this command only terminates the top level process, while any subprocesses are left running. On Linux, the OS level command to terminate the entire process group is "kill -TERM -<pid>", which has been tested successfully on Ubuntu. Note the hyphen in front of the <pid>, which treats the pid as a process group id, and terminates the entire process tree. (In contrast, the more common "kill -TERM <pid>" only terminates the top level process on Mac and Linux, leaving subprocesses running).
257
258For more information, see https://stackoverflow.com/questions/8533377/why-child-process-still-alive-after-parent-process-was-killed-in-linux
259
260NOTE:
261This note documents weird behaviour on Mac, which has not been explained yet, but which the above operating system commands to terminate the process group successfully taken care of anyway.
262
263When running the full-import.pl and full-buildcol.pl in sequence from the command line on a Mac, the regular "kill -TERM <pid> signal" only terminates the top level process and not subprocesses. To kill the entire process group, the OS specific command for killing a process (mentioned above) must be issued instead.
264When full-import.pl is run from GLI, both process.destroy() and the regular "kill -TERM <pid> signal" are able to successfully terminate the top level process, but not subprocesses. In this, sending the TERM signal results in behaviour similar to the situation with running full-import.pl from the command line.
265
266However, when full-buildcol.pl is run from GLI on a Mac the behaviour is inconsistent with the command line and with running full-import.pl from GLI. When full-buildcol.pl is run from GLI, the Process and subprocesses all terminate tidily on their own when cancel is pressed, not requiring either process.destroy() or the OS command to terminate even the top level process ID, nor requiring the command to terminate the process group either. This behaviour is similar to how both full-import.pl and full-buildcol.pl scripts come to a clean close upon cancel on Linux. In such cases, sending a TERM signal to the process (group) id returns an exitvalue of 1. This exitvalue can mean that the kill signal didn't work, but in these cases, it specifically indicates that the process was already terminated before the TERM signal was even sent to kill the process.
267
268A specific detail of the process of killing a process group on Linux and Mac is that a TERM signal is sent first. If the return value was 0, a signal was successfully sent. If it is 1 it failed in some way, including the possibility that process was already terminated. If neither of these, then it's assumed the return value indicates the termination signal was unsuccessful, and a KILL signal is sent next.
269
270Preliminary investigation of the weird behaviour on Mac has revealed the following:
271a. Sending a Ctrl-C is different in two ways from sending a kill -TERM signal.
272- Ctrl-C is "kill -2" or "kill -INT", whereas "kill -TERM" is the same as "kill -15" (and "kill -KILL" is "kill -9", which doesn't allow the killed process to do cleanup first).
273- Ctrl-C kills the foreground process of the progress group targeted by the Ctrl-C (or the process group indicated by pid). In contrast, kill -TERM or kill -KILL terminate the process with the given pid, which is likely to be a background process if any subprocesses were launched by it.
274
275b. Pressing Ctrl-C when full-import.pl is running may give the impression of having terminated the entire process group if you happen to issue the Ctrl-C at a point when full-import.pl hasn't launched any subprocesses. The parent process is killed, and therefore can't launch further subprocesses, so it looks like hte entire process tree was successfully terminated. However, if you issue Ctrl-C when a subprocess is running, that subprocess (the foreground process) is all that's terminated, while background processes are still running and can continue to launch subprocesses.
276
277Sending a regular kill -TERM or kill -KILL signal to the parent process of a process group will kill the parent process (but not the process group, unless you explicitly issued the command to terminate an entire process group). Killing the parent process will prevent future subprocesses from being launched by the parent process, but child processes remain runnning.
278
279The Java method Process.destroy(), used by GLI until the recent modifications to ensure the entire process group is terminated, behaves more like kill -TERM/kill -KILL than Ctrl-C: the actuall parent Process launched with Runtime.exec() is killed, but not subprocesses. This means that Ctrl-C is not a proper test of GLI's behaviour.
280
281Future investigation into the weird behaviour, if this ever needs to be explained, can proceed from the above. At present. issuing operating system commands to kill an entire process tree seem to work sufficiently in all cases tested, despite the weird distinctive behaviour of full-buildcol.pl on a Mac when run from GLI.
282
283__________________________________________________
284F. USAGE: CUSTOMISING
285__________________________________________________
286
287This section deals with if you want to provide customise behaviour for any of the worker threads that deal with the internal Process object's iostreams, or if you want to do some extra work at any major milestone in the outer SafeProcess instance's life cycle.
288
289
290CUSTOMISING THE BEHAVIOUR OF THE WORKER THREADS THAT HANDLE THE INTERNAL PROCESS' IOSTREAMS:
291
292Always bear in mind that each of the streams is handled in its own separate worker thread!
293So you'll need to take care of ThreadSafety yourself, if you're implementing a SafeProcess.LineByLineHandler or SafeProcess.CustomProcessHandler. You can maintain concurrency by using synchronization, for instance. See the section USEFUL SYNCHRONISATION NOTES.
294
295Customising the worker threads can be done in one of two ways:
296- you extend SafeProcess.LineByLineHandler (for the Process' stderr/stdout streams you're interested in) and then call the matching runProcess() method
297- if you want to do something drastically different, you extend SafeProcess.CustomProcessHandler (for any of the internal Process' iostreams you're interested in) and then call the matching runProcess() method
298
299    a. public int runProcess(SafeProcess.LineByLineHandler outLineByLineHandler, SafeProcess.LineByLineHandler errLineByLineHandler)
300
301If you want to deal with EACH LINE coming out of the internal Process' stdout and/or stderr streams yourself, implement SafeProcess.LineByLineHandler. You may want a different implementation if you want to deal with the Process' stdout and stderr streams differently, or the same implementation if you want to deal with them identically. Either way, you need a *separate* SafeProcess.LineByLineHandler instance for each of these two streams, since they're running in separate threads. Pass the distinct instances in to the runProcess method as parameters.
302
303You can pass null for either parameter, in which case the default behaviour for that iostream takes place, as described for the zero-argument runProcess() variant.
304
305
306Writing your own SafeProcess.LineByLineHandler:
307- extend SafeProcess.LineByLineHandler. You can make it a static inner class where you need it.
308- make the constructor call the super constructor, passing in one of SafeProcess.STDERR|STDOUT|STDIN, to indicate the specific iostream for any instantiated instance.
309- For debugging, in order to identify what stream (stderr or stdout) you're working with, you can print the thread's name by calling the CustomProcessHandler's method
310    public String getThreadNamePrefix()
311- Provide an implementation for the LineByLineHandler methods
312    public void gotLine(String line)
313    public void gotException(Exception e);
314
315Whenever the worker thread gets a line from that outputstream (stderr or stdout) of the internal process, it will call that stream's associated gotLine(line) implemention.
316Whenever the worker thread encounters an exception during the processing of that outputstream (stderr or stdout), it will call that stream's associated gotException(exception) implemention.
317
318Remember, you can use the same implementation (LineByLineHandler subclass) for both stderr and stdout outputstreams, but you must use different *instances* of the class for each stream!
319
320For EXAMPLES OF USE, see GLI's GShell.java.
321
322
323    b. public int runProcess(CustomProcessHandler procInHandler, CustomProcessHandler procOutHandler, CustomProcessHandler procErrHandler)
324
325If you want to completely override the default behaviour of any of SafeProcess' iostream related worker threads (such as if you want to read a char at a time from the stderr stream and do something, instead of the default behaviour of reading a line at a time from it), then call this method.
326
327You need to pass in an *instance* of SafeProcess.CustomProcessHandler for *each* of the 3 iostreams, since they're running in separate threads and so can't share the same instance.
328
329You can pass in null for any of these, if you want the default SafeProcess handling to apply for that stream's worker thread. For any iostream's default handling that you want to override, however, you'd implement SafeProcess.CustomProcessHandler.
330
331
332Writing your own SafeProcess.CustomProcessHandler:
333- extend SafeProcess.CustomProcessHandler. You can make it a static inner class where you need it.
334- make the constructor call the super constructor, passing in one of SafeProcess.STDERR|STDOUT|STDIN, to indicate the specific iostream for any instantiated instance.
335- implement the following method of CustomProcessHandler
336    public abstract void run(Closeable stream);
337
338Start by first casting the stream to InputStream, if the CustomProcessHandler is to read from the internal Process' stderr or stdout streams,
339else cast to Outputstream if CustomProcessHandler is to write to the internal Process' stdin stream.
340- Since you're implementing the "run()" method of the worker Thread for that iostream, you will write the code to handle any exceptions yourself
341- For debugging, in order to identify what stream you're working with, you can print the thread's name by calling the CustomProcessHandler's method
342    public String getThreadNamePrefix()
343
344Remember, you can use the same implementation (CustomProcessHandler subclass) for each of the internal process' iostreams, but you must use different *instances* of the class for each stream!
345
346For EXAMPLES OF USE, see GS3's GS2PerlConstructor.java and GLI's DownloadJob.java.
347
348
349ADDING CUSTOM HANDLERS TO HOOK INTO KEY MOMENTS OF THE EXECUTION OF THE PRIMARY THREAD:
350
351Remember that the primary thread is the thread in which the internal Process is executed in, which is whatever Thread the runProcess() method is called on your SafeProcess instance.
352
353If you just want to handle exceptions that may occur at any stage during the execution of the primary thread
354- provide an implementation of SafeProcess.ExceptionHandler:
355    public static interface ExceptionHandler {
356
357        public void gotException(Exception e);
358    }
359- configure your SafeProcess instance by calling setExceptionHandler(SafeProcess.ExceptionHandler eh) on it *before* calling a runProcess() method on that instance.
360- For EXAMPLES OF USE, see GLI's GShell.java.
361
362If, during the execution of the primary thread, you want to do some complex things during a cancel operation, such as or before and after a process is destroyed, then
363- implement SafeProcess.MainProcessHandler. See the Interface definition below. For further details, refer to the sections "CANCELLING" and "IMPLEMENTING HOOKS AROUND CANCELLING OR PROCESS' END".
364- configure your SafeProcess instance by calling setMainHandler(MainProcessHandler mph) on it *before* calling a runProcess() method on that instance.
365- For EXAMPLES OF USE, see GLI's DownloadJob.java.
366
367    public static interface MainProcessHandler {
368
369        /**
370         * Called before the streamgobbler join()s.
371         * If not overriding, the default implementation should be:
372         * public boolean beforeWaitingForStreamsToEnd(boolean forciblyTerminating) { return forciblyTerminating; }
373         * When overriding:
374         * @param forciblyTerminating is true if currently it's been decided that the process needs to be
375         * forcibly terminated. Return false if you don't want it to be. For a basic implementation,
376         * return the parameter.
377         * @return true if the process is still running and therefore still needs to be destroyed, or if
378         * you can't determine whether it's still running or not. Process.destroy() will then be called.
379         * @return false if the process has already naturally terminated by this stage. Process.destroy()
380         * won't be called, and neither will the before- and after- processDestroy methods of this class.
381        */
382        public boolean beforeWaitingForStreamsToEnd(boolean forciblyTerminating);
383
384        /**
385         * Called after the streamgobbler join()s have finished.
386         * If not overriding, the default implementation should be:
387         * public boolean afterStreamsEnded(boolean forciblyTerminating) { return forciblyTerminating; }
388         * When overriding:
389         * @param forciblyTerminating is true if currently it's been decided that the process needs to be
390         * forcibly terminated. Return false if you don't want it to be. For a basic implementation,
391         * return the parameter (usual case).
392         * @return true if the process is still running and therefore still needs to be destroyed, or if
393         * can't determine whether it's still running or not. Process.destroy() will then be called.
394         * @return false if the process has already naturally terminated by this stage. Process.destroy()
395         * won't be called, and neither will the before- and after- processDestroy methods of this class.
396        */
397        public boolean afterStreamsEnded(boolean forciblyTerminating);
398
399        /**
400         * called after join()s and before process.destroy()/destroyProcess(Process), iff forciblyTerminating
401         */
402        public void beforeProcessDestroy();
403 
404        /**
405         * Called after process.destroy()/destroyProcess(Process), iff forciblyTerminating
406         */
407        public void afterProcessDestroy();
408
409        /**
410         * Always called after process ended: whether it got destroyed or not
411         */
412        public void doneCleanup(boolean wasForciblyTerminated);
413    }
414
415__________________________________________________
416G. USAGE: CANCELLING
417__________________________________________________
418
419SafeProcess provides a way to cancel an internal Process that is run by SafeProcess, and which will ensure that the internal process and worker threads all safely terminate and tidily clean up after themselves.
420
421As explained, SafeProcess internally accomplishes this by sending an InterruptedException on the primary thread that's running the internal Process. An InterruptedException is sent only when SafeProcess is in an "interruptible" phase, as no interruption is necessary at any other time such as cleanup.
422
423Only a thread *external* to the SafeProcess' primary thread, such as a GUI thread, can call cancelRunningProcess() on a reference to the SafeProcess instance. Two variants are available:
424
425- boolean cancelRunningProcess()
426- boolean cancelRunningProcess(boolean forceWaitUntilInterruptible)
427
428The boolean value returned is true if an interrupt had to be sent, or false if one didn't need to be sent. An interrupt does not need to be sent if
429    - the process was never yet run by the time cancel was called or
430    - the internal Process has already naturally terminated and the SafeProcess has moved on to some stage of the cleanup phase by the time cancel was called.
431
432If you want to inspect the return value, a good variable name is sentInterrupt. For example,
433    boolean sentInterrupt = mySafeProc.cancelRunningProcess();
434
435
436Programming pattern:
437A graphical interface may have a Cancel button associated with an actionPerformed() method, which will be run from a GUI/EventDispatchThread when the user presses cancel. Either directly or indirectly, actionPerfomed() during a cancel action needs to lead to one of the cancelRunningProcess() method variants being called on the SafeProcess instance.
438
439The variant you choose depends on whether the thread that invokes cancel on the SafeProcess is an EventDispatchThread (a GUI thread), or else, whether you are willing to wait until SafeProcess is interruptible. If you're calling the cancel method from a GUI thread or otherwise don't care to wait, call the zero-argument variant. Otherwise, you can call the single argument version and pass in true for forceWaitUntilInterruptible.
440
441
442Usage:
443- Usually, there is no need to bother waiting for the uninterruptible phase to finish, so you can call the zero-argument version of cancelRunningProcess(), and continue as before.
444
445- If you're calling from a GUI thread and want to disable your cancel button during the uninterruptible phase, however short this may be, in order to give proper feedback to the user, then you can implement the hooks of the SafeProcess.MainHandler interface as explained in the next section.
446
447- Call the cancelRunningProcess(forceWaitUntilInterruptible=true) variant if you're not calling from a GUI thread and you want to wait until SafeProcess becomes interruptible again (if you happened to have invoked the cancel method during an uninterruptible phase).
448
449
450Further details:
451Calling cancelRunningProcess(forceWaitUntilInterruptible=true) forces the method to only return when the SafeProcess has internally become "interruptible". There is a very brief phase at the start of a SafeProcess' cleanup period that blocks until the worker threads have cleanly terminated. This phase should not be interrupted (as would have happened upon an InterruptedException being sent just then), as the uninterruptible phase happens after the internal process has finished and the process is properly terminating.
452
453If the cancel method happened to be called during this "blocking" phase and if this phase were to theoretically take long and if cancelRunningProcess() were not to return immediately if the calling thread is a GUI thread, then your graphical interface, or at least the cancel button, would have been unresponsive until the cancelRunningProcess() method finally returned. But this uninterruptible phase is short, and it always terminates, so calling cancelRunningProcess() is sufficient. And it always returns immediately if the caller is a GUI thread. The zero-argument variant of cancelRunningProecss() internally calls cancelRunningProcess(forceWaitUntilInterruptible=false) and causes the method to return immediately: either cancelRunningProcess was called when SafeProcess was interruptible and it sends an InterruptedException to the primary thread and returns, or cancelRunningProcess got called when SafeProcess was in an uninterruptible phase. And if the caller was a GUI thread (or if forceWaitUntilInterruptible was set to false), the cancelRunningProcess() method would also return immediately.
454
455
456For EXAMPLES OF USE, see GLI's GShell.java and DownloadJob.java.
457
458_________________________________________________________________
459H. USAGE: IMPLEMENTING HOOKS AROUND CANCELLING OR PROCESS' END
460_________________________________________________________________
461
462- Implement the SafeProcess.MainProcessHandler Interface, see the decription below and the section CUSTOMISING.
463- Write the hooks you want.
464- If you don't really intend to override beforeWaitingForStreamsToEnd(boolean forciblyTerminating) or afterStreamsEnded(boolean forciblyTerminating), then return the boolean parameter forciblyTerminating.
465- For EXAMPLES OF USE, see GLI's DownloadJob.java.
466
467
468The SafeProcess.MainProcessHandler provides hooks into various key stages of the SafeProcess instance's life cycle. Specifically, the clean up and termination stages of its life cycle. Currently, all stages that you can write hooks for come after the internal process has come to an end either naturally or prematurely.
469
470
471If your code needs to do some special processes during any of these stages, override them.
472
473The stages are:
474
475    1. boolean beforeWaitingForStreamsToEnd(boolean forciblyTerminating)
476
477    Called after Process.waitFor() (when the internal Process has naturally or prematurely ended) and before the worker threads are join()ed (when the cleanup phase has begun, which always occurs, regardless of whether the Process had ended naturally or was cancelled). The parameter forciblyTerminating is true if the Process had been cancelled during an interruptible phase, or if an unexpected exception had occurred. In both cases, the internal Process object may need to be destroy()ed. But if the Process had only been cancelled during an uninterruptible phase, it means it had already naturally terminated and has started cleaning up. In such cases, the internal Process object does not need to be destroy()ed and forciblyTerminating will be false.
478
479    If not overriding, return the parameter. If overriding, even if the parameter is true, you may determine the process has come to a natural end after all. In that case, the process need not be destroyed, so you'd return false. For an example, see GLI's DownloadJob.java.
480
481    If you want to disable your cancel button temporarily while the join() is taking place, this is the method to do it.
482
483
484    2. boolean afterStreamsEnded(boolean forciblyTerminating)
485
486    Called after the worker threads are join()ed, which is the start of the clean up phase. This method is also always called and the parameter indicates whether it's so far been determined that the process needs to be destroyed or not. If not overriding, return the parameter.
487
488    If you want to re-enable your cancel button since the join() have by now taken place, this is the method to do so.
489
490
491    3. void beforeProcessDestroy()
492    4. void afterProcessDestroy()
493
494    These two methods are only called if destroy() needed to be called on the internal Process object (if the Process did not come to a natural end for any reason, including being cancelled during an interruptible phase or if an unexpected exception occurred).
495
496
497    5. void doneCleanup(boolean wasForciblyTerminated)
498
499    This method is always called at the very end of the SafeProcess' lifecycle, regardless of whether the internal Process naturally terminated or needed to be destroyed as happens upon premature termination. If the process needed to be destroyed, then the parameter wasForciblyTerminated would be true.
500
501
502___________________________________________________
503I. USEFUL STATIC METHODS AND STATIC INNER CLASSES
504___________________________________________________
505
506public static boolean isAvailable(String programName)
507    - Runs the `which` cmd over the program and returns true or false
508    - `which` is included in winbin for Windows, and is part of unix systems
509
510static public boolean processRunning(Process process)
511    - returns true if the internal Process is currently running and hasn't come to an end. That is, this method returns true if Process.waitFor() is still blocking
512    - If the SafeProcess is in its cleanup/termination phase, then this method will return false, even though SafeProcess is still doing stuff.
513    - If you want to know when SafeProcess has finished in its entirety, call the instance method processRunning() on the SafeProcess object.
514
515public static long getProcessID(Process p)
516    - uses java native access (JNA, version 4.1.0 from 2013). The JNA jar files have been included into gli's lib and GS3's lib.
517    - For more details, see gli/lib/README.txt, section "B. THE jna.jar and jna-platform.jar FILES"
518
519public static void destroyProcess(Process p)
520    - will terminate any subprocesses launched by the Process p
521    - uses java native access to get processid
522    - uses OS system calls to terminate the Process and any subprocesses
523
524public static boolean closeProcess(Process prcs)
525    - will attempt to close your Process iostreams and destroy() the Process object at the end.
526
527public static boolean closeResource(Closeable resourceHandle)
528    - will attempt to cleanly close your resource (file/stream handle), logging on Exception
529
530public static boolean closeSocket(Socket resourceHandle)
531    - will attempt to cleanly close your Socket, logging on Exception.
532    - From Java 7 onwards, calling closeResource() would suffice on Sockets too. But in Java 6, Sockets didn't yet implement Closeable, so this method is to ensure backwards compatibility.
533
534public static class OutputStreamGobbler extends Thread
535    - A class that can write to a Process' inputstream from its own Thread.
536
537public static class InputStreamGobbler extends Thread
538    - Class that can read from a Process' output or error stream in its own Thread.
539    - A separate instance should be created for each stream, so that each stream has its own Thread.
540
541
542
543Package access: static methods to prematurely terminate any process denoted by processID and any subprocesses it may have launched
544
545static void killWinProcessWithID(long processID)
546
547static boolean killUnixProcessWithID(long processID)
548    - for Mac/Linux
549
550
551__________________________________________________
552J. OTHER USEFUL INSTANCE METHODS
553__________________________________________________
554
555public synchronized boolean processRunning()
556    - returns true if the SafeProcess instance has started its internal Process or is still cleaning up on its termination
557
558public boolean cancelRunningProcess()
559public boolean cancelRunningProcess(boolean forceWaitUntilInterruptible)
560    - cancel the SafeProcess instance
561    - untimately synchronized, so threadsafe
562    - returns true if the process were still running so that an interrupt needed to be sent to terminate it, returns false if sending an interrupt was unnecessary because the process had already entered the start of its termination phase when cancel was called. Will also return false if the process was never run by the time cancel was called.
563    - usually you want to call the zero-argument version
564    - See the section CANCELLING for further details
565
566_________________________________________________________________________________________________
567K. LINKS AND NOTES ON PROCESSES, PROCESS STREAMS, INTERRUPTING AND DESTROYING PROCESSES
568_________________________________________________________________________________________________
569
570Processes and streams
571
572- http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html?page=2
573explains why SafeProcess.java is implemented with worker threads to handle a Process' iostreams.
574- http://steveliles.github.io/invoking_processes_from_java.html
575- http://mark.koli.ch/leaky-pipes-remember-to-close-your-streams-when-using-javas-runtimegetruntimeexec
576
577
578Interrupting a Process:
579
580- http://stackoverflow.com/questions/2126997/who-is-calling-the-java-thread-interrupt-method-if-im-not
581- http://stackoverflow.com/questions/13623445/future-cancel-method-is-not-working?noredirect=1&lq=1
582- http://stackoverflow.com/questions/3976344/handling-interruptedexception-in-java
583- http://stackoverflow.com/questions/4906799/why-invoke-thread-currentthread-interrupt-when-catch-any-interruptexception
584- http://stackoverflow.com/questions/4906799/why-invoke-thread-currentthread-interrupt-when-catch-any-interruptexception
585- https://praveer09.github.io/technology/2015/12/06/understanding-thread-interruption-in-java/
586- http://michaelscharf.blogspot.co.nz/2006/09/dont-swallow-interruptedexception-call.html
587
588
589Destroying a process and subprocesses on various OS:
590
591- Need process id, pid, for which Java Native Access libraries (jar files) are required.
592http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program
593- To find the processID of the process launched by SafeProcess, need to use Java Native Access (JNA) jars, available jna.jar and jna-platform.jar.
594    http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program
595    http://stackoverflow.com/questions/35842/how-can-a-java-program-get-its-own-process-id
596    http://www.golesny.de/p/code/javagetpid
597    https://github.com/java-native-access/jna/blob/master/www/GettingStarted.md
598    We're using JNA v 4.1.0, downloaded from https://mvnrepository.com/artifact/net.java.dev.jna/jna
599
600WINDOWS NOTES:
601- Can't artificially send Ctrl-C: stackoverflow.com/questions/1835885/send-ctrl-c-to-process-open-by-java
602
603 On Windows, p.destroy() terminates process p that Java launched,
604 but does not terminate any processes that p may have launched. Presumably since they didn't form a proper process tree.
605    https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/e3cb7532-87f6-4ae3-9d80-a3afc8b9d437/how-to-kill-a-process-tree-in-cc-on-windows-platform?forum=vclanguage
606    https://msdn.microsoft.com/en-us/library/windows/desktop/ms684161(v=vs.85).aspx
607
608 Searching for: "forcibly terminate external process launched by Java on Windows"   
609 Not possible: stackoverflow.com/questions/1835885/send-ctrl-c-to-process-open-by-java
610 But can use taskkill or tskill or wmic commands to terminate a process by processID
611 stackoverflow.com/questions/912889/how-to-send-interrupt-key-sequence-to-a-java-process
612 Taskkill command can kill by Image Name, such as all running perl, e.g. taskkill /f /im perl.exe
613 But what if we kill perl instances not launched by GS?
614    /f Specifies to forcefully terminate the process(es). We need this flag switched on to kill childprocesses.
615    /t Terminates the specified process and any child processes which were started by it.
616            /t didn't work to terminate subprocesses. Maybe since the process wasn't launched as
617            a properly constructed processtree.
618    /im is the image name (the name of the program), see Image Name column in Win Task Manager.
619   
620 We don't want to kill all perl running processes.
621 Another option is to use wmic, available since Windows XP, to kill a process based on its command
622 which we sort of know (SafeProcess.command) and which can be seen in TaskManager under the
623 "Command Line" column of the Processes tab.
624    https://superuser.com/questions/52159/kill-a-process-with-a-specific-command-line-from-command-line
625 The following works kill any Command Line that matches -site localsite lucene-jdbm-demo
626    C:>wmic PATH win32_process Where "CommandLine like '%-site%localsite%%lucene-jdbm-demo%'" Call Terminate
627 "WMIC Wildcard Search using 'like' and %"
628    https://codeslammer.wordpress.com/2009/02/21/wmic-wildcard-search-using-like-and/
629 However, we're not even guaranteed that every perl command GS launches will contain the collection name
630 Nor do we want to kill all perl processes that GS launches with bin\windows\perl\bin\perl, though this works:
631    wmic PATH win32_process Where "CommandLine like '%bin%windows%perl%bin%perl%'" Call Terminate
632 The above could kill GS perl processes we don't intend to terminate, as they're not spawned by the particular
633 Process we're trying to terminate from the root down.
634   
635 Solution: We can use taskkill or the longstanding tskill or wmic to kill a process by ID. Since we can
636 kill an external process that SafeProcess launched OK, and only have trouble killing any child processes
637 it launched, we need to know the pids of the child processes.
638 
639 We can use Windows' wmic to discover the childpids of a process whose id we know.
640 And we can use JNA to get the process ID of the external process that SafeProcess launched.
641 
642 See links further above on how to find the processID of the process launched by SafeProcess on various OS,
643 and where to find the JNA jars needed for this.
644 
645 WMIC can show us a list of parent process id and process id of running processes, and then we can
646 kill those child processes with a specific process id.
647    https://superuser.com/questions/851692/track-which-program-launches-a-certain-process
648    http://stackoverflow.com/questions/7486717/finding-parent-process-id-on-windows
649 WMIC can get us the pids of all childprocesses launched by parent process denoted by parent pid.
650 And vice versa:
651    if you know the parent pid and want to know all the pids of the child processes spawned:
652        wmic process where (parentprocessid=596) get processid
653    if you know a child process id and want to know the parent's id:
654        wmic process where (processid=180) get parentprocessid
655 
656 The above is the current solution.
657 
658 Linux ps equivalent on Windows is "tasklist", see
659    http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program
660
661
662MAC/UNIX NOTES:
663- Kill signals, their names and numerical equivalents: http://www.faqs.org/qa/qa-831.html
664- Killing a processtree (a process group) on Unix:
665https://stackoverflow.com/questions/8533377/why-child-process-still-alive-after-parent-process-was-killed-in-linux
666Works on Linux but not Mac: kill -TERM -pid
667Works on Mac but not Linux: pkill -TERM -P pid did work
668- https://stackoverflow.com/questions/28332888/return-value-of-kill
669    "kill returns an exit code of 0 (true) if the process still existed it and was killed.
670    kill returns an exit code of 1 (false) if the kill failed, probably because the process was no longer running."
671- https://superuser.com/questions/343031/sigterm-with-a-keyboard-shortcut
672Ctrl-C sends a SIGNINT, not SIGTERM or SIGKILL. And on Ctrl-C, "the signal is sent to the foreground *process group*."
673- https://linux.die.net/man/1/kill (manual)
674- https://unix.stackexchange.com/questions/117227/why-pidof-and-pgrep-are-behaving-differently
675- https://unix.stackexchange.com/questions/67635/elegantly-get-list-of-children-processes
676- https://stackoverflow.com/questions/994033/mac-os-x-quickest-way-to-kill-quit-an-entire-process-tree-from-within-a-cocoa-a
677- https://unix.stackexchange.com/questions/132224/is-it-possible-to-get-process-group-id-from-proc
678- https://unix.stackexchange.com/questions/99112/default-exit-code-when-process-is-terminated
679
680_______________________________________________________________________________________
681L. USEFUL SYNCHRONISATION NOTES AND SUGGESTED READING ON THREADS AND THREAD SAFETY
682_______________________________________________________________________________________
683
684Things worth reading on Concurrency and Thread safety:
685
686- Deitel and Deitel's newer editions of their Java books have updated their chapter on Concurrency.
687- http://docs.oracle.com/javase/tutorial/uiswing/concurrency/
688series of Java articles on concurrency again.
689- https://docs.oracle.com/javase/tutorial/essential/concurrency/atomicvars.html
690On "primitive" like variables that provide instrinsic locks.
691
692
693Notes on synchronization:
694
695- You can declare methods are synchronized. For example,
696    public synchronized void doStuff()
697This implicitly synchronizes on the *instance* of the class on which this method is called.
698
699- Within any methods, you can synchronize on specific objects to just lock on them. You have a synchronized code block within which you can manipulate the object.
700
701Keep synchronized blocks and methods as short as you can.
702There are some subtle issues related to using synchronized methods, as explained below, that can result in deadlocks.
703
704
705More reading:
706
707- http://stackoverflow.com/questions/574240/is-there-an-advantage-to-use-a-synchronized-method-instead-of-a-synchronized-blo
708
709       "Not only do synchronized methods not lock the whole class, but they don't lock the whole instance either. Unsynchronized methods in the class may still proceed on the instance."
710       "Only the syncronized methods are locked. If there are fields you use within synced methods that are accessed by unsynced methods, you can run into race conditions."
711       
712       "synchronizing on "this" is considered in some circles to be an anti-pattern. The unintended consequence is that outside of the class someone can lock on an object reference that is equal to "this" and prevent other threads from passing the barriers within the class potentially creating a deadlock situation. Creating a "private final Object = new Object();" variable purely for locking purposes is the often used solution. Here's another question relating directly to this issue.
713
714- http://stackoverflow.com/questions/442564/avoid-synchronizedthis-in-java?lq=1"
715
716       "A private lock is a defensive mechanism, which is never a bad idea.
717
718       Also, as you alluded to, private locks can control granularity. One set of operations on an object might be totally unrelated to another but synchronized(this) will mutually exclude access to all of them."
719
720
721- http://stackoverflow.com/questions/8393883/is-synchronized-keyword-exception-safe
722     "In any scoped thread-safe block, the moment you get out of it, the thread-safety is gone."
723     "In case of an exception the lock will be released."
724
725- http://stackoverflow.com/questions/8259479/should-i-synchronize-listener-notifications-or-not
726     "Use a CopyOnWriteArrayList for your listener arrays."
727     "If you use the CopyOnWriteArrayList, then you don't have to synchronize when iterating."
728     "CopyOnWriteArrayList is thread-safe, so there is no need to synchronize."
729
730     "Use a ConcurrentLinkedQueue<Listener> ... for this kind of problems: adding, removing and iterating simultaneously on a collection.
731     A precision : this solution prevents a listener from being called from the very moment it is deregistered."
732     "It means that you start iterating, an element is added, it will be called, another is removed, it won't, all this in the same iteration cycle.
733     It's the best of both world: ensuring synchronization, while being fine grained on who gets called and who's not."
734
735Examples of using CopyOnWriteArrayList: in GLI's shell/GShell.java and GS3's gsdl3/build/CollectionConstructor.java
736
737- http://stackoverflow.com/questions/8260205/when-a-listener-is-removed-is-it-okay-that-the-event-be-called-on-that-listener
738
739- http://stackoverflow.com/questions/2282166/java-synchronizing-on-primitives
740     
741     1. You can't lock on a primitive and
742     2. Don't lock on a Long unless you're careful how you construct them. Long values created by autoboxing or Long.valueOf() in a certain range are guaranteed to be the same across the JVM which means other threads could be locking on the same exact Long object and giving you cross-talk. This can be a subtle concurrency bug (similar to locking on intern'ed strings).
743
744     Cross-talk:
745     "In electronics, crosstalk is any phenomenon by which a signal transmitted on one circuit or channel of a transmission system creates an undesired effect in another circuit or channel. Crosstalk is usually caused by undesired capacitive, inductive, or conductive coupling from one circuit, part of a circuit, or channel, to another."
746
747
748__________________________________________________
749M. TICKETS, COMMITS, TAGS
750__________________________________________________
751
752The following was already documented in ticket http://trac.greenstone.org/ticket/895
753
754    Other Java classes of GLI and GS3 src code were shifted from using Java's Process class directly to using our new SafeProcess class.
755
756    GLI src code classes that were changed:
757    /Scratch/ak19/gs3-svn-15Nov2016/gli>fgrep -rl "Process" src
758
759    + src/org/greenstone/gatherer/util/SafeProcess.java
760    + src/org/greenstone/gatherer/gui/FormatConversionDialog.java
761    + src/org/greenstone/gatherer/gui/DownloadPane.java
762    + src/org/greenstone/gatherer/util/GS3ServerThread.java
763    + 4 src/org/greenstone/gatherer/Gatherer.java [All 4 instances that used Process now go through SafeProcess instead]
764    + 1 src/org/greenstone/gatherer/download/ServerInfoDialog.java
765    + 2 src/org/greenstone/gatherer/greenstone/Classifiers.java
766    + 2 src/org/greenstone/gatherer/greenstone/Plugins.java
767    + 1 src/org/greenstone/gatherer/collection/ScriptOptions.java
768    + !! src/org/greenstone/gatherer/util/ExternalProgram.java && file/FileAssociationManager.java
769
770    After updating GS3 src code to using SafeProcess, returned to GLI:
771    - VERY HARD:
772     + 1 src/org/greenstone/gatherer/shell/GShell.java
773    - VERY, VERY HARD:
774     + 2 src/org/greenstone/gatherer/download/DownloadJob.java
775
776
777    GS3 src code classes that were changed:
778    > fgrep -r "Process" . | grep -v ".svn" | grep -v "~" | less
779
780    + 2 ./java/org/greenstone/admin/guiext/Command.java
781    X ./java/org/greenstone/gsdl3/util/Processing.java
782    + MANY OCCURRENCES ./java/org/greenstone/gsdl3/service/MapRetrieve.java
783    + Uses Processing 2 times: ./java/org/greenstone/gsdl3/util/GDBMWrapper.java
784    + ./java/org/greenstone/util/BrowserLauncher.java
785    + ./java/org/greenstone/util/RunTarget.java
786
787    In the GLI and GS3 src files modified to use SafeProcess above, only the modifications to GS3's Command.java, MapRetrieve.java and GDMBWrapper.java are untested. All others have been tested, at least on Linux. Many have also been tested on Windows.
788
789    Tested GLI's collection building (uses GShell.java's modifications) and DownloadJob.java on all 3 OS. Both have a Cancel button that should cancel any processes and subprocesses launched. Before the modifications, subprocesses would still be running on Windows and during (full-)import on the Mac when cancel was pressed. But now have added code to SafeProcess to issue OS specific system calls to terminate the process and its subprocesses successfully on Windows and Mac too.
790
791    Also tested GS3 user comments and the online GS3's doc editor's meta editing followed by rebuilding. All worked successfully on Linux, Windows and Mac with the modifications to use SafeProcess.
792
793    The modifications to GLI started with GS3ServerThread.java and other *easily* changed classes in
794    - http://trac.greenstone.org/changeset/31582 (beginning of GLI modification)
795    - until http://trac.greenstone.org/changeset/31665 (GS3 src code modification).
796
797    The version of GS3 and GLI Java code before the GS3 src was modified is at
798    - http://trac.greenstone.org/browser/main/tags/GS3-src-SafeProcess
799
800    Next, GLI's GShell.java and any associated files were modified along with SafeProcess, to support cancelling.
801
802    Then, around 22 May, GLI's DownloadJob.java and any associated class files were modified along with SafeProcess to ensure subprocesses were safely terminated on Windows when the download action was cancelled OR when building (GShell.java) was cancelled. The Mac modifications to SafeProcess to ensure subprocesses were terminated was finished on 26 May 2017.
803
804    The tagged version of the GS3 and GLI Java code for when all the changes related to using SafeProcess were made to GS3 and GLI code is at
805    - http://trac.greenstone.org/browser/main/tags/UsingSafeProcess
806
807
808
809More changesets:
810
811http://trac.greenstone.org/changeset/31711 to http://trac.greenstone.org/changeset/31718
812
813These changesets involved:
814
815- a README on SafeProcess?
816
817- the remaining 2 instances of using Runtime.exec/Java Process in Gatherer have been ported to using SafeProcess?. These instances were of the inner classes Gatherer.ExternalApplication? and Gatherer.BrowserApplication?, both of which needed SafeProcess? to have the cancel functionality before they could be ported to use SafeProcess?. Tested the modified inner classes on Linux.
818
819When tested on Mac, I found that with or without these new changes, GLI does not wait to exit if external applications are still running. On Linux (and the way GLI is coded), GLI's GUI may disappear, but the overall GLI process still waits for external processes launched via GLI to have exited before GLI exits. This doesn't happen on Mac with either the original or the modified code. Worth investigating. But at least the modified code has not broken something that used to work.
820
821
822_________________________________________________________
823N. GLI vs GS3 CODE's SAFEPROCESS.JAVA - THE DIFFERENCES
824_________________________________________________________
825
826- package name
827    GLI: package org.greenstone.gatherer.util;
828    GS3: package org.greenstone.util;
829- imports:
830    GLI: import org.greenstone.gatherer.DebugStream;
831- GS3 uses Misc.java and GLI uses Utility.java for determining the OS and the appropriate newline character representation for the OS.
832- GS3 uses its Logger object for logging messages, GLI uses System.err (Debugstream code is presently commented out).
833
834
835    $ diff gli/src/org/greenstone/gatherer/util/SafeProcess.java src/java/org/greenstone/util/SafeProcess.java
836    -----------------------------------------------------------------------------------------------------------
837    1c1
838    < package org.greenstone.gatherer.util;
839    ---
840    > package org.greenstone.util;
841    27c27
842    < import org.greenstone.gatherer.DebugStream;
843    ---
844    > //import org.greenstone.gatherer.DebugStream;
845    56c56
846    <     ///static Logger logger = Logger.getLogger(org.greenstone.util.SafeProcess.class.getName());
847    ---
848    >     static Logger logger = Logger.getLogger(org.greenstone.util.SafeProcess.class.getName());
849    153a154
850    >
851    816c817
852    <   if(Utility.isMac()) {
853    ---
854    >   if(Misc.isMac()) {
855    880c881
856    <     if(Utility.isWindows()) {
857    ---
858    >     if(Misc.isWindows()) {
859    909c910
860    <   if(!Utility.isMac() && canSkipExtraWorkIfLinux) {
861    ---
862    >   if(!Misc.isMac() && canSkipExtraWorkIfLinux) {
863    1284c1285
864    <           outputstr.append(Utility.NEWLINE); // "\n" is system dependent (Win must be "\r\n")
865    ---
866    >           outputstr.append(Misc.NEWLINE); // "\n" is system dependent (Win must be "\r\n")
867    1381c1382
868    <       /*if(Utility.isWindows()) {
869    ---
870    >       /*if(Misc.isWindows()) {
871    1416c1417
872    <   //logger.info(msg);
873    ---
874    >   logger.info(msg);
875    1418c1419
876    <   System.err.println(msg);
877    ---
878    >   //System.err.println(msg);
879    1424c1425
880    <   //logger.error(msg, e);
881    ---
882    >   logger.error(msg, e);
883    1426,1427c1427,1428
884    <   System.err.println(msg);
885    <   e.printStackTrace();
886    ---
887    >   //System.err.println(msg);
888    >   //e.printStackTrace();
889    1434c1435
890    <   //logger.error(e);
891    ---
892    >   logger.error(e);
893    1436c1437
894    <   e.printStackTrace();
895    ---
896    >   //e.printStackTrace();
897
898    -----------------------------------------------------------------------------------------------------------
899
900
901__________________________________________________
902O. FUTURE WORK, IMPROVEMENTS
903__________________________________________________
904
905- On Windows, Perl could launch processes as proper ProcessTrees: http://search.cpan.org/~gsar/libwin32-0.191/
906Then killing the root process will kill child processes naturally, and we won't need to call OS commands to terminate a process.
907- Need to find a Mac (or Unix) equivalent way creating Process Tree in Perl too.
908
909Eventually, instead of running a windows/mac or even linux command to kill the process ourselves, investigate if it is useful to change over to use
910    https://github.com/flapdoodle-oss/de.flapdoodle.embed.process/blob/master/src/main/java/de/flapdoodle/embed/process/runtime/Processes.java
911- mentioned by http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program
912- works with Apache license, http://www.apache.org/licenses/LICENSE-2.0
913- This is a Java class that uses JNA to terminate processes. It also has the getProcessID() method.
914- However, does it kill subprocesses? (The OS specific commands presently issued in SafeProcess.destroyProcess() ensure subprocesses are killed.)
915
916
917Currently, SafeProcess does Java things the way that older Java versions recognise and accept, not only for backwards compatibility with earlier versions of Java, but in order to minimise the number of drastic changes from porting the existing Greenstone GLI and GS3 code to using SafeProcess.
918
919- In future, think of changing the method doRuntimeExec() over to using ProcessBuilder, instead of Runtime.exec(). ProcessBuilder seems to have been introduced from Java 5.
920https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
921See also https://zeroturnaround.com/rebellabs/how-to-deal-with-subprocesses-in-java/
922which suggests using Apache Common Exec to launch processes and says what will be forthcoming in Java 9
923
924- Instead of creating and running Threads the usual way, should we use features of newer Java versions like the Concurrency API and Executor which uses Threadpools, and let Executor handle these details? Improvements would be with multicore systems
925https://stackoverflow.com/questions/3330430/does-java-have-support-for-multicore-processors-parallel-processing
926
927
928
929_____________________________________________________________________________________________________________________
930
931P. ALTERNATIVE FUTURE WORK: A case for rewriting SafeProcess to use IO::Select (Selector in Java) instead of multi-threading
932_____________________________________________________________________________________________________________________
933
934(a) STREAM GOBBLER **THREADS** ARE ALLOWED TO BLOCK (JVM WILL TIMESLICE). CAUSE INTERRUPTEDEXCEPTIONS TO END BLOCKS
935In the InputStreamGobbler Thread classes for handling the proc.errstream and proc.outstream, the run() methods have a loop on readLine():
936
937   while (!this.interrupted() && (line = br.readLine()) != null) { ... }
938
939This so far worked beautifully, since if a process was cancelled, SafeProcess would call gobbler.interrupt() and the above loop would terminate. If a process had ended and the process err and output streams therefore run out, line == null and the loop would terminate. In both cases, we're out of the while loop and the process' stream would have been proeprly handed and the clean up would have taken place.
940
941However, BufferedReader.read() and consequently its readLine() are actually blocking calls. This was discovered when the OpenOfficeExtension was included in a GS installation and didn't work properly: if it fails to launch soffice --headless to accept on some port, then soffice is not running headless in the background. At this point, the stderr stream of the perl process that launched the OpenOfficePlugin (and OpenOfficeConverter) worked fine and output the pluginfo for the plugin and finished, terminating the while loop for the err stream gobbler. However, the process' output stream gobbler never got any data, including eof(). No cancel button was pressed, so this.interrupted() wasn't true. And br.readLine() wouldn't return but blocked: it waited for data that never came. This blocked SafeProcess' call to join() on that gobbler thread, as its run() method never terminated.
942
943The first solution is checking bufferedReader.ready() BEFORE attempting to any read() operation on the BufferedReader:
944
945     // readLine() can block if nothing is forthcoming, including eof marker.
946     // So must check BufferedReader.ready() before attempting readLine() on it
947     // See https://stackoverflow.com/questions/15521352/bufferedreader-readline-blocks
948     // (also http://www.java-gaming.org/index.php?topic=37191.0)
949     while ( !this.isInterrupted() && br.ready() && (line = br.readLine()) != null ) { ...
950
951
952HOWEVER, there are 2 problems with this:
953
9541. As per the API, ready() returns "True if the next read() is guaranteed not to block for input, false otherwise. Note that returning false does not guarantee that the next read will block."
955
956   NOTE: As per Andrew Mackintosh the above does not mean what I understood it to mean. I thought it means that if a read() blocks at this moment and therefore ready() returns false now, it need not meant it will block 5 mins from now if there was finally data sent to the process' stdout 5 mins from now. Andrew says that what the above means is that if ready() returns false a read() operation may OR may not block at present, whereas if ready() returns true, doing read() is guaranteed to NOT block.
957
9582. In the example of the OpenOfficeExtension, what happened is that the process' exit value was always 141, non-zero hence not a success. 141 signifies a SIGPIPE error:
959
960        // NOTE: exitValue can be 141 when we got nothing not even eof from any of the process'
961        // output streams. According to http://tldp.org/LDP/abs/html/exitcodes.html
962        // an exit value of 128+n means Fatal error signal "n". 128 + 13 = 141, so n = 13.
963        // And "Signal 13 is SIGPIPE" see https://github.com/Tunnelblick/Tunnelblick/issues/272
964        // 'SIGPIPE is the "broken pipe" signal, which is sent to a process when it attempts
965        // to write to a pipe whose read end has closed (or when it attempts to write to a
966        // socket that is no longer open for reading), but not vice versa. The default action
967        // is to terminate the process.' as per https://www.quora.com/What-are-SIGPIPEs   
968       
969
970Having realised that br.read() itself can block and how to get such blocking gobbler threads to terminate, I discussed the problem with Andrew. He explained the solution:
971
972- blocking IO calls in loops is FINE as long as they are in a separate thread and not the main thread: the OS, or the JVM in this case rather, will know to use timeslicing and set the processor on other tasks until blocking IO calls are ready again.
973
974- to "wake" up blocking calls on process termination, the "Scheduler" parent class that launched the child process and the StreamGobbler Threads should cause InterruptedExceptions on its StreamGobbler threads once the child process has exited. The Scheduler should internally induce InterruptedExceptions on the StreamGobblers regardless of whether the process terminated naturally or was cancelled by the user. The Exception would then end the blocking br.readLine().
975
976In my case, the Scheduler is SafeProcess.java and is the interrupting Thread that is to send interrupted exceptions. So after doing
977  exitVal = proc.waitFor()
978the scheduler class SafeProcess should somehow cause InterruptedExceptions on its gobbler threads.
979
980I found these pages, which seem to imply indicate that the interrupting thread (SafeProcess in my case) should close the streams of the child process and that this will generate the necessary InterruptedExceptions stopping the read()/readLine() operations:
981
982- https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline (example speaks of bufferedreaders reading from sockets rather than from a process' IO streams)
983- https://arstechnica.com/civis/viewtopic.php?t=259678
984
985I can't get it to work yet though, as at 1 and 2 Aug 2018.
986
987
988(b) ANOTHER IMPORTANT LESSON FROM ANDREW:
989Andrew Mackintosh further said that the read() block calls in worker Threads are far better in terms of not wasting processor power than a constantly active while loop in the worker Thread.
9901. So doing something like this:
991
992    while(!thisthread.isInterrupted() && br.read() ) // br.read() possibly blocks
993
994is good since if the read() the blocks, the OS or JVM knows to reuse the processor for other tasks. (I think of it as the OS/JVM considers blocking calls as being idle processes and therefore puts other tasks onto processor in the meantime.)
995
996Whereas:
997   while(!thisthread.isInterrupted()) {
998      if(!br.ready) continue; // will keep testing interrupted and ready() and so keeps using the processor.
999      else br.read();
1000   }
1001will keep the processor/core wastefully occupied even though there may be long stretches when there's no data to read.
1002
1003
1004UNFORTUNATELY,
1005https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline
1006says that BufferedReader.readLine() can't even be interrupted. And their solution doesn't work either.
1007
1008Other links:
1009- EOF/EOS (End of Stream) is indicated by a return value of -1 for read and null for readLine().
1010- https://stackoverflow.com/questions/36569875/how-does-bufferedreader-readline-handle-eof-or-slow-input
1011- https://stackoverflow.com/questions/3714090/how-to-see-if-a-reader-is-at-eof
1012- https://arstechnica.com/civis/viewtopic.php?t=259678
1013
1014(c) I next tried reading a char at a time by calling BufferedReader.read(), but it blocked forever. With further testing, I confirmed that for the indefinitely blocking process stream (stdout in the use case where things went wrong), the bufferedreader and underlying stream is never ready().
1015
1016Andrew then pointed me to the API for the BufferedReader.read(cbuf) variant at https://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html#read(char[],%20int,%20int), which like the read() variant would return -1 on eof/eos. The API for read(cbuf) claimed that it would return the number of bytes read, -1 if eof/eos or return when ready() returns false. The latter was not true, and this overloaded method behaved the same as the read() variant that reads one char at a time: the buffer and underlying stream were never ready() and hence all variants of BufferedReader's reading methods always blocked.
1017
1018Following advice at https://www.experts-exchange.com/questions/20083639/BufferedReader-not-ready.html and elsewhere, I tried reading directly from InputStream a.o.t. going through BufferedReader, and tested the InputStream.available() (equivalent of ready()) before attempting to read() from the stream,
1019
1020- https://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html#available()
1021- https://stackoverflow.com/questions/3695372/what-does-inputstream-available-do-in-java
1022
1023(Further, the example of InterruptibleReadline at https://stackoverflow.com/questions/3595926/how-to-interrupt-bufferedreaders-readline is buggy and doesn't work when corrected: it's the same problem.)
1024
1025
1026I finally came to the realisation our threaded SafeProcess would never work, and the references on the web to using java.nio when you want to do non-blocking IO started to make sense.
1027I had considered and attempted setting a timeout to help detect when a stream might block forever, versus when a stream would eventually get data. The classes of java.nio were again recommended when you want to work with timeouts during IO operations.
1028
1029
1030(d) java.nio - stands for New IO
1031Thinking it stood for non-blocking IO, I googled for:
1032   java nio non blocking io
1033to confirm whether this was so, and not that nio stood for "native IO". The first search result was:
1034
1035"Blocking vs. Non-blocking IO. Java IO's various streams are blocking. That means, that when a thread invokes a read() or write() , that thread is blocked until there is some data to read, or the data is fully written. The thread can do nothing else in the meantime. Jun 23, 2014"
1036Java NIO vs. IO - Jenkov Tutorials
1037tutorials.jenkov.com/java-nio/nio-vs-io.html"
1038
1039If only I had found information like this earlier. However, I may not have understood it or its significance at the time (before working on NetStinky).
1040- https://www.experts-exchange.com/questions/20083639/BufferedReader-not-ready.html
1041- https://www.quora.com/How-can-I-get-BufferedReader-to-only-wait-for-an-input-from-System-in-for-a-given-number-of-milliseconds
1042
1043There is a Selector class in Java.nio which works like perl and C++ IO::Select().
1044
1045
1046Tutorials and further pages to read:
1047- http://tutorials.jenkov.com/java-nio/nio-vs-io.html (general comparison and when to use java.io vs java.nio)
1048- http://www.baeldung.com/java-nio-selector (the most useful for us)
1049
1050
1051Bookmarked:
1052- https://stackoverflow.com/questions/6619516/using-filechannel-to-write-any-inputstream
1053says that to get a channel for an inputstream, you do:
1054    Channels.newChannel(InputStream in)
1055    http://docs.oracle.com/javase/7/docs/api/java/nio/channels/Channels.html
1056- More information on channels: http://www.java2s.com/Tutorials/Java/Java_io/0930__Java_nio_Channels.htm   
1057- Different kind of buffers: http://www.ntu.edu.sg/home/ehchua/programming/java/J5b_IO_advanced.html
1058- https://docs.oracle.com/javase/7/docs/api/java/nio/channels/InterruptibleChannel.html (we want to use interruptible channels, so our channel must implement this)
1059- https://docs.oracle.com/javase/7/docs/api/java/nio/channels/spi/AbstractSelectableChannel.html (we want to add our channel to a select, so our channel must implement this)
1060- https://docs.oracle.com/javase/7/docs/api/java/nio/channels/Channels.html (factory for obtaining a channel)
1061- https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ReadableByteChannel.html (object returned by doing Channels.newChannel(InputStream in) is of type ReadableByteChannel)
1062
1063I think Selector is the way to go to reimplement SafeProcess using Select to do non-blocking IO (with built in timeouts) in place of multi-threading. Therefore read through http://www.baeldung.com/java-nio-selector to refresh memory on behaviour of IO::select() in general and learn about how to do it in Java in particular.
1064
1065The idea is to move the multi-threaded SafeProcess to SafeProcess2.java and re-write SafeProces.java to use Java's IO::Select() (Selector). Then the classes that use SafeProcess would make the same calls as before. If we ever do go this route, see Section Q below for the public API of SafeProcess that needs to be reimplemented with java.nio/Selector.
1066
1067
1068General information:
1069https://stackoverflow.com/questions/355089/difference-between-stringbuilder-and-stringbuffer
1070
1071
1072(e) Dr Bainbridge fixed the problem at the source: in the OpenOfficeConverter.pm which was launching soffice in headless mode with the wrong command: when 2>&1 is further redirected to a file (or somewhere else like /dev/null), need to do the file redirect first and then 2>&1.
1073
1074So, "cmd >file 2>&1" is the right way, whereas "cmd 2>&1 >file" is the wrong way around. OpenOfficeConverter.pm was launching soffice the wrong way around and Dr Bainbridge fixed it.
1075
1076He says we don't need to pursue rewriting SafeProcess to use Selector, as the issue in GLI was ultimately due to an error in the command that was launched in perl. However, Dr Bainbridge said it was OK to commit the links I've found on using Selector and a description of the issue that led to considering Selector.
1077
1078
1079
1080_____________________________________________________________
1081Q. PUBLIC API OF SAFEPROCESS, ITS SUB CLASSES AND INTERFACES
1082_____________________________________________________________
1083WE ONLY CARE ABOUT NON-GOBBLER CLASSES (SINCE STREAMGOBBLERS ARE THREAD SPECIFIC)
1084
1085public class SafeProcess
1086    public static int DEBUG = 1;
1087
1088    public static final int STDERR = 0;
1089    public static final int STDOUT = 1;
1090    public static final int STDIN = 2;
1091    public static String WIN_KILL_CMD;
1092    public Boolean interruptible = Boolean.TRUE;
1093
1094
1095 public SafeProcess(String[] cmd_args)
1096 public SafeProcess(String cmdStr)
1097 public SafeProcess(String[] cmd_args, String[] envparams, File launchDir)
1098 
1099
1100    public String getStdOutput() { return outputStr; }
1101    public String getStdError() { return errorStr; }
1102    public int getExitValue() { return exitValue; }
1103
1104 
1105    public void setInputString(String sendStr)
1106    public void setExceptionHandler(ExceptionHandler exception_handler)
1107    public void setMainHandler(MainProcessHandler handler)
1108
1109   
1110    public void setSplitStdOutputNewLines(boolean split)
1111    public void setSplitStdErrorNewLines(boolean split)
1112
1113    public boolean cancelRunningProcess() // calls cancelRunningProcess(!forceWaitUntilInterruptible);
1114    public synchronized boolean cancelRunningProcess(boolean forceWaitUntilInterruptible)
1115
1116
1117    public synchronized boolean processRunning()
1118
1119    public int runBasicProcess()
1120    public int runProcess()
1121    public int runProcess(CustomProcessHandler procInHandler,
1122               CustomProcessHandler procOutHandler,
1123               CustomProcessHandler procErrHandler)
1124    public int runProcess(LineByLineHandler outLineByLineHandler, LineByLineHandler errLineByLineHandler)
1125
1126
1127
1128// KEEP AS IS:
1129    public static long getProcessID(Process p)
1130    static void killWinProcessWithID(long processID)
1131    static boolean killUnixProcessWithID(long processID, boolean force, boolean killEntireTree)
1132    public static void destroyProcess(Process p)
1133
1134    // LOGGING - KEEP
1135    public static void log(String msg)
1136    public static void log(String msg, Exception e)
1137    public static void log(Exception e)
1138    public static void log(String msg, Exception e, boolean printStackTrace)
1139    public static String streamToString(int src)
1140
1141    // UTILITY FUNCTIONS - MOVE OUT OF CLASS INTO PACKAGE WHEN WE HAVE SAFEPROCESS AND SAFEPROCESS2
1142public static boolean closeResource(Closeable resourceHandle)
1143public static boolean closeSocket(Socket resourceHandle)
1144public static boolean closeProcess(Process prcs)
1145static public boolean processRunning(Process process)
1146
1147// Uses SafeProcess, but once that is working the new way can KEEP AS IS:
1148    public static boolean isAvailable(String program)
1149   
1150
1151INTERNAL INTERFACES AND ABSTRACT CLASSES:
1152
1153public static interface ExceptionHandler
1154    public void gotException(Exception e)
1155
1156public static interface MainProcessHandler
1157    public boolean beforeWaitingForStreamsToEnd(boolean forciblyTerminating);
1158    public boolean afterStreamsEnded(boolean forciblyTerminating);
1159    public void beforeProcessDestroy();
1160    public void afterProcessDestroy();
1161    public void doneCleanup(boolean wasForciblyTerminated);
1162
1163public static abstract class CustomProcessHandler
1164    public String getThreadNamePrefix()
1165    public abstract void run(Closeable stream); //InputStream or OutputStream
1166
1167
1168public static abstract class LineByLineHandler
1169    public String getThreadNamePrefix()
1170    public abstract void gotLine(String line); // first non-null line
1171    public abstract void gotException(Exception e); // for when an exception occurs instead of getting a line
1172
1173
Note: See TracBrowser for help on using the browser.