Changeset 31717 for main

Show
Ignore:
Timestamp:
30.05.2017 21:25:17 (3 years ago)
Author:
ak19
Message:

Finally finished first draft of SafeProcess?.java Readme. Added more sections besides filling in the remaining ones that previously only had section headings.

Files:
1 modified

Legend:

Unmodified
Added
Removed
  • main/trunk/gli/src/org/greenstone/gatherer/util/Readme_Using_SafeProcess.txt

    r31713 r31717  
    1 SAFEPROCESS README 
     1First drafts of this document 
     2ak19, 30 May 2017 
     3 
     4========================================= 
     5    SAFEPROCESS.JAVA README 
     6========================================= 
    27 
    38GS Java developers should use SafeProcess.java in place of directly using Java's Process class, unless any issues are discovered with SafeProcess.java hereafter. 
     
    813- greenstone3/src/java/org/greenstone/util/SafeProcess.java 
    914 
     15 
     16This document contains the following sections 
     17    A. WHY WE SHOULD GO THROUGH SAFEPROCESS INSTEAD OF USING JAVA'S PROCESS DIRECTLY 
     18    B. MODEL OF SAFEPROCESS THREADS AND INTERRUPT BEHAVIOUR 
     19    C. USAGE: DEBUGGING 
     20    D. USAGE: INSTANTIATION & DEFAULT RUNPROCESS VARIANTS 
     21    E. INTERNAL IMPLEMENTATION DETAILS - USEFUL IF CUSTOMISING SAFEPROCESS' BEHAVIOUR OR CALLING CANCEL ON SAFEPROCESS 
     22    F. USAGE: CUSTOMISING 
     23    G. USAGE: CANCELLING 
     24    H. USAGE: IMPLEMENTING HOOKS AROUND CANCELLING OR PROCESS' END 
     25    I. USEFUL STATIC METHODS AND STATIC INNER CLASSES 
     26    J. OTHER USEFUL INSTANCE METHODS 
     27    K. LINKS AND NOTES ON PROCESSES, PROCESS STREAMS, INTERRUPTING AND DESTROYING PROCESSES 
     28    L. USEFUL SYNCHRONISATION NOTES AND SUGGESTED READING ON THREADS AND THREAD SAFETY 
     29    M. TICKETS, COMMITS, TAGS 
     30    N. GLI vs GS3 CODE's SAFEPROCESS.JAVA - THE DIFFERENCES 
     31    O. FUTURE WORK, IMPROVEMENTS 
     32 
    1033_______________________________________________________________________________ 
    11 WHY WE SHOULD GO THROUGH SAFEPROCESS INSTEAD OF USING JAVA'S PROCESS DIRECTLY 
     34A. WHY WE SHOULD GO THROUGH SAFEPROCESS INSTEAD OF USING JAVA'S PROCESS DIRECTLY 
    1235_______________________________________________________________________________ 
    1336 
     
    2043 
    2144 
    22 _____________________________________________________ 
    23 MODEL OF SAFEPROCESS THREADS AND INTERRUPT BEHAVIOUR 
    24 _____________________________________________________ 
     45_________________________________________________________ 
     46B. MODEL OF SAFEPROCESS THREADS AND INTERRUPT BEHAVIOUR 
     47_________________________________________________________ 
    2548 
    2649This section is especially IMPORTANT TO READ IF you're thinking of customising SafeProcess' handling of the internal Process' iostreams, since each of them are always handled in their own distinct worker threads, and you need to remember to deal with ThreadSafety issues. 
     
    5275 
    5376__________________________________________________ 
    54 USAGE: DEBUGGING 
    55 __________________________________________________ 
    56  
    57 There's a lot of info that a SafeProcess instance will log during its execution and termination phases, including Exceptions but also very basic things, such as the command that the SafeProcess instance is running. 
    58  
    59 By default, such debugging-related logging is turned off. You can turn it on by adjusting the static SafeProcess.DEBUG = 1 and recompiling. 
     77C. USAGE: DEBUGGING 
     78__________________________________________________ 
     79 
     80There's a lot of info that a SafeProcess instance will log during its execution and termination phases, including Exceptions but also very basic things, such as the command that the SafeProcess instance is running and other messages. 
     81 
     82Exceptions are always logged, but debugging related messages can be turned off or on. By default, it's turned off. You can turn it on by adjusting the static SafeProcess.DEBUG = 1 and recompiling. 
    6083 
    6184For GS3, the log output uses log4j. 
     
    6487If you're using SafeProcess yourself and wish to log debug statements that are related to how well you're using SafeProcess, you can call one of the static SafeProcess.log() functions yourself. Remember to set the static SafeProcess.DEBUG = 1. 
    6588 
    66 __________________________________________________ 
    67 USAGE: INSTANTIATION & DEFAULT RUNPROCESS VARIANTS 
    68 __________________________________________________ 
     89 
     90There are 4 variants of the log() function: 
     91 
     92    - public static void log(String msg) 
     93    logs the message if debugging is on, DEBUG = 1 
     94 
     95    - public static void log(Exception e) 
     96    logs the stacktrace for the exception, regardless of if debugging is on 
     97 
     98    - public static void log(String msg, Exception e) 
     99    logs the message and logs the stacktrace for the exception, regardless of if debugging is on 
     100 
     101    - public static void log(String msg, Exception e, boolean printStackTrace) 
     102    if debugging is on or if the printStackTrace parameter is true, both the message and the stacktrace for the exception are logged. 
     103    if debugging is off and the printStackTrace parameter is false, nothing is logged. Call this variant if an exception is expected and you only want to log it in debugging mode. An example of an expected exception are some InterruptedExceptions that may occur over the course of SafeProcess 
     104 
     105_______________________________________________________ 
     106D. USAGE: INSTANTIATION & DEFAULT RUNPROCESS VARIANTS 
     107_______________________________________________________ 
    69108 
    70109Usage of the SafeProcess class generally follows the following sequence: 
     
    156195The exit value will also be -1 if you call getExitValue() before setting off the SafeProcess via any of the runProcess() variants. 
    157196 
    158 _________________________________________________________________________________________________________________ 
    159 INTERNAL IMPLEMENTATION DETAILS - USEFUL IF CUSTOMISING SAFEPROCESS' BEHAVIOUR OR CALLING CANCEL ON SAFEPROCESS 
    160 _________________________________________________________________________________________________________________ 
     197_____________________________________________________________________________________________________________________ 
     198E. INTERNAL IMPLEMENTATION DETAILS - USEFUL IF CUSTOMISING SAFEPROCESS' BEHAVIOUR OR CALLING CANCEL ON SAFEPROCESS 
     199_____________________________________________________________________________________________________________________ 
    161200 
    162201This section is for understanding how SafeProcess runs a Java Process internally. These methods are not public and their details are hidden away in the code. 
     
    239278 
    240279__________________________________________________ 
    241 USAGE: CUSTOMISING 
     280F. USAGE: CUSTOMISING 
    242281__________________________________________________ 
    243282 
     
    318357 
    319358If, during the execution of the primary thread, you want to do some complex things during a cancel operation, such as or before and after a process is destroyed, then  
    320 - implement SafeProcess.MainProcessHandler (see below) 
     359- implement SafeProcess.MainProcessHandler. See the Interface definition below. For further details, refer to the sections "CANCELLING" and "IMPLEMENTING HOOKS AROUND CANCELLING OR PROCESS' END". 
    321360- configure your SafeProcess instance by calling setMainHandler(MainProcessHandler mph) on it *before* calling a runProcess() method on that instance. 
    322361- For EXAMPLES OF USE, see GLI's DownloadJob.java. 
     
    370409    } 
    371410 
    372  
    373 __________________________________________________ 
    374 USAGE: CANCELLING 
    375 __________________________________________________ 
     411__________________________________________________ 
     412G. USAGE: CANCELLING 
     413__________________________________________________ 
     414 
     415SafeProcess provides a way to cancel an internal Process that is run by SafeProcess, and which will ensure that the internal process and worker threads all safely terminate and tidily clean up after themselves.  
     416 
     417As explained, SafeProcess internally accomplishes this by sending an InterruptedException on the primary thread that's running the internal Process. An InterruptedException is sent only when SafeProcess is in an "interruptible" phase, as no interruption is necessary at any other time such as cleanup. 
     418 
     419Only a thread *external* to the SafeProcess' primary thread, such as a GUI thread, can call cancelRunningProcess() on a reference to the SafeProcess instance. Two variants are available: 
     420 
     421- boolean cancelRunningProcess() 
     422- boolean cancelRunningProcess(boolean forceWaitUntilInterruptible) 
     423 
     424The boolean value returned is true if an interrupt had to be sent, or false if one didn't need to be sent. An interrupt does not need to be sent if 
     425    - the process was never yet run by the time cancel was called or 
     426    - the internal Process has already naturally terminated and the SafeProcess has moved on to some stage of the cleanup phase by the time cancel was called. 
     427 
     428If you want to inspect the return value, a good variable name is sentInterrupt. For example, 
     429    boolean sentInterrupt = mySafeProc.cancelRunningProcess(); 
     430 
     431 
     432Programming pattern: 
     433A graphical interface may have a Cancel button associated with an actionPerformed() method, which will be run from a GUI/EventDispatchThread when the user presses cancel. Either directly or indirectly, actionPerfomed() during a cancel action needs to lead to one of the cancelRunningProcess() method variants being called on the SafeProcess instance. 
     434 
     435The variant you choose depends on whether the thread that invokes cancel on the SafeProcess is an EventDispatchThread (a GUI thread), or else, whether you are willing to wait until SafeProcess is interruptible. If you're calling the cancel method from a GUI thread or otherwise don't care to wait, call the zero-argument variant. Otherwise, you can call the single argument version and pass in true for forceWaitUntilInterruptible.  
     436 
     437 
     438Usage: 
     439- Usually, there is no need to bother waiting for the uninterruptible phase to finish, so you can call the zero-argument version of cancelRunningProcess(), and continue as before. 
     440 
     441- If you're calling from a GUI thread and want to disable your cancel button during the uninterruptible phase, however short this may be, in order to give proper feedback to the user, then you can implement the hooks of the SafeProcess.MainHandler interface as explained in the next section. 
     442 
     443- Call the cancelRunningProcess(forceWaitUntilInterruptible=true) variant if you're not calling from a GUI thread and you want to wait until SafeProcess becomes interruptible again (if you happened to have invoked the cancel method during an uninterruptible phase). 
     444 
     445 
     446Further details: 
     447Calling cancelRunningProcess(forceWaitUntilInterruptible=true) forces the method to only return when the SafeProcess has internally become "interruptible". There is a very brief phase at the start of a SafeProcess' cleanup period that blocks until the worker threads have cleanly terminated. This phase should not be interrupted (as would have happened upon an InterruptedException being sent just then), as the uninterruptible phase happens after the internal process has finished and the process is properly terminating. 
     448 
     449If the cancel method happened to be called during this "blocking" phase and if this phase were to theoretically take long and if cancelRunningProcess() were not to return immediately if the calling thread is a GUI thread, then your graphical interface, or at least the cancel button, would have been unresponsive until the cancelRunningProcess() method finally returned. But this uninterruptible phase is short, and it always terminates, so calling cancelRunningProcess() is sufficient. And it always returns immediately if the caller is a GUI thread. The zero-argument variant of cancelRunningProecss() internally calls cancelRunningProcess(forceWaitUntilInterruptible=false) and causes the method to return immediately: either cancelRunningProcess was called when SafeProcess was interruptible and it sends an InterruptedException to the primary thread and returns, or cancelRunningProcess got called when SafeProcess was in an uninterruptible phase. And if the caller was a GUI thread (or if forceWaitUntilInterruptible was set to false), the cancelRunningProcess() method would also return immediately. 
     450 
    376451 
    377452For EXAMPLES OF USE, see GLI's GShell.java and DownloadJob.java. 
    378453 
    379 __________________________________________________ 
    380 USAGE: IMPLEMENTING HOOKS AROUND CANCELLING 
    381 __________________________________________________ 
    382  
    383 For EXAMPLES OF USE, see GLI's DownloadJob.java. 
    384  
    385 __________________________________________________ 
    386 USEFUL STATIC METHODS AND STATIC INNER CLASSES 
    387 __________________________________________________ 
     454_________________________________________________________________ 
     455H. USAGE: IMPLEMENTING HOOKS AROUND CANCELLING OR PROCESS' END 
     456_________________________________________________________________ 
     457 
     458- Implement the SafeProcess.MainProcessHandler Interface, see the decription below and the section CUSTOMISING. 
     459- Write the hooks you want. 
     460- If you don't really intend to override beforeWaitingForStreamsToEnd(boolean forciblyTerminating) or afterStreamsEnded(boolean forciblyTerminating), then return the boolean parameter forciblyTerminating. 
     461- For EXAMPLES OF USE, see GLI's DownloadJob.java. 
     462 
     463 
     464The SafeProcess.MainProcessHandler provides hooks into various key stages of the SafeProcess instance's life cycle. Specifically, the clean up and termination stages of its life cycle. Currently, all stages that you can write hooks for come after the internal process has come to an end either naturally or prematurely. 
     465 
     466 
     467If your code needs to do some special processes during any of these stages, override them. 
     468 
     469The stages are: 
     470 
     471    1. boolean beforeWaitingForStreamsToEnd(boolean forciblyTerminating) 
     472 
     473    Called after Process.waitFor() (when the internal Process has naturally or prematurely ended) and before the worker threads are join()ed (when the cleanup phase has begun, which always occurs, regardless of whether the Process had ended naturally or was cancelled). The parameter forciblyTerminating is true if the Process had been cancelled during an interruptible phase, or if an unexpected exception had occurred. In both cases, the internal Process object may need to be destroy()ed. But if the Process had only been cancelled during an uninterruptible phase, it means it had already naturally terminated and has started cleaning up. In such cases, the internal Process object does not need to be destroy()ed and forciblyTerminating will be false. 
     474 
     475    If not overriding, return the parameter. If overriding, even if the parameter is true, you may determine the process has come to a natural end after all. In that case, the process need not be destroyed, so you'd return false. For an example, see GLI's DownloadJob.java. 
     476 
     477    If you want to disable your cancel button temporarily while the join() is taking place, this is the method to do it. 
     478 
     479 
     480    2. boolean afterStreamsEnded(boolean forciblyTerminating) 
     481 
     482    Called after the worker threads are join()ed, which is the start of the clean up phase. This method is also always called and the parameter indicates whether it's so far been determined that the process needs to be destroyed or not. If not overriding, return the parameter. 
     483 
     484    If you want to re-enable your cancel button since the join() have by now taken place, this is the method to do so. 
     485 
     486 
     487    3. void beforeProcessDestroy() 
     488    4. void afterProcessDestroy() 
     489 
     490    These two methods are only called if destroy() needed to be called on the internal Process object (if the Process did not come to a natural end for any reason, including being cancelled during an interruptible phase or if an unexpected exception occurred). 
     491 
     492 
     493    5. void doneCleanup(boolean wasForciblyTerminated) 
     494 
     495    This method is always called at the very end of the SafeProcess' lifecycle, regardless of whether the internal Process naturally terminated or needed to be destroyed as happens upon premature termination. If the process needed to be destroyed, then the parameter wasForciblyTerminated would be true. 
     496 
     497 
     498___________________________________________________ 
     499I. USEFUL STATIC METHODS AND STATIC INNER CLASSES 
     500___________________________________________________ 
    388501 
    389502public static boolean isAvailable(String programName) 
    390503    - Runs the `which` cmd over the program and returns true or false 
    391     `which` is included in winbin for Windows, and is part of unix systems 
     504    - `which` is included in winbin for Windows, and is part of unix systems 
    392505 
    393506static public boolean processRunning(Process process) 
    394     - returns true if the Process is currently running and hasn't come to an end. 
    395     - It does not  
     507    - returns true if the internal Process is currently running and hasn't come to an end. That is, this method returns true if Process.waitFor() is still blocking 
     508    - If the SafeProcess is in its cleanup/termination phase, then this method will return false, even though SafeProcess is still doing stuff. 
     509    - If you want to know when SafeProcess has finished in its entirety, call the instance method processRunning() on the SafeProcess object. 
    396510 
    397511public static long getProcessID(Process p) 
     
    411525 
    412526public static boolean closeSocket(Socket resourceHandle) 
    413     - will attempt to cleanly close your Socket, logging on Exception. In Java 6, Sockets didn't yet implement Closeable, 
    414     so this method is to ensure backwards compatibility 
     527    - will attempt to cleanly close your Socket, logging on Exception. 
     528    - From Java 7 onwards, calling closeResource() would suffice on Sockets too. But in Java 6, Sockets didn't yet implement Closeable, so this method is to ensure backwards compatibility. 
    415529 
    416530public static class OutputStreamGobbler extends Thread 
     
    423537 
    424538 
    425 Package access: 
    426 static methods to prematurely terminate any process denoted by processID and any subprocesses it may have launched: 
    427     static boolean killUnixProcessWithID(long processID) 
    428     static void killWinProcessWithID(long processID) 
    429  
    430  
    431 __________________________________________________ 
    432 OTHER USEFUL INSTANCE METHODS 
     539Package access: static methods to prematurely terminate any process denoted by processID and any subprocesses it may have launched 
     540 
     541static void killWinProcessWithID(long processID) 
     542 
     543static boolean killUnixProcessWithID(long processID) 
     544    - for Mac/Linux 
     545 
     546 
     547__________________________________________________ 
     548J. OTHER USEFUL INSTANCE METHODS 
    433549__________________________________________________ 
    434550 
     
    437553 
    438554public boolean cancelRunningProcess() 
     555public boolean cancelRunningProcess(boolean forceWaitUntilInterruptible) 
    439556    - cancel the SafeProcess instance 
    440557    - untimately synchronized, so threadsafe 
    441 __________________________________________________ 
    442 USEFUL SYNCHRONISATION NOTES 
    443 __________________________________________________ 
    444 [Also add in links] 
    445  
    446  
    447 __________________________________________________ 
    448 TICKETS, COMMITS, TAGS 
    449 __________________________________________________ 
    450  
    451  
     558    - returns true if the process were still running so that an interrupt needed to be sent to terminate it, returns false if sending an interrupt was unnecessary because the process had already entered the start of its termination phase when cancel was called. Will also return false if the process was never run by the time cancel was called. 
     559    - usually you want to call the zero-argument version 
     560    - See the section CANCELLING for further details 
     561 
     562_________________________________________________________________________________________________ 
     563K. LINKS AND NOTES ON PROCESSES, PROCESS STREAMS, INTERRUPTING AND DESTROYING PROCESSES 
     564_________________________________________________________________________________________________ 
     565 
     566Processes and streams 
     567 
     568- http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html?page=2 
     569explains why SafeProcess.java is implemented with worker threads to handle a Process' iostreams. 
     570- http://steveliles.github.io/invoking_processes_from_java.html 
     571- http://mark.koli.ch/leaky-pipes-remember-to-close-your-streams-when-using-javas-runtimegetruntimeexec 
     572 
     573 
     574Interrupting a Process: 
     575 
     576- http://stackoverflow.com/questions/2126997/who-is-calling-the-java-thread-interrupt-method-if-im-not 
     577- http://stackoverflow.com/questions/13623445/future-cancel-method-is-not-working?noredirect=1&lq=1 
     578- http://stackoverflow.com/questions/3976344/handling-interruptedexception-in-java 
     579- http://stackoverflow.com/questions/4906799/why-invoke-thread-currentthread-interrupt-when-catch-any-interruptexception 
     580- http://stackoverflow.com/questions/4906799/why-invoke-thread-currentthread-interrupt-when-catch-any-interruptexception 
     581- https://praveer09.github.io/technology/2015/12/06/understanding-thread-interruption-in-java/ 
     582 
     583 
     584Destroying a process and subprocesses on various OS: 
     585 
     586- Need process id, pid, for which Java Native Access libraries (jar files) are required. 
     587http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program 
     588- To find the processID of the process launched by SafeProcess, need to use Java Native Access (JNA) jars, available jna.jar and jna-platform.jar. 
     589    http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program 
     590    http://stackoverflow.com/questions/35842/how-can-a-java-program-get-its-own-process-id 
     591    http://www.golesny.de/p/code/javagetpid 
     592    https://github.com/java-native-access/jna/blob/master/www/GettingStarted.md 
     593    We're using JNA v 4.1.0, downloaded from https://mvnrepository.com/artifact/net.java.dev.jna/jna 
     594 
     595WINDOWS NOTES: 
     596- Can't artificially send Ctrl-C: stackoverflow.com/questions/1835885/send-ctrl-c-to-process-open-by-java 
     597 
     598 On Windows, p.destroy() terminates process p that Java launched, 
     599 but does not terminate any processes that p may have launched. Presumably since they didn't form a proper process tree. 
     600    https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/e3cb7532-87f6-4ae3-9d80-a3afc8b9d437/how-to-kill-a-process-tree-in-cc-on-windows-platform?forum=vclanguage 
     601    https://msdn.microsoft.com/en-us/library/windows/desktop/ms684161(v=vs.85).aspx 
     602 
     603 Searching for: "forcibly terminate external process launched by Java on Windows"    
     604 Not possible: stackoverflow.com/questions/1835885/send-ctrl-c-to-process-open-by-java  
     605 But can use taskkill or tskill or wmic commands to terminate a process by processID 
     606 stackoverflow.com/questions/912889/how-to-send-interrupt-key-sequence-to-a-java-process 
     607 Taskkill command can kill by Image Name, such as all running perl, e.g. taskkill /f /im perl.exe 
     608 But what if we kill perl instances not launched by GS? 
     609    /f Specifies to forcefully terminate the process(es). We need this flag switched on to kill childprocesses. 
     610    /t Terminates the specified process and any child processes which were started by it.  
     611            /t didn't work to terminate subprocesses. Maybe since the process wasn't launched as  
     612            a properly constructed processtree. 
     613    /im is the image name (the name of the program), see Image Name column in Win Task Manager. 
     614     
     615 We don't want to kill all perl running processes.  
     616 Another option is to use wmic, available since Windows XP, to kill a process based on its command 
     617 which we sort of know (SafeProcess.command) and which can be seen in TaskManager under the  
     618 "Command Line" column of the Processes tab. 
     619    https://superuser.com/questions/52159/kill-a-process-with-a-specific-command-line-from-command-line  
     620 The following works kill any Command Line that matches -site localsite lucene-jdbm-demo 
     621    C:>wmic PATH win32_process Where "CommandLine like '%-site%localsite%%lucene-jdbm-demo%'" Call Terminate 
     622 "WMIC Wildcard Search using 'like' and %" 
     623    https://codeslammer.wordpress.com/2009/02/21/wmic-wildcard-search-using-like-and/ 
     624 However, we're not even guaranteed that every perl command GS launches will contain the collection name 
     625 Nor do we want to kill all perl processes that GS launches with bin\windows\perl\bin\perl, though this works: 
     626    wmic PATH win32_process Where "CommandLine like '%bin%windows%perl%bin%perl%'" Call Terminate 
     627 The above could kill GS perl processes we don't intend to terminate, as they're not spawned by the particular 
     628 Process we're trying to terminate from the root down. 
     629     
     630 Solution: We can use taskkill or the longstanding tskill or wmic to kill a process by ID. Since we can 
     631 kill an external process that SafeProcess launched OK, and only have trouble killing any child processes 
     632 it launched, we need to know the pids of the child processes.  
     633  
     634 We can use Windows' wmic to discover the childpids of a process whose id we know. 
     635 And we can use JNA to get the process ID of the external process that SafeProcess launched. 
     636  
     637 See links further above on how to find the processID of the process launched by SafeProcess on various OS, 
     638 and where to find the JNA jars needed for this. 
     639   
     640 WMIC can show us a list of parent process id and process id of running processes, and then we can 
     641 kill those child processes with a specific process id. 
     642    https://superuser.com/questions/851692/track-which-program-launches-a-certain-process 
     643    http://stackoverflow.com/questions/7486717/finding-parent-process-id-on-windows 
     644 WMIC can get us the pids of all childprocesses launched by parent process denoted by parent pid. 
     645 And vice versa: 
     646    if you know the parent pid and want to know all the pids of the child processes spawned: 
     647        wmic process where (parentprocessid=596) get processid 
     648    if you know a child process id and want to know the parent's id: 
     649        wmic process where (processid=180) get parentprocessid 
     650  
     651 The above is the current solution. 
     652  
     653 Linux ps equivalent on Windows is "tasklist", see 
     654    http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program 
     655 
     656 
     657MAC/UNIX NOTES: 
     658- Kill signals, their names and numerical equivalents: http://www.faqs.org/qa/qa-831.html 
     659- Killing a processtree (a process group) on Unix: 
     660https://stackoverflow.com/questions/8533377/why-child-process-still-alive-after-parent-process-was-killed-in-linux 
     661Works on Linux but not Mac: kill -TERM -pid 
     662Works on Mac but not Linux: pkill -TERM -P pid did work 
     663- https://stackoverflow.com/questions/28332888/return-value-of-kill 
     664    "kill returns an exit code of 0 (true) if the process still existed it and was killed. 
     665    kill returns an exit code of 1 (false) if the kill failed, probably because the process was no longer running." 
     666- https://superuser.com/questions/343031/sigterm-with-a-keyboard-shortcut 
     667Ctrl-C sends a SIGNINT, not SIGTERM or SIGKILL. And on Ctrl-C, "the signal is sent to the foreground *process group*." 
     668- https://linux.die.net/man/1/kill (manual) 
     669- https://unix.stackexchange.com/questions/117227/why-pidof-and-pgrep-are-behaving-differently 
     670- https://unix.stackexchange.com/questions/67635/elegantly-get-list-of-children-processes 
     671- https://stackoverflow.com/questions/994033/mac-os-x-quickest-way-to-kill-quit-an-entire-process-tree-from-within-a-cocoa-a 
     672- https://unix.stackexchange.com/questions/132224/is-it-possible-to-get-process-group-id-from-proc 
     673- https://unix.stackexchange.com/questions/99112/default-exit-code-when-process-is-terminated 
     674 
     675_______________________________________________________________________________________ 
     676L. USEFUL SYNCHRONISATION NOTES AND SUGGESTED READING ON THREADS AND THREAD SAFETY 
     677_______________________________________________________________________________________ 
     678 
     679Things worth reading on Concurrency and Thread safety: 
     680 
     681- Deitel and Deitel's newer editions of their Java books have updated their chapter on Concurrency. 
     682- http://docs.oracle.com/javase/tutorial/uiswing/concurrency/ 
     683series of Java articles on concurrency again. 
     684- https://docs.oracle.com/javase/tutorial/essential/concurrency/atomicvars.html 
     685On "primitive" like variables that provide instrinsic locks. 
     686 
     687 
     688Notes on synchronization: 
     689 
     690- You can declare methods are synchronized. For example, 
     691    public synchronized void doStuff() 
     692This implicitly synchronizes on the *instance* of the class on which this method is called. 
     693 
     694- Within any methods, you can synchronize on specific objects to just lock on them. You have a synchronized code block within which you can manipulate the object. 
     695 
     696Keep synchronized blocks and methods as short as you can. 
     697There are some subtle issues related to using synchronized methods, as explained below, that can result in deadlocks. 
     698 
     699 
     700More reading: 
     701 
     702- http://stackoverflow.com/questions/574240/is-there-an-advantage-to-use-a-synchronized-method-instead-of-a-synchronized-blo 
     703 
     704       "Not only do synchronized methods not lock the whole class, but they don't lock the whole instance either. Unsynchronized methods in the class may still proceed on the instance." 
     705       "Only the syncronized methods are locked. If there are fields you use within synced methods that are accessed by unsynced methods, you can run into race conditions." 
     706        
     707       "synchronizing on "this" is considered in some circles to be an anti-pattern. The unintended consequence is that outside of the class someone can lock on an object reference that is equal to "this" and prevent other threads from passing the barriers within the class potentially creating a deadlock situation. Creating a "private final Object = new Object();" variable purely for locking purposes is the often used solution. Here's another question relating directly to this issue. 
     708 
     709- http://stackoverflow.com/questions/442564/avoid-synchronizedthis-in-java?lq=1" 
     710 
     711       "A private lock is a defensive mechanism, which is never a bad idea. 
     712 
     713       Also, as you alluded to, private locks can control granularity. One set of operations on an object might be totally unrelated to another but synchronized(this) will mutually exclude access to all of them." 
     714 
     715 
     716- http://stackoverflow.com/questions/8393883/is-synchronized-keyword-exception-safe 
     717     "In any scoped thread-safe block, the moment you get out of it, the thread-safety is gone." 
     718     "In case of an exception the lock will be released." 
     719 
     720- http://stackoverflow.com/questions/8259479/should-i-synchronize-listener-notifications-or-not 
     721     "Use a CopyOnWriteArrayList for your listener arrays." 
     722     "If you use the CopyOnWriteArrayList, then you don't have to synchronize when iterating." 
     723     "CopyOnWriteArrayList is thread-safe, so there is no need to synchronize." 
     724 
     725     "Use a ConcurrentLinkedQueue<Listener> ... for this kind of problems: adding, removing and iterating simultaneously on a collection. 
     726     A precision : this solution prevents a listener from being called from the very moment it is deregistered." 
     727     "It means that you start iterating, an element is added, it will be called, another is removed, it won't, all this in the same iteration cycle. 
     728     It's the best of both world: ensuring synchronization, while being fine grained on who gets called and who's not." 
     729 
     730Examples of using CopyOnWriteArrayList: in GLI's shell/GShell.java and GS3's gsdl3/build/CollectionConstructor.java 
     731 
     732- http://stackoverflow.com/questions/8260205/when-a-listener-is-removed-is-it-okay-that-the-event-be-called-on-that-listener 
     733 
     734- http://stackoverflow.com/questions/2282166/java-synchronizing-on-primitives 
     735      
     736     1. You can't lock on a primitive and 
     737     2. Don't lock on a Long unless you're careful how you construct them. Long values created by autoboxing or Long.valueOf() in a certain range are guaranteed to be the same across the JVM which means other threads could be locking on the same exact Long object and giving you cross-talk. This can be a subtle concurrency bug (similar to locking on intern'ed strings). 
     738 
     739     Cross-talk: 
     740     "In electronics, crosstalk is any phenomenon by which a signal transmitted on one circuit or channel of a transmission system creates an undesired effect in another circuit or channel. Crosstalk is usually caused by undesired capacitive, inductive, or conductive coupling from one circuit, part of a circuit, or channel, to another." 
     741 
     742 
     743__________________________________________________ 
     744M. TICKETS, COMMITS, TAGS 
     745__________________________________________________ 
     746 
     747The following was already documented in ticket http://trac.greenstone.org/ticket/895 
     748 
     749    Other Java classes of GLI and GS3 src code were shifted from using Java's Process class directly to using our new SafeProcess class. 
     750 
     751    GLI src code classes that were changed: 
     752    /Scratch/ak19/gs3-svn-15Nov2016/gli>fgrep -rl "Process" src 
     753 
     754    + src/org/greenstone/gatherer/util/SafeProcess.java 
     755    + src/org/greenstone/gatherer/gui/FormatConversionDialog.java 
     756    + src/org/greenstone/gatherer/gui/DownloadPane.java 
     757    + src/org/greenstone/gatherer/util/GS3ServerThread.java 
     758    + 2 src/org/greenstone/gatherer/Gatherer.java [2 MORE but don't want to mess with those] 
     759    + 1 src/org/greenstone/gatherer/download/ServerInfoDialog.java 
     760    + 2 src/org/greenstone/gatherer/greenstone/Classifiers.java 
     761    + 2 src/org/greenstone/gatherer/greenstone/Plugins.java 
     762    + 1 src/org/greenstone/gatherer/collection/ScriptOptions.java 
     763    + !! src/org/greenstone/gatherer/util/ExternalProgram.java && file/FileAssociationManager.java 
     764 
     765    After updating GS3 src code to using SafeProcess, returned to GLI: 
     766    - VERY HARD: 
     767     + 1 src/org/greenstone/gatherer/shell/GShell.java 
     768    - VERY, VERY HARD: 
     769     + 2 src/org/greenstone/gatherer/download/DownloadJob.java 
     770 
     771 
     772    GS3 src code classes that were changed: 
     773    > fgrep -r "Process" . | grep -v ".svn" | grep -v "~" | less 
     774 
     775    + 2 ./java/org/greenstone/admin/guiext/Command.java 
     776    X ./java/org/greenstone/gsdl3/util/Processing.java 
     777    + MANY OCCURRENCES ./java/org/greenstone/gsdl3/service/MapRetrieve.java 
     778    + Uses Processing 2 times: ./java/org/greenstone/gsdl3/util/GDBMWrapper.java 
     779    + ./java/org/greenstone/util/BrowserLauncher.java 
     780    + ./java/org/greenstone/util/RunTarget.java 
     781 
     782    In the GLI and GS3 src files modified to use SafeProcess above, only the modifications to GS3's Command.java, MapRetrieve.java and GDMBWrapper.java are untested. All others have been tested, at least on Linux. Many have also been tested on Windows. 
     783 
     784    Tested GLI's collection building (uses GShell.java's modifications) and DownloadJob.java on all 3 OS. Both have a Cancel button that should cancel any processes and subprocesses launched. Before the modifications, subprocesses would still be running on Windows and during (full-)import on the Mac when cancel was pressed. But now have added code to SafeProcess to issue OS specific system calls to terminate the process and its subprocesses successfully on Windows and Mac too. 
     785 
     786    Also tested GS3 user comments and the online GS3's doc editor's meta editing followed by rebuilding. All worked successfully on Linux, Windows and Mac with the modifications to use SafeProcess. 
     787 
     788    The modifications to GLI started with GS3ServerThread.java and other *easily* changed classes in  
     789    - http://trac.greenstone.org/changeset/31582 (beginning of GLI modification)  
     790    - until http://trac.greenstone.org/changeset/31665 (GS3 src code modification). 
     791 
     792    The version of GS3 and GLI Java code before the GS3 src was modified is at  
     793    - http://trac.greenstone.org/browser/main/tags/GS3-src-SafeProcess 
     794 
     795    Next, GLI's GShell.java and any associated files were modified along with SafeProcess, to support cancelling. 
     796 
     797    Then, around 22 May, GLI's DownloadJob.java and any associated class files were modified along with SafeProcess to ensure subprocesses were safely terminated on Windows when the download action was cancelled OR when building (GShell.java) was cancelled. The Mac modifications to SafeProcess to ensure subprocesses were terminated was finished on 26 May 2017. 
     798 
     799    The tagged version of the GS3 and GLI Java code for when all the changes related to using SafeProcess were made to GS3 and GLI code is at  
     800    - http://trac.greenstone.org/browser/main/tags/UsingSafeProcess 
     801 
     802_________________________________________________________ 
     803N. GLI vs GS3 CODE's SAFEPROCESS.JAVA - THE DIFFERENCES 
     804_________________________________________________________ 
     805 
     806- package name 
     807    GLI: package org.greenstone.gatherer.util; 
     808    GS3: package org.greenstone.util; 
     809- imports: 
     810    GLI: import org.greenstone.gatherer.DebugStream; 
     811- GS3 uses Misc.java and GLI uses Utility.java for determining the OS and the appropriate newline character representation for the OS. 
     812- GS3 uses its Logger object for logging messages, GLI uses System.err (Debugstream code is presently commented out). 
     813 
     814 
     815    $ diff gli/src/org/greenstone/gatherer/util/SafeProcess.java src/java/org/greenstone/util/SafeProcess.java  
     816    ----------------------------------------------------------------------------------------------------------- 
     817    1c1 
     818    < package org.greenstone.gatherer.util; 
     819    --- 
     820    > package org.greenstone.util; 
     821    27c27 
     822    < import org.greenstone.gatherer.DebugStream; 
     823    --- 
     824    > //import org.greenstone.gatherer.DebugStream; 
     825    56c56 
     826    <     ///static Logger logger = Logger.getLogger(org.greenstone.util.SafeProcess.class.getName()); 
     827    --- 
     828    >     static Logger logger = Logger.getLogger(org.greenstone.util.SafeProcess.class.getName()); 
     829    153a154 
     830    >  
     831    816c817 
     832    <   if(Utility.isMac()) {  
     833    --- 
     834    >   if(Misc.isMac()) {  
     835    880c881 
     836    <     if(Utility.isWindows()) { 
     837    --- 
     838    >     if(Misc.isWindows()) { 
     839    909c910 
     840    <   if(!Utility.isMac() && canSkipExtraWorkIfLinux) { 
     841    --- 
     842    >   if(!Misc.isMac() && canSkipExtraWorkIfLinux) { 
     843    1284c1285 
     844    <           outputstr.append(Utility.NEWLINE); // "\n" is system dependent (Win must be "\r\n") 
     845    --- 
     846    >           outputstr.append(Misc.NEWLINE); // "\n" is system dependent (Win must be "\r\n") 
     847    1381c1382 
     848    <       /*if(Utility.isWindows()) { 
     849    --- 
     850    >       /*if(Misc.isWindows()) { 
     851    1416c1417 
     852    <   //logger.info(msg); 
     853    --- 
     854    >   logger.info(msg); 
     855    1418c1419 
     856    <   System.err.println(msg); 
     857    --- 
     858    >   //System.err.println(msg); 
     859    1424c1425 
     860    <   //logger.error(msg, e); 
     861    --- 
     862    >   logger.error(msg, e); 
     863    1426,1427c1427,1428 
     864    <   System.err.println(msg); 
     865    <   e.printStackTrace(); 
     866    --- 
     867    >   //System.err.println(msg); 
     868    >   //e.printStackTrace(); 
     869    1434c1435 
     870    <   //logger.error(e); 
     871    --- 
     872    >   logger.error(e); 
     873    1436c1437 
     874    <   e.printStackTrace(); 
     875    --- 
     876    >   //e.printStackTrace(); 
     877 
     878    ----------------------------------------------------------------------------------------------------------- 
     879 
     880 
     881__________________________________________________ 
     882O. FUTURE WORK, IMPROVEMENTS 
     883__________________________________________________ 
     884 
     885- On Windows, Perl could launch processes as proper ProcessTrees: http://search.cpan.org/~gsar/libwin32-0.191/ 
     886Then killing the root process will kill child processes naturally, and we won't need to call OS commands to terminate a process. 
     887- Need to find a Mac (or Unix) equivalent way creating Process Tree in Perl too. 
     888 
     889Eventually, instead of running a windows/mac or even linux command to kill the process ourselves, investigate if it is useful to change over to use 
     890    https://github.com/flapdoodle-oss/de.flapdoodle.embed.process/blob/master/src/main/java/de/flapdoodle/embed/process/runtime/Processes.java  
     891- mentioned by http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program 
     892- works with Apache license, http://www.apache.org/licenses/LICENSE-2.0 
     893- This is a Java class that uses JNA to terminate processes. It also has the getProcessID() method. 
     894- However, does it kill subprocesses? (The OS specific commands presently issued in SafeProcess.destroyProcess() ensure subprocesses are killed.) 
     895 
     896 
     897Currently, SafeProcess does Java things the way that older Java versions recognise and accept, not only for backwards compatibility with earlier versions of Java, but in order to minimise the number of drastic changes from porting the existing Greenstone GLI and GS3 code to using SafeProcess. 
     898 
     899- In future, think of changing the method doRuntimeExec() over to using ProcessBuilder, instead of Runtime.exec(). ProcessBuilder seems to have been introduced from Java 5. 
     900https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html 
     901See also https://zeroturnaround.com/rebellabs/how-to-deal-with-subprocesses-in-java/ 
     902which suggests using Apache Common Exec to launch processes and says what will be forthcoming in Java 9 
     903 
     904- Instead of creating and running Threads the usual way, should we use features of newer Java versions like the Concurrency API and Executor which uses Threadpools, and let Executor handle these details? Improvements would be with multicore systems 
     905https://stackoverflow.com/questions/3330430/does-java-have-support-for-multicore-processors-parallel-processing 
     906 
     907 
     908 
     909