source: gs2-extensions/pdf-box/trunk/GS_PDFBox_README.txt@ 32278

__________________________________________________________
README for GS modifications to the PDFBox App.
__________________________________________________________

5 June 2018

PROBLEM:
PDFBoxConverter.pm used the PDFBox App (v 1.8.2 at the time) to allow us the option of either extracting text from the entire PDF or saving each page of the PDF as an image.
Kathy wanted the option to have both the PDF's pages as images but also text extracted for each page (if extractable), so that contents could be searched and viewed in image format. Kathy said the HTML output was ugly, so we just pursue txt extraction.

IDEA FOR SOLUTION:
Kathy had found that the PDFBox app had an API. She looked at the API documentation for ExtractImages.java, which had a main function.
She explained that our GS perl code was calling the main function of classes such as this in order to do PDF to txt or img conversion.

Instead, of doing that, Kathy's idea was to modify the PDFBox app's ExtractImages.java into a GS version that would additionally extract text. The additional code could be copied in from the ExtractText.java class.

Note: In looking into this, I found that ExtractImages literally extracts any images in the PDF, rather than converting pages to images. The actual PDFBox class that our perl code calls to do the PDF pages to images conversion is PDFToImage.java.


CONTENTS
The following are the steps I found I needed to go through and which are described in this readme:
- upgrade to pdfbox app 2.09, 
- check the new version didn't break the stuff for which we specifically settled on version 1.8.2 before
- grab the pdfbox 2.09 source code
- introduce the new class PDFBoxToImagesAndText.java (originally added to the pdfbox app 2.09 src locally, now maintained separately with a GS Java package name). The new class is based on Apache PDFBox's PDFToImage.java, with added code to extract text based on Apache PDFBox's ExtractText.java
- OLD: rebuild the source, a maven project. Then modify the pdfbox app 2.09 (non source) ext jar file by including the new .class file
- compiling the new java class against the pdfbox-app.jar
- Now we're possibly generating a txtfile for each img file (assuming there was extractable txt), some modifications needed to be made to the generated .item file to ensure these referred to the txt file.
- work that remains to be done


1. Used to use pdfbox app version 1.8.2, see commit message at http://trac.greenstone.org/changeset/29029/gs2-extensions/pdf-box/trunk/java/lib/java/pdfbox-app.jar

2. Upgrading to pdfbox app 2.0.9 (downloaded from https://pdfbox.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-us.apache.org%2Fdist%2F):

Doesn't work out of the box, such as to extract text from a PDF file.

But if references to org.apache.pdfbox.* in PDFBoxConverter.pm are changed to org.apache.pdfbox.tools.*, the pdf is still processed.

And the extracted text contains fi ("file") and fl ("workflow") instead of single character ligatures for these. Once again, refer to the commit message at http://trac.greenstone.org/changeset/29029/gs2-extensions/pdf-box/trunk/java/lib/java/pdfbox-app.jar


BUT we get to see this warning message now:
    ReadTextFile: WARNING: /home/greenstone/gs3-svn-26Mar2018/gs2build/tmp/F996.html appears to be encoded in an unsupported encoding ("utf_8) - using utf8


-> In ReadTextFile.pm change:
    if ($best_encoding eq "utf_8") { $best_encoding = "utf8" }
to:
    if ($best_encoding eq "utf_8" ||  $best_encoding =~ /utf_8/) { $best_encoding = "utf8" }

3. To recompile the new PDFBoxToImagesAndText.java code:

- grab the svn version of the Greenstone pdfbox extension
- Then from the svn checked out pdfbox (trunk/java) folder, run 
    $ javac -cp `pwd`/lib/java/pdfbox-app.jar -d `pwd`/build src/org/greenstone/pdfbox/PDFBoxToImagesAndText.java

which will compile our custom PDFBoxToImagesAndText.java file against the pdfbox-app.jar in the classpath (-cp) and output the .class file into the directory denoted by -d

To run, that build folder needs to be on the classpath, besides pdfbox-app.jar itself. See PDFBoxConverter.pm

4. For convenience PDFBoxToImagesAndText.java further no longer generates <PDF filename prefix><sequential number>.img-ext (jpg/gif/png) and matchingly named txt file, but just <sequential number>.img-ext and <sequential number>.txt.

Further, needed to modify GS perllib's util::create_itemfile(), so that it no longer leaves each txtfile="" property empty in the item file if we've generated matching text files for the image files. In the past, util::create_itemfile() only worked with images, but now it's modified to additionally deal with matching text files.


5. No need to rename the pdf-box, its containing tarball and its jar to gs-pdf-box. Doing that may have required changes to get nightly builds, releases, diffcol to get them to still work.

However, Dr Bainbridge explained that the solution is to not rename pdfbox-app.jar to gs-pdfbox-app.jar, but have our new custom java file living separately rather than within the pdfbox-app.jar though compiled against it. The java file will have its own greenstone Java package (org.greenstone.pdfbox) and the class file produced from our custom class will then be run from its package by PDFBoxConverter.pm


6. I was not sure what the licensing information for pdfbox/java/src/PDFBoxToImagesAndText.java should be in its preamble comment section, since most of the code is a copy or slight modification of the Apache PDFBox app's code. I asked Dr Nichols about the licensing, and he pointed out that Apache's PDFBox pages referred to https://www.apache.org/licenses/LICENSE-2.0
Dr Nichols drew attention to its section "Redistribution", saying that that should cover things. I've tried to follow the points there, and have therefore included apache's LICENSE.txt and NOTICE.txt files into our pdfbox/java/src and pdfbox/java/src build folders where PDFBoxToImagesAndText.java and PDFBoxToImagesAndText.class, respectively, will live.

------------------


EXTRA INFORMATION

Older but useful information, as it covers how to use maven to build the pdfbox-app, and include a new java file to be built into the pdfbox-app and distribute the file within the app.

1. Originally I thought I would need to edit org.apache.pdfbox.tools.PDFToImage (not ExtractImages), as that's the tool that converts a PDF's individual pages to images. Eventually, since there was a new class PDFBoxToImagesAndText.java, Dr Bainbridge explained that this could just be compiled against an exiting pdfbox-app.jar, rather than including it inside the jar and recompiling the pdfbox-app in entirety.

When I was still compiling the pdfbox-app in entirety, with and without my new code included therein, to obtain the basecode and build the app, the steps were:


- unzipped the src version of pdfbox 2.09
- downloaded and unzipped maven
    Got it from https://maven.apache.org/download.cgi
- added maven/bin to PATH and set JAVA_TOOL_OPTIONS as below* in setenv.sh then sourced setenv.sh (instructions at https://maven.apache.org/install.html)
    export PATH=$JAVA_HOME/bin:/home/greenstone/apache-maven-3.5.3/bin:$PATH
    export JAVA_TOOL_OPTIONS="-Dhttps.protocols=TLSv1.2"

(JAVA_TOOL_OPTIONS needs to be set as above because when using maven to build pdfbox src without it, saw mvn fail to build with the error "SSLException: Received fatal alert: protocol_version" explained at: https://github.com/jenkinsci/ghprb-plugin/issues/638)

- As per pdfbox 2.09 src's README.md, the build command is:
    mvn clean install

Run that command after first going into the extracted pdfbox-2.09 src code's folder, wherever this is extracted, e.g.
    greenstone@bedrock:~/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9

- The source code to edit is at /home/greenstone/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9/tools/src/main/java/org/apache/pdfbox/tools
Create PDFBoxToImagesAndText.java based on PDFToImage.java into which to incorporate from ExtractText.java

- When built, will get /home/greenstone/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9/tools/target/classes/org/apache/pdfbox/tools
Put that into the unzipped pdfbox app 2.09 version (a.o.t. the unzipped pdfbox src 2.09)


2. Once the pdfbox-app has been entirely built the first time, want smaller rebuilds.

Final section of output of full rebuild:


[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] PDFBox parent ...................................... SUCCESS [  1.457 s]
[INFO] Apache FontBox ..................................... SUCCESS [ 16.135 s]
[INFO] Apache XmpBox ...................................... SUCCESS [  7.046 s]
[INFO] Apache PDFBox ...................................... SUCCESS [01:37 min]
[INFO] Apache Preflight ................................... SUCCESS [ 20.115 s]
[INFO] Apache Preflight application ....................... SUCCESS [  8.270 s]
[INFO] Apache PDFBox Debugger ............................. SUCCESS [  2.246 s]
[INFO] Apache PDFBox tools ................................ SUCCESS [  8.567 s]
[INFO] Apache PDFBox application .......................... SUCCESS [  7.401 s]
[INFO] Apache PDFBox Debugger application ................. SUCCESS [  7.184 s]
[INFO] Apache PDFBox examples ............................. SUCCESS [ 15.077 s]
[INFO] PDFBox reactor 2.0.9 ............................... SUCCESS [  0.148 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:12 min
[INFO] Finished at: 2018-06-05T16:03:27+12:00
[INFO] ------------------------------------------------------------------------



The maven module in pdfbox-app to actually build: "Apache PDFBox tools", as this contains ExtractText.java, PDFToImage.java and the new class we're going to put in there.

To build a project component ("module") in maven, see
- https://maven.apache.org/guides/mini/guide-multiple-modules.html
- https://stackoverflow.com/questions/1114026/maven-modules-building-a-single-specific-module
- https://stackoverflow.com/questions/23075415/maven-build-second-level-child-projects-using-reactor-option-pl

So since we want to build the package "tools" (and its dependencies), we do:
    mvn [clean] install -pl tools -am

Again, run the above wherever the pdfbox app's src code v 2.09 is extracted:
    greenstone@bedrock:~/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9$ mvn clean install -pl tools -am

(This will also compile our new PDFBoxToImagesAndText.java and put its class file PDFBoxToImagesAndText.class into the target dir /home/greenstone/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9/tools/target/classes/org/apache/pdfbox/tools)

__________________________________________________________
Unused, originally considered as starting points. Kathy's searches:

Google: pdfbox api extract text
https://stackoverflow.com/questions/23813727/how-to-extract-text-from-a-pdf-file-with-apache-pdfbox

Google: pdfbox extractimages
https://pdfbox.apache.org/docs/2.0.8/javadocs/org/apache/pdfbox/tools/ExtractImages.html
https://stackoverflow.com/questions/8705163/extract-images-from-pdf-using-pdfbox