Index: /gs2-extensions/pdf-box/trunk/GS_PDFBox_README.txt
===================================================================
--- /gs2-extensions/pdf-box/trunk/GS_PDFBox_README.txt	(revision 32193)
+++ /gs2-extensions/pdf-box/trunk/GS_PDFBox_README.txt	(revision 32193)
@@ -0,0 +1,161 @@
+__________________________________________________________
+README for GS modifications to the PDFBox App.
+__________________________________________________________
+
+5 June 2018
+
+PROBLEM:
+PDFBoxConverter.pm used the PDFBox App (v 1.8.2 at the time) to allow us the option of either extracting text from the entire PDF or saving each page of the PDF as an image.
+Kathy wanted the option to have both the PDF's pages as images but also text extracted for each page (if extractable), so that contents could be searched and viewed in image format. Kathy said the HTML output was ugly, so we just pursue txt extraction.
+
+IDEA FOR SOLUTION:
+Kathy had found that the PDFBox app had an API. She looked at the API documentation for ExtractImages.java, which had a main function.
+She explained that our GS perl code was calling the main function of classes such as this in order to do PDF to txt or img conversion.
+
+Instead, of doing that, Kathy's idea was to modify the PDFBox app's ExtractImages.java into a GS version that would additionally extract text. The additional code could be copied in from the ExtractText.java class.
+
+Note: In looking into this, I found that ExtractImages literally extracts any images in the PDF, rather than converting pages to images. The actual PDFBox class that our perl code calls to do the PDF pages to images conversion is PDFToImage.java.
+
+
+CONTENTS
+The following are the steps I found I needed to go through and which are described in this readme:
+- upgrade to pdfbox app 2.09, 
+- check the new version didn't break the stuff for which we specifically settled on version 1.8.2 before
+- grab the pdfbox 2.09 source code
+- introduce the new class GS_PDFToImagesAndText.java to the pdfbox app 2.09 src, based on its PDFToImage.java with added code to extract text based on ExtractText.java
+- rebuild the source, a maven project
+- modify the pdfbox app 2.09 (non source) ext jar file by including the new .class file
+- Now we're possible generating a txtfile for each img file (assuming there was extractable txt), some modifications needed to be made to the generated .item file to ensure these referred to the txt file.
+- work that remains to be done
+
+
+0. Used to use pdfbox app version 1.8.2, see commit message at http://trac.greenstone.org/browser/gs2-extensions/pdf-box/trunk/java/lib/java/pdfbox-app.jar
+
+1. Upgrading to pdfbox app 2.0.9 (downloaded from https://pdfbox.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-us.apache.org%2Fdist%2F):
+
+Doesn't work out of the box, such as to extract text from a PDF file.
+
+But if references to org.apache.pdfbox.* in PDFBoxConverter.pm are changed to org.apache.pdfbox.tools.*, the pdf is still processed.
+
+And the extracted text contains fil ("file") and fl ("workflow") instead of ligatures. Once again, refer to the commit message at http://trac.greenstone.org/browser/gs2-extensions/pdf-box/trunk/java/lib/java/pdfbox-app.jar
+
+
+BUT we get to see this warning message now:
+	ReadTextFile: WARNING: /home/greenstone/gs3-svn-26Mar2018/gs2build/tmp/F996.html appears to be encoded in an unsupported encoding ("utf_8) - using utf8
+
+
+-> In ReadTextFile.pm change:
+	if ($best_encoding eq "utf_8") { $best_encoding = "utf8" }
+to:
+	if ($best_encoding eq "utf_8" ||  $best_encoding =~ /utf_8/) { $best_encoding = "utf8" }
+
+
+2. Need to edit org.apache.pdfbox.tools.PDFToImage (not ExtractImages) as that's the tool that converts a PDF's individual pages to images
+
+
+- unzipped the src version of pdfbox 2.09
+- downloaded and unzipped maven
+	Got it from https://maven.apache.org/download.cgi
+- added maven/bin to PATH and set JAVA_TOOL_OPTIONS as below* in setenv.sh then sourced setenv.sh (instructions at https://maven.apache.org/install.html)
+	export PATH=$JAVA_HOME/bin:/home/greenstone/apache-maven-3.5.3/bin:$PATH
+	export JAVA_TOOL_OPTIONS="-Dhttps.protocols=TLSv1.2"
+
+(JAVA_TOOL_OPTIONS needs to be set as above because when using maven to build pdfbox src without it, saw mvn fail to build with the error "SSLException: Received fatal alert: protocol_version" explained at: https://github.com/jenkinsci/ghprb-plugin/issues/638)
+
+- As per pdfbox 2.09 src's README.md, the build command is:
+	mvn clean install
+
+Run that command after first going into the extracted pdfbox-2.09 src code's folder, wherever this is extracted, e.g.
+	greenstone@bedrock:~/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9
+
+- The source code to edit is at /home/greenstone/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9/tools/src/main/java/org/apache/pdfbox/tools
+Create GS_PDFToImagesAndText.java based on PDFToImage.java into which to incorporate from ExtractText.java
+
+- When built, will get /home/greenstone/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9/tools/target/classes/org/apache/pdfbox/tools
+Put that into the unzipped pdfbox app 2.09 version (a.o.t. the unzipped pdfbox src 2.09)
+
+
+3. Once the pdfbox-app has been entirely built the first time, want smaller rebuilds.
+
+Final section of output of full rebuild:
+
+
+[INFO] ------------------------------------------------------------------------
+[INFO] Reactor Summary:
+[INFO] 
+[INFO] PDFBox parent ...................................... SUCCESS [  1.457 s]
+[INFO] Apache FontBox ..................................... SUCCESS [ 16.135 s]
+[INFO] Apache XmpBox ...................................... SUCCESS [  7.046 s]
+[INFO] Apache PDFBox ...................................... SUCCESS [01:37 min]
+[INFO] Apache Preflight ................................... SUCCESS [ 20.115 s]
+[INFO] Apache Preflight application ....................... SUCCESS [  8.270 s]
+[INFO] Apache PDFBox Debugger ............................. SUCCESS [  2.246 s]
+[INFO] Apache PDFBox tools ................................ SUCCESS [  8.567 s]
+[INFO] Apache PDFBox application .......................... SUCCESS [  7.401 s]
+[INFO] Apache PDFBox Debugger application ................. SUCCESS [  7.184 s]
+[INFO] Apache PDFBox examples ............................. SUCCESS [ 15.077 s]
+[INFO] PDFBox reactor 2.0.9 ............................... SUCCESS [  0.148 s]
+[INFO] ------------------------------------------------------------------------
+[INFO] BUILD SUCCESS
+[INFO] ------------------------------------------------------------------------
+[INFO] Total time: 03:12 min
+[INFO] Finished at: 2018-06-05T16:03:27+12:00
+[INFO] ------------------------------------------------------------------------
+
+
+
+The maven module in pdfbox-app to actually build: "Apache PDFBox tools", as this contains ExtractText.java, PDFToImage.java and the new class we're going to put in there.
+
+To build a project component ("module") in maven, see
+- https://maven.apache.org/guides/mini/guide-multiple-modules.html
+- https://stackoverflow.com/questions/1114026/maven-modules-building-a-single-specific-module
+- https://stackoverflow.com/questions/23075415/maven-build-second-level-child-projects-using-reactor-option-pl
+
+So since we want to build the package "tools" (and its dependencies), we do:
+	mvn [clean] install -pl tools -am
+
+Again, run the above wherever the pdfbox app's src code v 2.09 is extracted:
+	greenstone@bedrock:~/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9$ mvn clean install -pl tools -am
+
+(This will also compile our new GS_PDFToImagesAndText.java and put its class file GS_PDFToImagesAndText.class into the target dir /home/greenstone/gs3-svn-26Mar2018/gs2build/ext/pdfbox-2.0.9/tools/target/classes/org/apache/pdfbox/tools)
+
+
+4. For convenience GS_PDFToImagesAndText.java further no longer generates <PDF filename prefix><sequential number>.img-ext (jpg/gif/png) and matchingly named txt file, but just <sequential number>.img-ext and <sequential number>.txt.
+
+Further, needed to modify GS perllib's util::create_itemfile(), so that it no longer leaves each txtfile="" property empty in the item file if we've generated matching text files for the image files. In the past, util::create_itemfile() only worked with images, but now it's modified to additionally deal with matching text files.
+
+
+5. Remaining Issues.
+
+a. Build messages (when building lower level PDF conversion commands):
+ImageConverter: "/home/greenstone/gs3-svn-26Mar2018/gs2build/tmp/F143/A9-access-best-practices1.txt" too small, skipping
+Same for: A9-access-best-practices56.txt
+
+
+Lower level PDF conversion commands:
+calling cmd "/usr/bin/perl" -S gsConvert.pl -verbose 2 -pdf_zoom 2 -errlog "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/tmp/1528179077/err.log" -output pagedimg_jpg "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/tmp/1528179077/A9-access-best-practices.pdf"
+import.pl> 
+
+
+"/usr/bin/perl" -S gsConvert.pl -verbose 2 -pdf_zoom 2 -errlog "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/tmp/1528179077/err.log" -output pagedimg_jpg "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices.pdf"
+
+
+"/usr/bin/perl" -S pdfpstoimg.pl -convert_to jpg "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices.pdf" "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices" > "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices.out" 2> "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices.err"
+
+
+@@@@ pdfpstoimg.pl::pdfps_to_img(): cmd = "/usr/bin/perl" -S gs-magick.pl convert "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices.pdf" "/home/greenstone/gs3-svn-26Mar2018/web/sites/localsite/collect/pdftst1/import/A9-access-best-practices/A9-access-best-practices.jpg"
+
+
+b. Have not yet renamed the pdf-box, its containing tarball and its jar to gs-pdf-box yet, since a lot of things foreseen (nightly builds, releases, diffcol) and unforeseen can break due to the name change.
+
+c. Not sure what the licensing information for pdfbox/java/src/GS_PDFToImagesAndText.java should be in its preamble comment section, since most of the code is a copy or slight modification of the Apache PDFBox app's code.
+
+__________________________________________________________
+Unused, originally considered as starting points. Kathy's searches:
+
+Google: pdfbox api extract text
+https://stackoverflow.com/questions/23813727/how-to-extract-text-from-a-pdf-file-with-apache-pdfbox
+
+Google: pdfbox extractimages
+https://pdfbox.apache.org/docs/2.0.8/javadocs/org/apache/pdfbox/tools/ExtractImages.html
+https://stackoverflow.com/questions/8705163/extract-images-from-pdf-using-pdfbox
Index: /gs2-extensions/pdf-box/trunk/java/perllib/plugins/PDFBoxConverter.pm
===================================================================
--- /gs2-extensions/pdf-box/trunk/java/perllib/plugins/PDFBoxConverter.pm	(revision 32192)
+++ /gs2-extensions/pdf-box/trunk/java/perllib/plugins/PDFBoxConverter.pm	(revision 32193)
@@ -127,8 +127,11 @@
 	my $pbajar = &FileUtils::filenameConcatenate($gextpb_home,"lib","java","pdfbox-app.jar");
 	my $java = &util::get_java_command();
-	my $launch_cmd = "$java -cp \"$pbajar\" -Dline.separator=\"<br />\" org.apache.pdfbox.ExtractText";
+	my $launch_cmd = "$java -cp \"$pbajar\" -Dline.separator=\"<br />\" org.apache.pdfbox.tools.ExtractText";
 	
 	$self->{'pdfbox_launch_cmd'} = $launch_cmd;
-	$self->{'pdfbox_img_launch_cmd'} = "java -cp \"$pbajar\" org.apache.pdfbox.PDFToImage"; # cmd for converting pages to images (gif, jpg, png)
+	#$self->{'pdfbox_img_launch_cmd'} = "java -cp \"$pbajar\" org.apache.pdfbox.tools.PDFToImage"; # pdfbox 2.09 cmd for converting each PDF page to an image (gif, jpg, png)
+	# Now: use this cmd to launch our new custom PDFBox class (GS_PDFToImagesAndText.java) to convert each PDF page into an image (gif, jpg, png)
+	# AND its extracted text. An item file is still generated, but this time referring to txtfiles too, not just the images. Result: searchable paged output.
+	$self->{'pdfbox_img_launch_cmd'} = "java -cp \"$pbajar\" org.apache.pdfbox.tools.GS_PDFToImagesAndText"; 
     }
     else {       
@@ -311,3 +314,3 @@
 
 
-1;	
+1;
Index: /gs2-extensions/pdf-box/trunk/java/src/GS_PDFToImagesAndText.java
===================================================================
--- /gs2-extensions/pdf-box/trunk/java/src/GS_PDFToImagesAndText.java	(revision 32193)
+++ /gs2-extensions/pdf-box/trunk/java/src/GS_PDFToImagesAndText.java	(revision 32193)
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.pdfbox.tools;
+
+import java.awt.HeadlessException;
+import java.awt.Toolkit;
+import java.awt.image.BufferedImage;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import org.apache.pdfbox.io.IOUtils;
+
+import javax.imageio.ImageIO;
+
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.apache.pdfbox.pdmodel.PDPage;
+import org.apache.pdfbox.pdmodel.common.PDRectangle;
+import org.apache.pdfbox.rendering.ImageType;
+import org.apache.pdfbox.rendering.PDFRenderer;
+import org.apache.pdfbox.tools.imageio.ImageIOUtil;
+
+import org.apache.pdfbox.pdmodel.encryption.AccessPermission;
+import org.apache.pdfbox.text.PDFTextStripper;
+
+/**
+ * This class is based on PDFToImage.java which converts
+ * the pages of a PDF document to images.
+ * This class should convert the pages to images and 
+ * extract the text of each page. That part of the code
+ * to be taken from ExtractText.java
+ *
+ * Built on Apache PDFBox's PDFToImage.java with minor modifications.
+ * ak19
+ */
+public final class GS_PDFToImagesAndText
+{
+    private static final String PASSWORD = "-password";
+    private static final String ENCODING = "-encoding";
+    private static final String START_PAGE = "-startPage";
+    private static final String END_PAGE = "-endPage";
+    private static final String PAGE = "-page";
+    private static final String IMAGE_TYPE = "-imageType";
+    private static final String FORMAT = "-format";
+    private static final String OUTPUT_PREFIX = "-outputPrefix";
+    private static final String PREFIX = "-prefix";
+    private static final String COLOR = "-color";
+    private static final String RESOLUTION = "-resolution";
+    private static final String DPI = "-dpi";
+    private static final String CROPBOX = "-cropbox";
+    private static final String TIME = "-time";
+
+    private static final String STD_ENCODING = "UTF-8";
+
+    /**
+     * private constructor.
+    */
+    private GS_PDFToImagesAndText()
+    {
+        //static class
+    }
+
+    /**
+     * Infamous main method.
+     *
+     * @param args Command line arguments, should be one and a reference to a file.
+     *
+     * @throws IOException If there is an error parsing the document.
+     */
+    public static void main( String[] args ) throws IOException
+    {
+        try
+        {
+            // force KCMS (faster than LCMS) if available
+            Class.forName("sun.java2d.cmm.kcms.KcmsServiceProvider");
+            System.setProperty("sun.java2d.cmm", "sun.java2d.cmm.kcms.KcmsServiceProvider");
+        }
+        catch (ClassNotFoundException e)
+        {
+            // do nothing
+        }
+
+        // suppress the Dock icon on OS X
+        System.setProperty("apple.awt.UIElement", "true");
+
+        String password = "";
+	String encoding = STD_ENCODING;
+        String pdfFile = null;
+        String outputPrefix = null;
+        String imageFormat = "jpg";
+        int startPage = 1;
+        int endPage = Integer.MAX_VALUE;
+        String color = "rgb";
+        int dpi;
+        float cropBoxLowerLeftX = 0;
+        float cropBoxLowerLeftY = 0;
+        float cropBoxUpperRightX = 0;
+        float cropBoxUpperRightY = 0;
+        boolean showTime = false;
+        try
+        {
+            dpi = Toolkit.getDefaultToolkit().getScreenResolution();
+        }
+        catch( HeadlessException e )
+        {
+            dpi = 96;
+        }
+        for( int i = 0; i < args.length; i++ )
+        {
+            if( args[i].equals( PASSWORD ) )
+            {
+                i++;
+                if( i >= args.length )
+                {
+                    usage();
+                }
+                password = args[i];
+            }
+            else if( args[i].equals( ENCODING ) )
+            {
+                i++;
+                if( i >= args.length )
+                {
+                    usage();
+                }
+                encoding = args[i];
+            }
+            else if( args[i].equals( START_PAGE ) )
+            {
+                i++;
+                if( i >= args.length )
+                {
+                    usage();
+                }
+                startPage = Integer.parseInt( args[i] );
+            }
+            else if( args[i].equals( END_PAGE ) )
+            {
+                i++;
+                if( i >= args.length )
+                {
+                    usage();
+                }
+                endPage = Integer.parseInt( args[i] );
+            }
+            else if( args[i].equals( PAGE ) )
+            {
+                i++;
+                if( i >= args.length )
+                {
+                    usage();
+                }
+                startPage = Integer.parseInt( args[i] );
+                endPage = Integer.parseInt( args[i] );
+            }
+            else if( args[i].equals(IMAGE_TYPE) || args[i].equals(FORMAT) )
+            {
+                i++;
+                imageFormat = args[i];
+            }
+            else if( args[i].equals( OUTPUT_PREFIX ) || args[i].equals( PREFIX ) )
+            {
+                i++;
+                outputPrefix = args[i];
+            }
+            else if( args[i].equals( COLOR ) )
+            {
+                i++;
+                color = args[i];
+            }
+            else if( args[i].equals( RESOLUTION ) || args[i].equals( DPI ) )
+            {
+                i++;
+                dpi = Integer.parseInt(args[i]);
+            }
+            else if( args[i].equals( CROPBOX ) )
+            {
+                i++;
+                cropBoxLowerLeftX = Float.valueOf(args[i]);
+                i++;
+                cropBoxLowerLeftY = Float.valueOf(args[i]);
+                i++;
+                cropBoxUpperRightX = Float.valueOf(args[i]);
+                i++;
+                cropBoxUpperRightY = Float.valueOf(args[i]);
+            }
+            else if( args[i].equals( TIME ) )
+            {
+                showTime = true;
+            }
+            else
+            {
+                if( pdfFile == null )
+                {
+                    pdfFile = args[i];
+                }
+            }
+        }
+        if( pdfFile == null )
+        {
+            usage();
+        }
+        else
+        {
+            if(outputPrefix == null)
+            {
+                outputPrefix = pdfFile.substring( 0, pdfFile.lastIndexOf( '.' ));
+            }
+
+            PDDocument document = null;
+            try
+            {
+		boolean extractingTextAllowed = true;		
+		//String outputFile = null;
+
+		/*startProcessing("Loading PDF "+pdfFile);
+		if( outputFile == null && pdfFile.length() >4 )
+                {
+                    outputFile = new File( pdfFile.substring( 0, pdfFile.length() -4 ) + ext ).getAbsolutePath();
+                }*/
+
+                document = PDDocument.load(new File(pdfFile), password);
+
+		AccessPermission ap = document.getCurrentAccessPermission();
+                if( ! ap.canExtractContent() )
+                {
+                    //throw new IOException( "You do not have permission to extract text" );
+		    System.err.println( "*** You do not have permission to extract text" ); // still extract the pages as images then?
+		    extractingTextAllowed = false;
+                }
+		//stopProcessing("Time for loading: ", startTime);
+
+		// don't extract to HTML in this class, just extract to txt
+		PDFTextStripper stripper = new PDFTextStripper();            
+                //stripper.setSortByPosition( sort );
+                //stripper.setShouldSeparateByBeads( separateBeads );
+		stripper.setShouldSeparateByBeads( true );
+                
+
+                ImageType imageType = null;
+                if ("bilevel".equalsIgnoreCase(color))
+                {
+                    imageType = ImageType.BINARY;
+                }
+                else if ("gray".equalsIgnoreCase(color))
+                {
+                    imageType = ImageType.GRAY;
+                }
+                else if ("rgb".equalsIgnoreCase(color))
+                {
+                    imageType = ImageType.RGB;
+                }
+                else if ("rgba".equalsIgnoreCase(color))
+                {
+                    imageType = ImageType.ARGB;
+                }
+                
+                if (imageType == null)
+                {
+                    System.err.println( "Error: Invalid color." );
+                    System.exit( 2 );
+                }
+
+                //if a CropBox has been specified, update the CropBox:
+                //changeCropBoxes(PDDocument document,float a, float b, float c,float d)
+                if ( cropBoxLowerLeftX!=0 || cropBoxLowerLeftY!=0
+                        || cropBoxUpperRightX!=0 || cropBoxUpperRightY!=0 )
+                {
+                    changeCropBox(document,
+                            cropBoxLowerLeftX, cropBoxLowerLeftY,
+                            cropBoxUpperRightX, cropBoxUpperRightY);
+                }
+
+                long startTime = System.nanoTime();
+		
+                // render the pages
+                boolean success = true;
+                endPage = Math.min(endPage, document.getNumberOfPages());
+                PDFRenderer renderer = new PDFRenderer(document);
+                for (int i = startPage - 1; i < endPage; i++)
+                {
+                    // turn page into image
+                    BufferedImage image = renderer.renderImageWithDPI(i, dpi, imageType);
+                    int lastSlash = outputPrefix.lastIndexOf(File.separator);
+                    outputPrefix = outputPrefix.substring(0, lastSlash+1); // include the folder (/) but not the filename prefix
+                    String fileName = outputPrefix + (i + 1) + ".";
+                    success &= ImageIOUtil.writeImage(image, fileName+imageFormat, dpi);
+
+
+                    // image version of page done, now extract text from current page
+                    if(extractingTextAllowed) {
+			    Writer output = null;	
+			    try {
+				    output = new OutputStreamWriter( new FileOutputStream( fileName+"txt" ), encoding );
+				    stripper.setStartPage( i+1 );
+				    stripper.setEndPage( i+1 );
+				
+				    //if (debug)
+				    //{
+				    	System.err.println("Writing to "+fileName);
+				    //}
+
+				    // Extract text for main document, the specified pages
+				    stripper.writeText( document, output );
+			    } catch (Exception ex) {
+				System.err.println( "*** Unable to create txt file " + fileName + "txt. Exception: " + ex.getMessage());
+			    } finally {
+		            	IOUtils.closeQuietly(output);
+			    }
+                    }
+                }
+
+		// GS NOTE: We just extracted text for (each page of) the main document, but
+		// we're not additionally extracting text for any "embedded PDFs" as is done in ExtractText.java
+
+                // performance stats
+                long endTime = System.nanoTime();
+                long duration = endTime - startTime;
+                int count = 1 + endPage - startPage;
+                if (showTime)
+                {
+                    System.err.printf("Rendered %d page%s in %dms\n", count, count == 1 ? "" : "s",
+                                      duration / 1000000);
+                }
+
+                if (!success)
+                {
+                    System.err.println( "Error: no writer found for image format '"
+                            + imageFormat + "'" );
+                    System.exit(1);
+                }
+            }
+            finally
+            {
+                if( document != null )
+                {
+                    document.close();
+                }
+            }
+        }
+    }
+
+    /**
+     * This will print the usage requirements and exit.
+     */
+    private static void usage()
+    {
+        String message = "Usage: java -jar pdfbox-app-x.y.z.jar GS_PDFToImagesAndText [options] <inputfile>\n"
+            + "\nOptions:\n"
+            + "  -password  <password>            : Password to decrypt document\n"
+            + "  -encoding  <output encoding> : UTF-8 (default) or ISO-8859-1, UTF-16BE, UTF-16LE, etc.\n"
+            + "  -format <string>                 : Image format: " + getImageFormats() + "\n"
+            + "  -prefix <string>                 : Filename prefix for image files\n"
+            + "  -page <number>                   : The only page to extract (1-based)\n"
+            + "  -startPage <int>                 : The first page to start extraction (1-based)\n"
+            + "  -endPage <int>                   : The last page to extract(inclusive)\n"
+            + "  -color <int>                     : The color depth (valid: bilevel, gray, rgb, rgba)\n"
+            + "  -dpi <int>                       : The DPI of the output image\n"
+            + "  -cropbox <int> <int> <int> <int> : The page area to export\n"
+            + "  -time                            : Prints timing information to stdout\n"
+            + "  <inputfile>                      : The PDF document to use\n";
+        
+        System.err.println(message);
+        System.exit( 1 );
+    }
+
+    private static String getImageFormats()
+    {
+        StringBuilder retval = new StringBuilder();
+        String[] formats = ImageIO.getReaderFormatNames();
+        for( int i = 0; i < formats.length; i++ )
+        {
+           if (formats[i].equalsIgnoreCase(formats[i]))
+           {
+               retval.append( formats[i] );
+               if( i + 1 < formats.length )
+               {
+                   retval.append( ", " );
+               }
+           }
+        }
+        return retval.toString();
+    }
+
+    private static void changeCropBox(PDDocument document, float a, float b, float c, float d)
+    {
+        for (PDPage page : document.getPages())
+        {
+            System.out.println("resizing page");
+            PDRectangle rectangle = new PDRectangle();
+            rectangle.setLowerLeftX(a);
+            rectangle.setLowerLeftY(b);
+            rectangle.setUpperRightX(c);
+            rectangle.setUpperRightY(d);
+            page.setCropBox(rectangle);
+
+        }
+    }
+}
Index: /main/trunk/greenstone2/perllib/util.pm
===================================================================
--- /main/trunk/greenstone2/perllib/util.pm	(revision 32192)
+++ /main/trunk/greenstone2/perllib/util.pm	(revision 32193)
@@ -1716,5 +1716,5 @@
 
 # Used by pdfpstoimg.pl and PDFBoxConverter to create a .item file from
-# a directory containing sequentially numbered images.
+# a directory containing sequentially numbered images (and optional matching sequentially numbered .txt files).
 sub create_itemfile
 {
@@ -1752,9 +1752,24 @@
     print $item_fh "<PagedDocument>\n";
 
+    # In the past, sub create_itemfile() never output txtfile names into the item file (they were left as empty strings), 
+    # only image file names. Now that PDFBox is being customised for GS with the new GS_PDFToImagesAndText.java class to 
+    # create images of each PDF page and extract text for that page if extractable, we can have matching txt files for
+    # each img file. So now we can output txt file names if we're working with txt files.
+    # We just test if a text file exists in the same dir that matches the name of the first image file
+    # if a matching txt file does not exist, don't output txtfile names into the item file
+	
+    my ($tailname, $dirname, $suffix) = &File::Basename::fileparse($firstfile, "\\.[^\\.]+\$"); # relative filenames so no dirname
+    my $txtfilename = &FileUtils::filenameConcatenate($output_dir, $tailname . ".txt");
+    my $hasTxtFile = &FileUtils::fileExists($txtfilename);
+
     foreach my $file (@dir_files){
-	if ($file !~ /\.item/i){
+	if ($file !~ /\.item/i && $file !~ /\.txt/i){
 	    $page_num = page_number($file);
 	    $page_num++ if $starts_at_0; # image numbers start at 0, so add 1
-	    print $item_fh "   <Page pagenum=\"$page_num\" imgfile=\"$file\" txtfile=\"\"/>\n";
+	    if($hasTxtFile) {
+		print $item_fh "   <Page pagenum=\"$page_num\" imgfile=\"$file\" txtfile=\"$page_num.txt\"/>\n";
+	    } else {
+		print $item_fh "   <Page pagenum=\"$page_num\" imgfile=\"$file\" txtfile=\"\"/>\n";
+	    }
 	}
     }
