Index: /gs2-extensions/tesseract/trunk/GETTING-OCR-SUPPORT-FOR-MORE-LANGS.txt
===================================================================
--- /gs2-extensions/tesseract/trunk/GETTING-OCR-SUPPORT-FOR-MORE-LANGS.txt	(revision 34190)
+++ /gs2-extensions/tesseract/trunk/GETTING-OCR-SUPPORT-FOR-MORE-LANGS.txt	(revision 34190)
@@ -0,0 +1,88 @@
+------------------------------------------------------------------------
+README FOR Greenstone USERS: TO SUPPORT ADDITIONAL LANGUAGES FOR OCR
+------------------------------------------------------------------------
+Greenstone can be configured to use Tesseract to OCR images, and use Tika
+in combination with Tesseract to OCR PDFs.
+
+By default, the Greenstone Tesseract extension only comes with support for OCR-ing
+English and Onscreen Display text, as otherwise the extension will become too large.
+
+Tesseract supports OCR for many languages (for the scripts of many languages).
+The supported languages are at https://github.com/tesseract-ocr/tessdata,
+where they're indicated by their official 3 letter language code.
+(You can Google to find the 3 letter lang code for your languages of interest).
+
+To obtain support for other languages, you can do one of:
+
+a. manually download the <3-letter-langcode>.traineddata files for languages
+you want from  https://github.com/tesseract-ocr/tessdata
+
+b. Run the following from the toplevel of your GS3 installation:
+   source ./gs3-setup.sh
+   cd gs2build/ext/tesseract/linux/share/tessdata
+
+Then for each language code, run the following with <3-letter-langcode> adjusted
+accordingly:
+   wget https://github.com/tesseract-ocr/tessdata/raw/master/<3-letter-langcode>.traineddata
+
+c. You can download all the supported languages in one step if you have git
+installed. First move (or remove) the existing "tessdata" folder, before running
+git clone to get all the languages that have OCR support:
+
+   cd gs2build/ext/tesseract/linux/share
+   #rm -rf tessdata
+   mv tessdata tessdata.basic
+   git clone https://github.com/tesseract-ocr/tessdata
+
+
+------------------------------
+Background Information:
+------------------------------
+Greenstone can only index text in documents that contain extractable text.
+Not documents that only have images of text ("photos" of text don't contain
+selectable text).
+
+There is a process called OCR (Optical Character Recognition) to recognise
+any individual characters constituting text represented in images, and thereby
+produce the text in images that otherwise have no extractable text.
+
+Tesseract is OCR software licensed under the Apache 2.0 License. Tesseract can
+be used by Greenstone for OCR-ing images, to thus get text from those images
+which Greenstone can then index for full text searching on that image document.
+
+Tesseract cannot OCR PDFs, only images. However, Apache Tika can work with Tesseract
+(both licensed under the Apache 2.0 License) to OCR PDFs that contain pages
+which are only images of text rather than actual extractable text.
+
+Greenstone can use the combination of Apache Tika and Tesseract to further process
+any PDFs of images of text too, the OCR process producing text that Greenstone can
+index to enable full text searching on the original document (which otherwise
+contained no extractable text, only images of text).
+
+Important Notes:
+
+a. Where OCR is involved in any process, the quality of the OCR-ed text that is
+produced is tightly dependent on the quality of image files that went into the
+process. The higher the DPI (dots per inch) of the images and the clearer the
+legibility of the images of text that go into the digital OCR-ing process, the more
+sensible and accurate the OCR-ed text that results. In cases of poor quality images,
+gibberish will be produced. With average-quality input images, the OCR-ed text is a
+combination of text accurate to the original interspersed occasionally by strange
+characters.
+
+b. OCR is for recognising characters constituting text in images. Characters are
+components of scripts, and there are many language scripts in the world. As a result,
+in order for OCR to recognise the characters that constitute the script of the
+language your document contains, there needs to be support for that language's script
+in the OCR software used, in this case Tesseract.
+
+The languages' scripts that Tesseract supports (indicated by their 3 letter language
+codes) are at https://github.com/tesseract-ocr/tessdata
+
+By default, the Greenstone Tesseract extension only comes with support for OCR-ing
+English and Onscreen Display text, as otherwise the extension will become too large.
+
+To allow the Greenstone Tesseract extension to OCR further languages that
+Tesseract already supports, read the section "TO SUPPORT ADDITIONAL LANGUAGES FOR OCR".
+
+------------------------------------------------------------------------
Index: /gs2-extensions/tesseract/trunk/README.txt
===================================================================
--- /gs2-extensions/tesseract/trunk/README.txt	(revision 34189)
+++ /gs2-extensions/tesseract/trunk/README.txt	(revision 34190)
@@ -19,12 +19,16 @@
 at http://trac.greenstone.org/browser/gs2-extensions/imagemagick/trunk/README
 
+
 1. Find a location on your machine
+
 
 2. Check out the tesseract extension from gs2-extensions
    svn co http://trac.greenstone.org/browser/gs2-extensions/tesseract/trunk tesseract
 
+
 3. Compile it all up (tesseract and dependencies):
    cd tesseract
    ./CASCADE-MAKE.sh
+
 
 4. Open a fresh terminal and check that the tesseract now installed in src/linux/bin works:
@@ -43,26 +47,44 @@
    cat out.txt
 
+If you run Tesseract with the hocr config file, you can get the OCR output in
+nicely formatted html more representative of the input structure:
+
+       tesseract sample.tif hocrtest
+
+The OCR output in html format will be in hocrtest.hocr:
+
+    cat hocrtest.hocr
+
+
 5. If successful,
 
-a. create a folder at the same level as src called tesseract
+ a. create a folder at the same level as src called tesseract
    cd src
    cd ..
    mkdir tesseract
 
-b. COPY the setup files and MOVE the installed folder (src/linux) into the new cut-down tesseract folder:
+ b. COPY the setup files and MOVE the installed folder (src/linux) into the new cut-down tesseract folder:
 
    cp src/setup.ba* tesseract/.
    mv src/linux tesseract/.
 
-c. COPY the TESSERACT-APACHE-LICENSE and LEPTONICA-LICENSE txt files (note it uses
-American spelling!) from src/packages into the cut-down tesseract/linux:
+ c. COPY the TESSERACT-APACHE-LICENSE and LEPTONICA-LICENSE txt files (note it uses
+ American spelling!) from src/packages into the cut-down tesseract/linux:
 
    cp src/packages/*LICENSE.txt tesseract/linux/.
 
-d. REMOVE folder "man" from tesseract/linux:
+ d. Copy the top-level GETTING-OCR-SUPPORT-FOR-MORE-LANGS.txt file into the cutdown tesseract:
+   cp GETTING-OCR-SUPPORT-FOR-MORE-LANGS.txt tesseract/.
+
+ e. REMOVE folder "man" from tesseract/linux:
    rm -rf tesseract/linux/man
+
+ f. REMOVE everything EXCEPT the "tessdata" folder in tesseract/linux/share.
+ (The other things in that  location are either unnecessary or created by tesseract's dependencies).
+
 
 6. Create a tarball of the cut down tesseract folder named tesseract-<os>-<arch>.tar.gz:
    tar -cvzf tesseract-linux-x64.tar.gz tesseract
+
 
 7. (Add/SVN up and) commit that to svn:
@@ -111,5 +133,7 @@
 	           hocr_font_info 0
 		   
-2. In order to have a tessdata/configs/hocr, needed to correct the Tesseract we were
+
+
+In order to have a tessdata/configs/hocr, needed to correct the Tesseract we were
 cascade-making to get it to put the "configs" subfolder inside the installed Tessereact's
 tessdata folder. The source version of tesseract has this folder, but it wasn't getting
Index: /gs2-extensions/tesseract/trunk/src/packages/CASCADE-MAKE/TESSERACT.sh
===================================================================
--- /gs2-extensions/tesseract/trunk/src/packages/CASCADE-MAKE/TESSERACT.sh	(revision 34189)
+++ /gs2-extensions/tesseract/trunk/src/packages/CASCADE-MAKE/TESSERACT.sh	(revision 34190)
@@ -45,28 +45,26 @@
 
 echo "Installing basic tesseract languages support (tessdata)"
-cp $GEXTTESS_DEVEL/packages/tessdata-langs.tar.gz $GEXTTESS_INSTALLED/.
-pushd $GEXTTESS_INSTALLED
+# Untar OCR language support tarball one level above TESSDATA_PREFIX ($GEXTTESS_INSTALLED/shared),
+# Then go into that folder to finish setting up language files.
+cp $GEXTTESS_DEVEL/packages/tessdata-langs.tar.gz $TESSDATA_PREFIX/../.
+pushd $TESSDATA_PREFIX/..
 tar -xvzf tessdata-langs.tar.gz
+# Above creates linux/shared/tessdata-langs folder - move files there into
+# linux/shared/tessdata (i.e. TESSDATA_PREFIX) and delete both tarball and temporary
+# tessdata-langs folder created at current location of one level up from TESSDATA_PREFIX
+mv tessdata-langs/*.traineddata $TESSDATA_PREFIX/.
 rm tessdata-langs.tar.gz
-mkdir -p tessdata/tessconfigs
+rm -rf tessdata-langs
 popd
 
-# Not sure why source package's tessdata didn't get installed in installdir
-# despite exporting TESSDATA_PREFIX at the start at cascade-make process.
-cp -r $package$version/tessdata/configs $GEXTTESS_INSTALLED/tessdata/
-cp $package$version/tessdata/eng.user-patterns $GEXTTESS_INSTALLED/tessdata/.
-cp $package$version/tessdata/eng.user-words $GEXTTESS_INSTALLED/tessdata/.
-cp $package$version/tessdata/tessconfigs/*batch* $GEXTTESS_INSTALLED/tessdata/tessconfigs/.
-cp $package$version/tessdata/tessconfigs/*demo* $GEXTTESS_INSTALLED/tessdata/tessconfigs/.
 
-
-echo "Done installing basic tesseract languages"
-echo "Visit https://github.com/tesseract-ocr/tessdata for a full list of trained language data."
-echo "To download support for any specific language(s), note the 3 letter code of that language"
-echo "Go into your $GEXTTESS_INSTALLED/tessdata and for each language run: "
+echo "Done installing basic tesseract languages for OCR (Optical Character Recognition, to recognise text from images)."
+echo "Visit https://github.com/tesseract-ocr/tessdata for a full list of trained language data for OCR."
+echo "To download OCR support for any specific language(s), note the 3 letter code of that language"
+echo "Go into your $TESSDATA_PREFIX folder and for each language you want OCR abilities for, run: "
 echo "   wget https://github.com/tesseract-ocr/tessdata/raw/master/<3-letter-lang-code>.traineddata"
-echo "To get all languages currently supported by Tesseract, delete"
-echo "$GEXTTESS_INSTALLED/tessdata"
-echo "and in $GEXTTES_INSTALLED run:"
+echo "To get all languages currently supported by Tesseract (beware, this may be a few Gigabytes), delete"
+echo "$TESSDATA_PREFIX"
+echo "and in $GEXTTES_INSTALLED/shared run:"
 echo "   git clone https://github.com/tesseract-ocr/tessdata"
 echo ""
Index: /gs2-extensions/tesseract/trunk/src/setup.bash
===================================================================
--- /gs2-extensions/tesseract/trunk/src/setup.bash	(revision 34189)
+++ /gs2-extensions/tesseract/trunk/src/setup.bash	(revision 34190)
@@ -20,5 +20,5 @@
   
   # tesseract needs the TESSDATA_PREFIX env var set to the languages folder (tessdata)
-  export TESSDATA_PREFIX=$GEXTTESS_INSTALLED/tessdata
+  export TESSDATA_PREFIX=$GEXTTESS_INSTALLED/share/tessdata
 
   export PATH=$GEXTTESS_INSTALLED/bin:$PATH
Index: /gs2-extensions/tesseract/trunk/src/setup.bat
===================================================================
--- /gs2-extensions/tesseract/trunk/src/setup.bat	(revision 34189)
+++ /gs2-extensions/tesseract/trunk/src/setup.bat	(revision 34190)
@@ -36,5 +36,5 @@
 :Success
 rem tesseract needs the TESSDATA_PREFIX env var set to the languages folder (tessdata)
-set TESSDATA_PREFIX="%GEXTTESS_INSTALLED%\tessdata"
+set TESSDATA_PREFIX="%GEXTTESS_INSTALLED%\share\tessdata"
 
 set fulldir=%~dp0
