Changeset 32248

Show
Ignore:
Timestamp:
06.07.2018 21:31:53 (2 weeks ago)
Author:
ak19
Message:

1. Overhaul to the GS-README for the xpdf-tools extension. 2. Moved a lot of verbose comments that explain reasonings behind doing things from gs-CMakeLists.txt and somewhat from XPDFTOOLS.sh

Location:
gs2-extensions/xpdf-tools/trunk/src/packages
Files:
3 modified

Legend:

Unmodified
Added
Removed
  • gs2-extensions/xpdf-tools/trunk/src/packages/CASCADE-MAKE/XPDFTOOLS.sh

    r32246 r32248  
    1919export LDFLAGS="$LDFLAGS -L$GEXTXPDFTOOLS_INSTALLED/lib" 
    2020 
    21 # For Mac OSX, when -static is passed in for linking, we end up with the error "ld: library not found for -lcrt0.o" 
    22 # See https://stackoverflow.com/questions/3801011/ld-library-not-found-for-lcrt0-o-on-osx-10-6-with-gcc-clang-static-flag 
    23 # On Mac OSX, compile it with -Bstatic instead, as per https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x 
    24 # We will pass in the correct static variant as the custom GSDLFLAG_STATIC, so it will be set to either "-static" (for linux) 
    25 # or "-Bstatic" for darwin when CMake starts compiling up xpdf-tools. When the GSDLFLAG_STATIC is set, it will also trigger 
    26 # the branch of the gs-CMakeLists.txt code that does static compiling. If not set, xpdf-tools are compiled against dynamically 
    27 # linked libraries. 
     21# See section D, 4c of the GS-README of this extension. 
    2822if [ "x$GSDLOS" == "xdarwin" ] ; then 
    2923    static_flag=-Bstatic 
     
    10397fi 
    10498 
    105 # If compiling statically, need these above in place of the references to .so files 
     99## If compiling statically make sure the above CMake command contains the following 
     100## with these values: 
    106101#       -DZLIB_LIBRARY=$prefix/lib/libz.a \ 
    107102#       -DPNG_LIBRARY=$prefix/lib/libpng15.a \ 
    108103#       -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \ 
    109 # And throw in -DGSDLFLAG_STATIC_BUILD="-static" for linux or with "-Bstatic" for Macs 
    110 # In place of FREETYPE_LIBRARY, could also try the following (then check built bins 
    111 # by running ldd and file over them): 
    112 ##      -DFREETYPE_DIR=$prefix \ 
     104# and 
     105#       -DGSDLFLAG_STATIC="$static_flag" \ 
    113106 
    114107 
    115 ## When -DFREETYPE_DIR is used in place of -DFREETYPE_LIBRARY above, it makes 
    116 ## xpdf-tools compilation find the "libfreetype.so" (no versioning at end) in 
    117 ## our gs ext. 
    118 ## But -DZLIB_DIR and -DPNG_DIR are ignored because they go unrecognised, whereas 
    119 ## -DZLIB_LIBRARY and -DPNG_LIBRARY are recognised by XpdfTools' CMake files. 
    120 ## -DZLIB_LIBRARY and -DPNG_LIBRARY require full paths. So the first 2 lines of the 
    121 ## following won't work: 
    122 #       -DZLIB_DIR=$prefix \ 
    123 #       -DPNG_DIR=$prefix \ 
    124 #       -DFREETYPE_DIR=$prefix \ 
    125  
    126 ## When specified in the following way, all 3 lines are recognised (freetype, zlib, 
    127 ## png). The 3rd line can alternatively still use "-DFREETYPE_DIR=$prefix" 
     108## For building dynamic xpdf-tools executables, change the above to have the following instead: 
    128109#       -DZLIB_LIBRARY=$prefix/lib/libz.so.1.2.7 \ 
    129110#       -DPNG_LIBRARY=$prefix/lib/libpng15.so.15.30.0 \ 
    130111#       -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.so.6.3.20 \ 
    131 # The above is for building dynamic xpdf-tools executables. For which, also don't 
    132 # pass in -DGSDLFLAG_STATIC_BUILD="..." (remove the line, don't comment it out, nor 
    133 # set the value to "") 
     112# (The 3rd line can alternatively be "-DFREETYPE_DIR=$prefix") 
     113# And remove the following line. Don't comment it out, nor set it to "", but remove it: 
     114#       -DGSDLFLAG_STATIC="$static_flag" \ 
     115 
    134116 
    135117# If compilation was successful, then we don't need cmake binaries anymore when 
  • gs2-extensions/xpdf-tools/trunk/src/packages/GS-README.txt

    r32229 r32248  
    1 ------------------------------------------------------------------- 
    2 Modfications to xpdf-4.00.tar.gz to produce gs-xpdf-4.00.tar.gz 
    3 ------------------------------------------------------------------- 
    4 In order to successfully compile xpdf-tools-4.00 against *dynamically* linked libraries for zlib (and libpng), needed to modify the CMakeLists.txt. The changes. 
    5  
    6     if (PNG_FOUND) 
    7       include_directories("${PNG_INCLUDE_DIRS}") 
    8       add_definitions("${PNG_DEFINITIONS}") 
    9     endif () 
    10     #--------- NEW 
    11     # https://stackoverflow.com/questions/36126333/mingw-cmake-undefined-reference-to-zlib 
    12     if (ZLIB_FOUND) 
    13       include_directories("${ZLIB_INCLUDE_DIRS}") 
    14       add_definitions("${ZLIB_DEFINITIONS}") 
    15     endif () 
    16     #--------- ENDNEW 
    17     if (HAVE_LCMS) 
     1__________________________________________________________ 
     2A. XPDF 
     3__________________________________________________________ 
     4 
     5Xpdf's last mod date is in 2017 and it includes its own pdftohtml utility tool, whereas the old "pdftohtml" tool that GS used was last updated 2013 (and itself made use of Xpdf, possible older versions). 
     6 
     7The tool takes a PDF and produces an HTML file for each page of the PDF, consisting of selectable HTML text overlaid on top of "screenshot" image of the page. (A page's text is not part of the screenshot.) 
     8 
     91. https://www.xpdfreader.com/download.html 
     10 
     11As per the Readme file found in the linux binary of Xpdf Tools, the Xpdf Viewer requires the qt toolkit, but not the Xpdf Tools. Have not read the Install file to confirm whether the same is the case for when compiling the command line tools. (But in that case, can't we just include the tools binary available for all 3 OS, instead of compiling on each platform) 
     12 
     13    - Using Xpdf's pdftohtml tool: 
     14    greenstone@bedrock:~/Downloads/xpdf-tools-linux-4.00/bin64$./pdftohtml -z 1.5 ~/Downloads/ApacheLicence.pdf licence 
     15 
     16        where licence is a folder. 
     17 
     18    - Using Xpdf's pdftotext tool: 
     19    greenstone@bedrock:~/Downloads/xpdf-tools-linux-4.00/bin64$./pdftotext -nopgbrk ~/Downloads/ApacheLicence.pdf ~/Downloads/ApacheLicence.txt 
     20 
     21        where the output text file must be specified with a full path name. 
     22 
     23 
     242. Documentation on Xpdf-Tools: 
     25- https://www.xpdfreader.com/support.html 
     26    for example, the pdftohtml man page: https://www.xpdfreader.com/pdftohtml-man.html 
     27- https://linux.die.net/man/5/xpdfrc 
     28(Configuration flags you can put into ~/.xpdfrc to use as defaults when running xpdf tool commands) 
     29 
     303. We're using Xpdf Tools version: xpdf-tools-linux-4.00 
     31 
     324. We started by working with the ready-made Xpdf-tools binaries available for download from the xpdf site for Win, Linux and Mac. 
     33 
     345. We're now moving to compiling up Xpdf-tools ourselves using CASCADE-MAKE, which we have so far got to successfully compile statically on Linux (LSB environment inclusive) to build working binaries.  
     35 
     36On Mac, I've been unable to get it to produce statically linked libraries: at this stage they're dynamically linked. 
     37 
     38 
     39__________________________________________________________ 
     40B. Mojo::DOM perl package for parsing HTML 
     41__________________________________________________________ 
     42 
     43XPDF's pdftohtml conversion of a single PDF document produces multiple HTML files: one for each page in the source PDF. 
     44We want the output to be "paged_html": a single HTML file that is sectionalised, each section representing a page of the 
     45original PDF. 
     46 
     47We need to be able to parse the many HTML pages produced by XPDF's pdftohtml conversion of a doc, in order to massage the output 
     48into the single sectionalised HTML file. For this we needed a HTML parser package for Perl. 
     49 
     501. Before Dr Bainbridge found Mojo::DOM, he looked at 
     51* https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers 
     52* http://radar.oreilly.com/2014/02/parsing-html-with-perl-2.html 
     53 
     542. Main links for Mojo::DOM 
     55* https://mojolicious.org/perldoc/Mojo/DOM 
     56* https://metacpan.org/pod/Mojo::DOM 
     57    Dependencies: http://deps.cpantesters.org/?module=Mojo%3A%3ADOM;perl=latest 
     58 
     593. Once you've downloaded Mojo::DOM's src, follow Dr Bainbridge's sequence of commands for building the Mojo::DOM CPAN module of perl below. 
     60We'll be using this module to be used for parsing the HTML output by XPDF tool pdftohtml 
     61 
     62 
     63    mkdir cpan 
     64     2020  tar xvzf Mojolicious-7.84.tar.gz  
     65     2021  cd Mojolicious-7.84/ 
     66     2028  perl ./Makefile.PL PREFIX=`pwd`/installed 
     67     2030  make 
     68     2031  make install 
     69     2033  cp -r installed/share/perl/5.18.2 ../cpan 
     70    cd .. 
     71     2044  export PERL5LIB=`pwd`/cpan 
     72 
     73     2053  emacs -nw test.pl 
     74 
     75    #!/usr/bin/perl -w 
     76    add in 'use v5.10;' 
     77      
     78     2054  chmod a+x test.pl 
     79     2055  ./test.pl  
     80 
     81 
     82__________________________________________________________ 
     83C. Compiling Xpdf-Tools: statically or dynamically linked 
     84__________________________________________________________ 
     85 
     86As explained in detail in section D below, gs-CMakeLists.txt allows us to compile xpdf-tools statically (as we've now set it up for by default) or dynamically (as its CMake makefiles were originally set up for). 
     87 
     881. To compile Xpdf-Tools statically, packages/CASCADE-MAKE/XPDFTOOLS.sh should contain: 
     89 
     90    cmake -DCMAKE_BUILD_TYPE=Release \ 
     91        -DCMAKE_INSTALL_PREFIX=$prefix \ 
     92        -DZLIB_LIBRARY=$prefix/lib/libz.a \         # <========= THIS 
     93        -DPNG_LIBRARY=$prefix/lib/libpng15.a \      # <========= THIS 
     94        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \  # <========= THIS 
     95        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \ 
     96        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \ 
     97        -DCMAKE_C_FLAGS="$CFLAGS" \ 
     98        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \ 
     99        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \ 
     100        -DGSDLFLAG_STATIC="$static_flag" \          # <========= THIS 
     101        $GEXT_XPDFTOOLS/packages/$package$version 
     102 
     103In place of FREETYPE_LIBRARY above, could also try the following,  
     104        -DFREETYPE_DIR=$prefix \ 
     105but then check the built binaries by running "ldd" and "file" over them, to make sure they're not referencing any .so dynamic link libraries: 
     106 
     107 
     1082. To compile Xpdf-Tools dynamically and make it find *our* dynamically linked libraries for its helper packages zlib, libpng and freetype, edit packages/CASCADE-MAKE/XPDFTOOLS.sh to contain: 
     109 
     110    cmake -DCMAKE_BUILD_TYPE=Release \ 
     111        -DCMAKE_INSTALL_PREFIX=$prefix \ 
     112        -DZLIB_LIBRARY=$prefix/lib/libz.so.1.2.7 \          # <========= THIS 
     113        -DPNG_LIBRARY=$prefix/lib/libpng15.so.15.30.0 \     # <========= THIS 
     114        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.so.6.3.20 \  # <========= THIS 
     115        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \ 
     116        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \ 
     117        -DCMAKE_C_FLAGS="$CFLAGS" \ 
     118        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \ 
     119        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \ 
     120        $GEXT_XPDFTOOLS/packages/$package$version       # <=== -DGSDLFLAG_STATIC removed 
     121 
     122 
     123 
     124    (1) In the above, you could also set  
     125        -DFREETYPE_DIR=$prefix 
     126    in place of 
     127        -DGSDLFLAG_STATIC="$static_flag" 
     128 
     129    In that case it makes, xpdf-tools compilation find the "libfreetype.so" (no versioning at end) in our gs2-extension. 
     130    After successfully building, make sure to have sourced the gs2-extension's setup.bash before running "ldd" over the 
     131    generated xpdf-tools binaries, in order to let it use the $LD_LIBRARY_PATH we set to find our .so files. 
     132 
     133    (2) Note that there are no equivalent for ZLIB and LIBPNG: doing -DZLIB_DIR=$prefix or -DPNG_DIR=$prefix will be 
     134    ineffective, as neither are recognised by xpdf-tools' CMake set up. 
     135 
     136__________________________________________________________ 
     137D. How we got Xpdf-Tools to compile using CASCADE-MAKE 
     138__________________________________________________________ 
     139 
     140The process: 
     141 
     1421. We set up a CASCADE-MAKE GS2-extension "xpdf-tools" at trac.greenstone.org/browser/gs2-extensions/xpdf-tools/trunk/src 
     143Be aware that its lowercased "cascade-make" subfolder is an svn external, the original is at http://trac.greenstone.org/browser/other-projects/cascade-make/trunk/ 
     144 
     145So far, this CASCADE-MAKE project includes the Xpdf-Tools source tarball, its helper packages zlib, libpng and freetype, as well as CMake to compile the Xpdf-Tools source code. 
     146The next step is to include JPEG and TIFF libraries too. 
     147 
     1482a. We downloaded the Xpdf-Tools source tarball, xpdf-4.00.tar.gz, from the xpdf site at https://www.xpdfreader.com/download.html under section "Download the Xpdf source code". 
     149 
     150The xpdf-tools source code tarball consists of the source for Xpf-tools and Xpdf (Xpdf-Reader). The Xpdf-Reader additionally requires Qt to build and run, but we don't want the Xpdf-Reader, just Xpdf-Tools. 
     151 
     152b. Compiling Xpdf-Tools fron source and running them requires the following packages and libraries, as per the xpdf-tools source code INSTALL file: 
     153 
     154To build xpdf-tools: 
     155- CMake 2.8.8 or newer  
     156 
     157Libraries to link against and used by xpdf-tools: 
     158- FreeType 2.0.5 or newer 
     159- libpng (for pdftoppm and pdftohtml) 
     160- zlib (for pdftoppm and pdftohtml) 
     161 
     162 
     1633. Compilation of xpdf-tools worked with CMake 3.11.4 on the linux resnet machine. However, CMake 3.11.3 itself failed to compile in the LSB environment and on the Mac Mountain Lion machine because of a version incompatibility between the older g++ installed there and the advanced version of CMake 3.11.4. 
     164 
     165CMake version 3.9.6 however is supposed to be compatible with older versions of g++, as per https://stackoverflow.com/questions/47886400/cmake-configure-error-in-3-10-1-but-not-in-3-9-6 
     166To avoid installing newer versions of g++ and clang in the LSB virtual machine and the Mac, I've shifted the CMake version back to version 3.9.6, still  
     167 
     168 
     1694a. On building xpdf-tools to work with dynamically linked libs found anywhere. 
     170 
     171If compiling xpdf-tools against dynamic linked libraries for these packages, then the basic CMake command in packages/CASECADE-MAKE/XPDFTOOLS.sh can look like: 
     172    cmake -DCMAKE_BUILD_TYPE=Release \ 
     173        -DCMAKE_INSTALL_PREFIX=$prefix \ 
     174        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \ 
     175        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \ 
     176        -DCMAKE_C_FLAGS="$CFLAGS" \ 
     177        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \ 
     178        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \ 
     179        $GEXT_XPDFTOOLS/packages/$package$version   # Note: no -DGSDLFLAG_STATIC=... 
     180 
     181With the above, the xpdf-tools source code and its make files work out of the box. 
     182 
     1834b. On building xpdf-tools to work with the dynamically linked libs for freetype libpng, zlib that we produce when cascade-making the xpdf-tools gs2-extension. 
     184 
     185Since we're compiling up freetype, libpng and zlib packages as part of the Xpdf-Tools GS2-extension with CASCADE-MAKE, the next step was to compile xpdf-tools by dynamically linking against our .so files for these 3 libraries. To do so, XPDFTOOL.sh should have the following changes 
     186 
     187    (1) set up CFLAGS, CXXFLAGS, CPPFLAGS and LDFLAGS to help linkage of xpdf-tools find our .so versions of the necessary libs: 
     188 
     189    export CFLAGS="$CFLAGS -I$GEXTXPDFTOOLS_INSTALLED/include -I$GEXTXPDFTOOLS_INSTALLED/include/libpng15" 
     190    export CPPFLAGS="$CPPFLAGS -I$GEXTXPDFTOOLS_INSTALLED/include -I$GEXTXPDFTOOLS_INSTALLED/include/libpng15" 
     191    export CXXFLAGS="$CXXFLAGS -I$GEXTXPDFTOOLS_INSTALLED/include -I$GEXTXPDFTOOLS_INSTALLED/include/libpng15" 
     192    export LDFLAGS="$LDFLAGS -L$GEXTXPDFTOOLS_INSTALLED/lib" 
     193 
     194    (2) The CMAKE command we run must pass the full paths to the actual .so library files (the ones with specific 
     195    versions in their files names) rather than the symbolically linked generally-named .so files (the latter won't 
     196    be found when building xpdf-tools and CMake will try to look for the .so library files elsewhere on the system): 
     197 
     198    cmake -DCMAKE_BUILD_TYPE=Release \ 
     199        -DCMAKE_INSTALL_PREFIX=$prefix \ 
     200        -DZLIB_LIBRARY=$prefix/lib/libz.so.1.2.7 \              # <========= NEW 
     201        -DPNG_LIBRARY=$prefix/lib/libpng15.so.15.30.0 \         # <========= NEW 
     202        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.so.6.3.20 \      # <========= NEW 
     203        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \ 
     204        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \ 
     205        -DCMAKE_C_FLAGS="$CFLAGS" \ 
     206        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \ 
     207        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \ 
     208        $GEXT_XPDFTOOLS/packages/$package$version   # Again: no -DGSDLFLAG_STATIC=... 
     209 
     210Further, the "xpdf/CMakeLists.txt" file within the xpdf-4.00.tar.gz source code tarball needs to be modified to refer to ZLIB_LIBRARIES when linking pdftops and pdftoppm. The linking commands for *both* the "pdftops" and "pdftoppm" executable targets in xpdf/CMakeLists.txt should look like the following,  
     211 
     212        target_link_libraries(pdftoppm goo fofi splash 
     213                        ${FREETYPE_LIBRARY} ${FREETYPE_OTHER_LIBS} 
     214                        ${DTYPE_LIBRARY} 
     215                        ${LCMS_LIBRARY} 
     216            ${ZLIB_LIBRARIES})              # <========= NEW 
     217 
     218 
     219    (3) Since CMakeLists.txt has been modified, we initially renamed the xpdf src tarball to gs-xpdf-4.00.tar.gz. 
     220    However, the current version works with the regular downloaded xpdf-4.00.tar.gz tarball. But after extraction, 
     221    XPDFTOOLS.sh copies across the custom packages/gs-CMakeLists.txt into the extracted tarball's xpdf subdirectory, 
     222    renaming the file as CMakeLists.txt (so the path to it becomes "xpdf-4.00/xpdf/CMakeLists.txt"). In XPDFTOOLS.sh:    
     223 
     224    # patch the original tarball with our custom makefile 
     225    if [[ -d "$package$version/xpdf" && -f "gs-CMakeLists.txt" ]]; then 
     226        echo "*******************************************************************" 
     227        echo "Using our custom gs-CMakeLists.txt instead of the one included in $package$version" 
     228        echo "Renaming gs-CMakeLists.txt to $package$version/xpdf/CMakeLists.txt" 
     229        echo "*******************************************************************" 
     230 
     231        cp "gs-CMakeLists.txt" "$package$version/xpdf/CMakeLists.txt" 
     232    fi 
     233 
     234 
     2354c. On building static xpdf-tools binaries using the static *.a freetype libpng, zlib libraries that we produce when cascade-making the xpdf-tools gs2-extension. 
     236 
     237In order to compile up xpdf-tools *statically*, so that it builds against the static *.a libraries of freetype, libpng and zlib that we produce during the gs2-extension's CASCADE-MAKE process, we have to make further modifications.  
     238 
     239    (1) First, the XPDFTOOLS.sh cascade-make file should pass the full paths to the actual (non-symbolic link) .a file for each library. 
     240    A custom GS flag, GSDLFLAG_STATIC, is also invented in gs-CMakeLists.txt and assigned "-static for linux 
     241    and "-Bstatic" for Mac, to pass in during the linking stage of building xpdf-tools. 
     242 
     243    For Mac OSX, when -static is passed in for linking as on linux, this produced the error 
     244    "ld: library not found for -lcrt0.o" during the build of the xpdf-tools package. For information, see 
     245    https://stackoverflow.com/questions/3801011/ld-library-not-found-for-lcrt0-o-on-osx-10-6-with-gcc-clang-static-flag 
     246    The page https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x mentions compiling 
     247    with -Bstatic on Mac OSX instead. To do so, XPDFTOOLS.sh passes in the GSDLFLAG_STATIC set to either 
     248    "-static" (for linux) or "-Bstatic" for darwin. 
     249    However the last mentioned stackoverflow page also says that -Bstatic is a no-op, and this appears to be 
     250    the case when "otool -L" is run over the generated xpdf-tools binaries: the binaries are all dynamically 
     251    linked. Although they're finding our .so files of freetype, libpng and zlib, they're not finding the .a 
     252    versions, even though XPDFTOOLS.sh tries to point gs-CMakeLists.txt to the correct .a files.  
     253 
     254    The new modifications to XPDFTOOLS.sh: 
     255 
     256    if [ "x$GSDLOS" == "xdarwin" ] ; then 
     257        static_flag=-Bstatic 
     258    else 
     259        static_flag=-static 
     260    fi 
     261 
    18262    ... 
    19  
    20  
    21 Note that the xpdftools binaries produced in this fashion are still linked to the system zlib and libpng rather than the ones we're compiling up as part of the GEXT XPDFTOOLS extension. But just to get the compilation to succeed with the references to our local library in XPDFTOOLS.sh below (a step toward getting the statically linked xpdftools binaries to compile), we needed the above changes. 
    22  
    23     cmake -DCMAKE_BUILD_TYPE=Release \ 
    24         -DCMAKE_INSTALL_PREFIX=$prefix \ 
    25         -DZLIB_LIBRARY=$prefix/lib/libz.a \ <======== 
    26         -DPNG_LIBRARY=$prefix/lib/libpng15.a \ <======== 
    27         -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \ <======== 
    28         -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \ 
    29         -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \ 
    30         -DCMAKE_C_FLAGS="$CFLAGS" \ 
    31         -DCMAKE_CXX_FLAGS="$CXXFLAGS" \ 
    32         -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \ 
    33         -DPNG_LINK_FLAGS="-lpng15" \ 
     263    cmake -DCMAKE_BUILD_TYPE=Release \ 
     264        -DCMAKE_INSTALL_PREFIX=$prefix \ 
     265        -DZLIB_LIBRARY=$prefix/lib/libz.a \                 # <========= MODIFIED TO .a 
     266        -DPNG_LIBRARY=$prefix/lib/libpng15.a \              # <========= MODIFIED TO .a 
     267        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \          # <========= MODIFIED TO .a 
     268        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \ 
     269        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \ 
     270        -DCMAKE_C_FLAGS="$CFLAGS" \ 
     271        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \ 
     272        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \ 
     273        -DGSDLFLAG_STATIC="$static_flag" \                  # <========= NEW 
    34274        $GEXT_XPDFTOOLS/packages/$package$version 
    35275 
    36  
    37 ------------------------------------------------------------------- 
    38 Redundant CMakeLists.txt files 
    39 ------------------------------------------------------------------- 
    40 I've also committed work in progress versions of CMakeLists.txt 
    41 - CMakeLists.txt.orig: the original version of this file in the unmodified xpdf-4.00.tar.gz 
    42 - CMakeLists.txt.static1_notworking: Sets up CMake settings for static compilation. Which then required some further changes to stop compilation from complaining about undefined references to thread mutex or something. (-lpthread/pthread library was necessary, but needed to set this the CMake way). Not a working version. 
    43 - CMakeLists.txt.static2_notworking: more changes in an attempt to bypass the errors about undefined references to png (and 'inflate' of zlib). Still not working. It's this version that briefly attempted to use the override to PNG_LINK_FLAGS I am setting in XPDFTOOLS.sh (the line containing: -DPNG_LINK_FLAGS="-lpng15" \). Not sure why it never got set. 
    44 - CMakeLists.txt: the version of this file first committed with the modified gs-xpdf-4.00.tar.gz. For getting xpdf-tools to compile with dynamic linking and with the current setup of XPDFTOOLS.sh 
    45  
    46 There are comments in the modified CMakeLists.txt file above to provide URLs explaining why I made various changes. 
    47  
     276    (2) Our customised gs-CMakeLists.txt file now checks for this flag GSDLFLAG_STATIC being set and, if it is, 
     277    uses it during the linking stage. As in (1) above, it will be set to "-static" for Linux and "-Bstatic" for Mac. 
     278     
     279    - When the flag is set, the linking flags passed into each occurrence of target_link_libraries() in 
     280    gs-CMakeLists.txt is moreover manually written in the form of "-static -l<libs>" rather than using 
     281    the default linking commands inherited from the original CMakeLists.txt. 
     282    - If GSDLFLAG_STATIC isn't set, then we don't build statically, and the linking flags passed to each 
     283    target_link_libraries() are mostly the original ones. 
     284 
     285    For example, 
     286 
     287        if(GSDLFLAG_STATIC) 
     288            target_link_libraries(pdftoppm goo fofi splash 
     289              ${GSDLFLAG_STATIC} -lfreetype ${DTYPE_LIBRARY} ${LCMS_LIBRARY} -lz -lm -lc -lpthread) 
     290        else () 
     291            target_link_libraries(pdftoppm goo fofi splash 
     292                            ${FREETYPE_LIBRARY} ${FREETYPE_OTHER_LIBS} 
     293                            ${DTYPE_LIBRARY} 
     294                           ${LCMS_LIBRARY} 
     295                    ${ZLIB_LIBRARIES}) 
     296        endif () 
     297 
     298    DETAILED EXPLANATION: 
     299    We found that when building *statically*, gs-CMakeLists.txt needed to NOT use the PNG_LIBRARIES, ZLIB_LIBRARIES 
     300    and FREETYPE_LIBRARY in its linker commands, target_link_libraries(), as doing so produced partially dynamic 
     301    xpdf-tools executables which were moreover BROKEN. They wouldn't run, and in fact attempting to run an xpdf-tool, 
     302    like "./pdftohtml", would produce a file not found error. Something like "bash: no such file or directory". 
     303 
     304    Online discussions mentioned that this generally happened when attempting to run 32 bit executables on 64 bit 
     305    linux when 32 bit loaders are not installed. (In such cases, the solution was to apt-get install some 32 bit package.) 
     306    However, our broken binaries were all 64 bit, as indicated when running the "file" command on them. However, their 
     307    being further partially dynamically linked executables didn't imply that they would be broken, as we were eventually 
     308    able to produce partially dynamic executables that did work, before solving static linking altogether. 
     309 
     310    The real issue was that including references to  ${FREETYPE_LIBRARY} ${FREETYPE_OTHER_LIBS}, ${PNG_LIBRARIES} and 
     311    ${ZLIB_LIBRARIES} in any target_link_libraries() resulted in the wrong linking command producing broken binaries. 
     312 
     313    Doing the regular target_link_libraries() in static mode results in building with  
     314    "-Wl,-Bstatic -lfreetype -lpng15 -lz -Wl,-Bdynamic -lpthread" at end of link line 
     315    and produces broken binaries for pdftohtml/pdftoppm/pdftops/pdftopng. 
     316 
     317    Note that PNG_LIBRARIES includes zlib/lz: "-lpng -lz", and along with freetype, 
     318    these are linked statically. However, Threads/lpthread is included as a dynamically 
     319    linked library instead of including a .a (regardless of whether it's appended 
     320    as -lpthread or Threads::Threads in the target_link_libraries()), contributing to 
     321    the pdfhtml binary produced being a partially static, partially dynamic one, 
     322    so a dynamic executable overall. 
     323 
     324    The order of dynamic .so files listed by ldd in the broken static binary of pdftohtml differs from 
     325    a manually statically linked working version of pdftohtml, and seems to be the only difference 
     326    between the two in ldd's output. Not using "-Wl,-Bstatic" and using -static (-Bstatic on Mac) 
     327    in its place creates a partially static dynamic executable that isn't broken, whereas 
     328    additionally removing "-Wl,-Bdynamic -lpthread" and replacing it with -lpthread 
     329    moreover produces a working pdftohtml that is a fully static linked executable. 
     330 
     331    The inclusion of the math lib and c lib (lm and lc) in the final link command 
     332    are to completely bypass the remaining .so dependencies that were present in 
     333    the executable and produce the fully static executable. The lm and lc libs were referenced 
     334    by all xpdf-tool binaries (as indicated when generating dynamic ones and running ldd over them) 
     335    but Dr Bainbridge said that -lm and -lc were some libs passed in by the compiler by default, 
     336    which would explain why explicitly setting them for some xpdftools and not other may not have 
     337    mattered. 
     338 
     339NOTES: 
     340Initial attempts at modifying gs-CMakeLists.txt for static compiling that proved to be unnecessary: 
     341 
     342    (1) Setting -static globally doesn't have a useful effect. 
     343 
     344    # We want to build static xpdf-tools binaries. See 
     345    # https://stackoverflow.com/questions/24648357/compiling-a-static-executable-with-cmake 
     346    # Want to make the min number of changes for building statically, so using the way 
     347    # below. Beware, must *append* "-static" to existing CMAKE_EXE_LINKER_FLAGS=LD_FLAGS 
     348    ##SET(CMAKE_FIND_LIBRARY_SUFFIXES ".a") 
     349    ##SET(BUILD_SHARED_LIBS OFF) 
     350    ##SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static") 
     351 
     352    The above 3 lines just add a -static before the "-O2 -Wall -fPIC -rdynamic ..." during linking, such as below. 
     353    But they have no further effect on whether static building actually succeeds or not. The only effective static 
     354    linking command (for Linux so far) was to pass -static in the target_link_libraries() command followed by the 
     355    "-l<libname>" for each library in the correct order. 
     356 
     357---- 
     358/usr/bin/c++  -I/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/include  -I/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/include -I/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/include/libpng15 -O3 -Wall -fPIC  -L/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/lib  -L/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/lib -static ***** <- HERE ****** -O2 -Wall -fPIC -rdynamic CMakeFiles/pdftohtml.dir/HTMLGen.cc.o CMakeFiles/pdftohtml.dir/SplashOutputDev.cc.o CMakeFiles/pdftohtml.dir/TextOutputDev.cc.o CMakeFiles/pdftohtml.dir/pdftohtml.cc.o CMakeFiles/xpdf_objs.dir/AcroForm.cc.o CMakeFiles/xpdf_objs.dir/Annot.cc.o CMakeFiles/xpdf_objs.dir/Array.cc.o CMakeFiles/xpdf_objs.dir/BuiltinFont.cc.o CMakeFiles/xpdf_objs.dir/BuiltinFontTables.cc.o CMakeFiles/xpdf_objs.dir/Catalog.cc.o CMakeFiles/xpdf_objs.dir/CharCodeToUnicode.cc.o CMakeFiles/xpdf_objs.dir/CMap.cc.o CMakeFiles/xpdf_objs.dir/Decrypt.cc.o CMakeFiles/xpdf_objs.dir/Dict.cc.o CMakeFiles/xpdf_objs.dir/Error.cc.o CMakeFiles/xpdf_objs.dir/FontEncodingTables.cc.o CMakeFiles/xpdf_objs.dir/Form.cc.o CMakeFiles/xpdf_objs.dir/Function.cc.o CMakeFiles/xpdf_objs.dir/Gfx.cc.o CMakeFiles/xpdf_objs.dir/GfxFont.cc.o CMakeFiles/xpdf_objs.dir/GfxState.cc.o CMakeFiles/xpdf_objs.dir/GlobalParams.cc.o CMakeFiles/xpdf_objs.dir/JArithmeticDecoder.cc.o CMakeFiles/xpdf_objs.dir/JBIG2Stream.cc.o CMakeFiles/xpdf_objs.dir/JPXStream.cc.o CMakeFiles/xpdf_objs.dir/Lexer.cc.o CMakeFiles/xpdf_objs.dir/Link.cc.o CMakeFiles/xpdf_objs.dir/NameToCharCode.cc.o CMakeFiles/xpdf_objs.dir/Object.cc.o CMakeFiles/xpdf_objs.dir/OptionalContent.cc.o CMakeFiles/xpdf_objs.dir/Outline.cc.o CMakeFiles/xpdf_objs.dir/OutputDev.cc.o CMakeFiles/xpdf_objs.dir/Page.cc.o CMakeFiles/xpdf_objs.dir/Parser.cc.o CMakeFiles/xpdf_objs.dir/PDFDoc.cc.o CMakeFiles/xpdf_objs.dir/PDFDocEncoding.cc.o CMakeFiles/xpdf_objs.dir/PSTokenizer.cc.o CMakeFiles/xpdf_objs.dir/SecurityHandler.cc.o CMakeFiles/xpdf_objs.dir/Stream.cc.o CMakeFiles/xpdf_objs.dir/TextString.cc.o CMakeFiles/xpdf_objs.dir/UnicodeMap.cc.o CMakeFiles/xpdf_objs.dir/UnicodeTypeTable.cc.o CMakeFiles/xpdf_objs.dir/UTF8.cc.o CMakeFiles/xpdf_objs.dir/XFAForm.cc.o CMakeFiles/xpdf_objs.dir/XRef.cc.o CMakeFiles/xpdf_objs.dir/Zoox.cc.o  -o pdftohtml ../goo/libgoo.a ../fofi/libfofi.a ../splash/libsplash.a -static -lfreetype -lpng -lz -lm -lc -lpthread  
     359---- 
     360 
     361    (2) Threads::Threads instead of -lpthread results in a partially dynamic executable. 
     362 
     363    # The original, unmodified CMakeLists.txt was not set up sufficiently 
     364    # for static compilation of xpdf-tools. As a result, compile would first fail 
     365    # with errors about undefined refs to mutex / lpthread. 
     366    # When building xpdf-tools statically, need to add the following 2 lines as well 
     367    # as append "Threads::Threads" to the end of each "target_link_libraries(<list>)" 
     368    # See https://stackoverflow.com/questions/1620918/cmake-and-libpthread 
     369    # found googling cmake and "-lpthread" (pthread) after ERRORS to do with this, like: 
     370    #   undefined reference to `pthread_mutex_unlock' 
     371    ##set(THREADS_PREFER_PTHREAD_FLAG ON) 
     372    ##find_package(Threads REQUIRED) 
     373 
     374    In instances when compilation was successful, including the above 2 lines in combination with "Threads::Threads" 
     375    as the final argument to every target_link_libraries(...) occurrence in gs-CMakeLists.txt would only manage to 
     376    produce partially dynamically linked xpdftools binaries. (Depending on what the linking command was when building 
     377    Xpdf-Tools, the partially dynamically linked executables may work or may be broken. See explanation further above.) 
     378    We wanted fully statically linked binaries, for which we needed to pass in "-lpthread" as the trailing argument 
     379    to each target_link_libraries(...). So without either, compilation will fail. However, with "Threads::Threads" 
     380    the binaries weren't fully static, whereas with -lpthread the xpdftools executables were fully static as CMake no 
     381    longer tried to link against a dynamic Threads library. 
     382 
     383 
     384(5) To view the unmodified CMakeLists.txt included in the xpdf-4.00 source code tarball, untar it and look for its "xpdf/CMakeLists.txt" (not the toplevel file of the same name). 
     385Run a 'diff' against gs-CMakeLists.txt to see further differences, such as debug statements and comments. Most comments have been removed and placed into this readme file instead. 
     386 
     387 
     388(6) When CASCADE-MAKE is run on the xpdf-tools GS2-extension, it first compiles up CMake, needed to compile up xpdf-tools. 
     389Unlike the library packages like freetype, libpng and zlib that we also build for xpdf-tools as part of this gs2-extension, CMake's build products don't need to be included in the distribution tarball of our built xpdf-tools executables. 
     390 
     391There's a "move-cmake.sh" script in the xpdf-tools gs2-extension that can be run with the "away" and "back" options to move the CMake stuff out of the way (into a "devel" folder) after successfully building xpdf binaries and that can also be run to move them back if wanting to recompile. 
     392 
     393The script can be run manually, but it's also run by the extension: 
     394- packages/CASCADE-MAKE/XPDFTOOLS.sh runs "move-cmake.sh away" after xpdf-tools has been built, so that the extension's install location is ready for tarring up for distribution. 
     395- When recompiling the xpdf-tools extenion, the CASCADE-MAKE process will run packages/CASCADE-MAKE/CMAKE.sh file which in turn runs "move-cmake.sh back" if there's a prebuilt CMake which had earlier been moved out of the way. 
     396 
     397 
  • gs2-extensions/xpdf-tools/trunk/src/packages/gs-CMakeLists.txt

    r32247 r32248  
    1010 
    1111#======================================================================== 
    12 # This is a customised version of Xpdf Tools' CMake file. 
    13 # Customised for Greenstone. 
     12# This is a customised version of Xpdf Tools' CMake file for Greenstone 
     13# where we want to build static xpdf-tools executables. 
     14# Refer to the GS-README.txt file for the Greenstone "xpdf-tools" 
     15# gs2-extension and read section D for the GS specific customisations 
     16# in this file and the reasoning behind them, most of which have to do 
     17# with attempting to successfully build xpdf-tools statically. 
    1418#======================================================================== 
    1519 
     
    2125 
    2226 
    23 # For Mac OSX, when -static is passed in for linking, we end up with the error "ld: library not found for -lcrt0.o" 
    24 # See https://stackoverflow.com/questions/3801011/ld-library-not-found-for-lcrt0-o-on-osx-10-6-with-gcc-clang-static-flag 
    25 # On Mac OSX, compile it with -Bstatic, as per https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x 
    26 # XPDFTOOLS.sh will have set the GSDLFLAG_STATIC to either "-static" (for linux) or "-Bstatic" for darwin 
     27# When compiling statically, XPDFTOOLS.sh will have set the custom 
     28# GSDLFLAG_STATIC flag to either "-static" for linux or "-Bstatic" for darwin 
    2729 
    2830if(GSDLFLAG_STATIC) 
     
    131133endif () 
    132134 
    133 # We want to build static xpdf-tools binaries. See 
    134 # https://stackoverflow.com/questions/24648357/compiling-a-static-executable-with-cmake 
    135 # Want to make the min number of changes for building statically, so using the way 
    136 # below. Beware, must *append* "-static" to existing CMAKE_EXE_LINKER_FLAGS=LD_FLAGS 
    137 ##SET(CMAKE_FIND_LIBRARY_SUFFIXES ".a") 
    138 ##SET(BUILD_SHARED_LIBS OFF) 
    139 ##SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static") 
    140  
    141 # The original, unmodified CMakeLists.txt was not set up sufficiently 
    142 # for static compilation of xpdf-tools. As a result, compile would first fail 
    143 # with errors about undefined refs to mutex / lpthread. 
    144 # When building xpdf-tools statically, need to add the following 2 lines as well 
    145 # as append "Threads::Threads" to the end of each "target_link_libraries(<list>)" 
    146 # See https://stackoverflow.com/questions/1620918/cmake-and-libpthread 
    147 # found googling cmake and "-lpthread" (pthread) after ERRORS to do with this, like: 
    148 #   undefined reference to `pthread_mutex_unlock' 
    149 ##set(THREADS_PREFER_PTHREAD_FLAG ON) 
    150 ##find_package(Threads REQUIRED) 
    151  
    152135#--- pdftops 
    153136 
     
    222205                          ${LCMS_LIBRARY} ${PNG_LIBRARIES}) 
    223206  endif () 
    224 # Doing the regular target_link_libraries() in static mode results in building with  
    225 # "-Wl,-Bstatic -lfreetype -lpng15 -lz -Wl,-Bdynamic -lpthread" at end of link line 
    226 # and produces broken binaries for pdftohtml/pdftoppm/pdftops/pdftopng. 
    227 # Note that PNG_LIBRARIES includes zlib/lz: "-lpng -lz", and along with freetype, 
    228 # these are linked statically. However, Threads/lpthread is included as a dynamically 
    229 # linked library instead of including a .a (regardless of whether it's appended 
    230 # as -lpthread or Threads::Threads in else above), contributing to the pdfhtml binary 
    231 # produced being a partially static, partially dynamic one, 
    232 # so a dynamic executable overall. 
    233 # The order of dynamic .so files listed by ldd in the broken binary differs from 
    234 # a manually linked working version of pdftohtml, and seems to be the only difference 
    235 # between the two in ldd output. Not using "-Wl,-Bstatic" and using -static (-Bstatic on Mac) 
    236 # in its place creates a partially static dynamic executable that isn't broken, whereas 
    237 # additionally removing "-Wl,-Bdynamic -lpthread" and replacing it with -lpthread 
    238 # moreover produces a working pdftohtml that is a fully static linked executable. 
    239 # The inclusion of the math lib and c lib (lm and lc) in the final link command 
    240 # are to completely bypass the remaining .so dependencies that were present in 
    241 # the executable and produce the fully static executable. 
    242 # Not sure what colormanager (lcms) and typelibrary, or how they're written in -l 
    243 # form as they were not present when the original unmodified static linking command 
    244 # was run, so they've not been included in the replacement static linking command.  
    245207  install(TARGETS pdftohtml RUNTIME DESTINATION bin) 
    246208  install(FILES ${PROJECT_SOURCE_DIR}/doc/pdftohtml.1 DESTINATION man/man1)