Changeset 32248 for gs2-extensions


Ignore:
Timestamp:
2018-07-06T21:31:53+12:00 (6 years ago)
Author:
ak19
Message:
  1. Overhaul to the GS-README for the xpdf-tools extension. 2. Moved a lot of verbose comments that explain reasonings behind doing things from gs-CMakeLists.txt and somewhat from XPDFTOOLS.sh
Location:
gs2-extensions/xpdf-tools/trunk/src/packages
Files:
3 edited

Legend:

Unmodified
Added
Removed

  • gs2-extensions/xpdf-tools/trunk/src/packages/CASCADE-MAKE/XPDFTOOLS.sh

    r32246 r32248  
    1919export LDFLAGS="$LDFLAGS -L$GEXTXPDFTOOLS_INSTALLED/lib"
    2020
    21 # For Mac OSX, when -static is passed in for linking, we end up with the error "ld: library not found for -lcrt0.o"
    22 # See https://stackoverflow.com/questions/3801011/ld-library-not-found-for-lcrt0-o-on-osx-10-6-with-gcc-clang-static-flag
    23 # On Mac OSX, compile it with -Bstatic instead, as per https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x
    24 # We will pass in the correct static variant as the custom GSDLFLAG_STATIC, so it will be set to either "-static" (for linux)
    25 # or "-Bstatic" for darwin when CMake starts compiling up xpdf-tools. When the GSDLFLAG_STATIC is set, it will also trigger
    26 # the branch of the gs-CMakeLists.txt code that does static compiling. If not set, xpdf-tools are compiled against dynamically
    27 # linked libraries.
     21# See section D, 4c of the GS-README of this extension.
    2822if [ "x$GSDLOS" == "xdarwin" ] ; then
    2923    static_flag=-Bstatic
    … …  
    10397fi
    10498
    105 # If compiling statically, need these above in place of the references to .so files
     99## If compiling statically make sure the above CMake command contains the following
     100## with these values:
    106101#       -DZLIB_LIBRARY=$prefix/lib/libz.a \
    107102#       -DPNG_LIBRARY=$prefix/lib/libpng15.a \
    108103#       -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \
    109 # And throw in -DGSDLFLAG_STATIC_BUILD="-static" for linux or with "-Bstatic" for Macs
    110 # In place of FREETYPE_LIBRARY, could also try the following (then check built bins
    111 # by running ldd and file over them):
    112 ##      -DFREETYPE_DIR=$prefix \
     104# and
     105#       -DGSDLFLAG_STATIC="$static_flag" \
    113106
    114107
    115 ## When -DFREETYPE_DIR is used in place of -DFREETYPE_LIBRARY above, it makes
    116 ## xpdf-tools compilation find the "libfreetype.so" (no versioning at end) in
    117 ## our gs ext.
    118 ## But -DZLIB_DIR and -DPNG_DIR are ignored because they go unrecognised, whereas
    119 ## -DZLIB_LIBRARY and -DPNG_LIBRARY are recognised by XpdfTools' CMake files.
    120 ## -DZLIB_LIBRARY and -DPNG_LIBRARY require full paths. So the first 2 lines of the
    121 ## following won't work:
    122 #       -DZLIB_DIR=$prefix \
    123 #       -DPNG_DIR=$prefix \
    124 #       -DFREETYPE_DIR=$prefix \
    125 
    126 ## When specified in the following way, all 3 lines are recognised (freetype, zlib,
    127 ## png). The 3rd line can alternatively still use "-DFREETYPE_DIR=$prefix"
     108## For building dynamic xpdf-tools executables, change the above to have the following instead:
    128109#       -DZLIB_LIBRARY=$prefix/lib/libz.so.1.2.7 \
    129110#       -DPNG_LIBRARY=$prefix/lib/libpng15.so.15.30.0 \
    130111#       -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.so.6.3.20 \
    131 # The above is for building dynamic xpdf-tools executables. For which, also don't
    132 # pass in -DGSDLFLAG_STATIC_BUILD="..." (remove the line, don't comment it out, nor
    133 # set the value to "")
     112# (The 3rd line can alternatively be "-DFREETYPE_DIR=$prefix")
     113# And remove the following line. Don't comment it out, nor set it to "", but remove it:
     114#       -DGSDLFLAG_STATIC="$static_flag" \
     115
    134116
    135117# If compilation was successful, then we don't need cmake binaries anymore when

  • gs2-extensions/xpdf-tools/trunk/src/packages/GS-README.txt

    r32229 r32248  
    1 -------------------------------------------------------------------
    2 Modfications to xpdf-4.00.tar.gz to produce gs-xpdf-4.00.tar.gz
    3 -------------------------------------------------------------------
    4 In order to successfully compile xpdf-tools-4.00 against *dynamically* linked libraries for zlib (and libpng), needed to modify the CMakeLists.txt. The changes.
    5 
    6     if (PNG_FOUND)
    7       include_directories("${PNG_INCLUDE_DIRS}")
    8       add_definitions("${PNG_DEFINITIONS}")
    9     endif ()
    10     #--------- NEW
    11     # https://stackoverflow.com/questions/36126333/mingw-cmake-undefined-reference-to-zlib
    12     if (ZLIB_FOUND)
    13       include_directories("${ZLIB_INCLUDE_DIRS}")
    14       add_definitions("${ZLIB_DEFINITIONS}")
    15     endif ()
    16     #--------- ENDNEW
    17     if (HAVE_LCMS)
     1__________________________________________________________
     2A. XPDF
     3__________________________________________________________
     4
     5Xpdf's last mod date is in 2017 and it includes its own pdftohtml utility tool, whereas the old "pdftohtml" tool that GS used was last updated 2013 (and itself made use of Xpdf, possible older versions).
     6
     7The tool takes a PDF and produces an HTML file for each page of the PDF, consisting of selectable HTML text overlaid on top of "screenshot" image of the page. (A page's text is not part of the screenshot.)
     8
     91. https://www.xpdfreader.com/download.html
     10
     11As per the Readme file found in the linux binary of Xpdf Tools, the Xpdf Viewer requires the qt toolkit, but not the Xpdf Tools. Have not read the Install file to confirm whether the same is the case for when compiling the command line tools. (But in that case, can't we just include the tools binary available for all 3 OS, instead of compiling on each platform)
     12
     13    - Using Xpdf's pdftohtml tool:
     14    greenstone@bedrock:~/Downloads/xpdf-tools-linux-4.00/bin64$./pdftohtml -z 1.5 ~/Downloads/ApacheLicence.pdf licence
     15
     16        where licence is a folder.
     17
     18    - Using Xpdf's pdftotext tool:
     19    greenstone@bedrock:~/Downloads/xpdf-tools-linux-4.00/bin64$./pdftotext -nopgbrk ~/Downloads/ApacheLicence.pdf ~/Downloads/ApacheLicence.txt
     20
     21        where the output text file must be specified with a full path name.
     22
     23
     242. Documentation on Xpdf-Tools:
     25- https://www.xpdfreader.com/support.html
     26    for example, the pdftohtml man page: https://www.xpdfreader.com/pdftohtml-man.html
     27- https://linux.die.net/man/5/xpdfrc
     28(Configuration flags you can put into ~/.xpdfrc to use as defaults when running xpdf tool commands)
     29
     303. We're using Xpdf Tools version: xpdf-tools-linux-4.00
     31
     324. We started by working with the ready-made Xpdf-tools binaries available for download from the xpdf site for Win, Linux and Mac.
     33
     345. We're now moving to compiling up Xpdf-tools ourselves using CASCADE-MAKE, which we have so far got to successfully compile statically on Linux (LSB environment inclusive) to build working binaries.
     35
     36On Mac, I've been unable to get it to produce statically linked libraries: at this stage they're dynamically linked.
     37
     38
     39__________________________________________________________
     40B. Mojo::DOM perl package for parsing HTML
     41__________________________________________________________
     42
     43XPDF's pdftohtml conversion of a single PDF document produces multiple HTML files: one for each page in the source PDF.
     44We want the output to be "paged_html": a single HTML file that is sectionalised, each section representing a page of the
     45original PDF.
     46
     47We need to be able to parse the many HTML pages produced by XPDF's pdftohtml conversion of a doc, in order to massage the output
     48into the single sectionalised HTML file. For this we needed a HTML parser package for Perl.
     49
     501. Before Dr Bainbridge found Mojo::DOM, he looked at
     51* https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers
     52* http://radar.oreilly.com/2014/02/parsing-html-with-perl-2.html
     53
     542. Main links for Mojo::DOM
     55* https://mojolicious.org/perldoc/Mojo/DOM
     56* https://metacpan.org/pod/Mojo::DOM
     57    Dependencies: http://deps.cpantesters.org/?module=Mojo%3A%3ADOM;perl=latest
     58
     593. Once you've downloaded Mojo::DOM's src, follow Dr Bainbridge's sequence of commands for building the Mojo::DOM CPAN module of perl below.
     60We'll be using this module to be used for parsing the HTML output by XPDF tool pdftohtml
     61
     62
     63    mkdir cpan
     64     2020  tar xvzf Mojolicious-7.84.tar.gz
     65     2021  cd Mojolicious-7.84/
     66     2028  perl ./Makefile.PL PREFIX=`pwd`/installed
     67     2030  make
     68     2031  make install
     69     2033  cp -r installed/share/perl/5.18.2 ../cpan
     70    cd ..
     71     2044  export PERL5LIB=`pwd`/cpan
     72
     73     2053  emacs -nw test.pl
     74
     75    #!/usr/bin/perl -w
     76    add in 'use v5.10;'
     77     
     78     2054  chmod a+x test.pl
     79     2055  ./test.pl
     80
     81
     82__________________________________________________________
     83C. Compiling Xpdf-Tools: statically or dynamically linked
     84__________________________________________________________
     85
     86As explained in detail in section D below, gs-CMakeLists.txt allows us to compile xpdf-tools statically (as we've now set it up for by default) or dynamically (as its CMake makefiles were originally set up for).
     87
     881. To compile Xpdf-Tools statically, packages/CASCADE-MAKE/XPDFTOOLS.sh should contain:
     89
     90    cmake -DCMAKE_BUILD_TYPE=Release \
     91        -DCMAKE_INSTALL_PREFIX=$prefix \
     92        -DZLIB_LIBRARY=$prefix/lib/libz.a \         # <========= THIS
     93        -DPNG_LIBRARY=$prefix/lib/libpng15.a \      # <========= THIS
     94        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \  # <========= THIS
     95        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \
     96        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \
     97        -DCMAKE_C_FLAGS="$CFLAGS" \
     98        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
     99        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
     100        -DGSDLFLAG_STATIC="$static_flag" \          # <========= THIS
     101        $GEXT_XPDFTOOLS/packages/$package$version
     102
     103In place of FREETYPE_LIBRARY above, could also try the following,
     104        -DFREETYPE_DIR=$prefix \
     105but then check the built binaries by running "ldd" and "file" over them, to make sure they're not referencing any .so dynamic link libraries:
     106
     107
     1082. To compile Xpdf-Tools dynamically and make it find *our* dynamically linked libraries for its helper packages zlib, libpng and freetype, edit packages/CASCADE-MAKE/XPDFTOOLS.sh to contain:
     109
     110    cmake -DCMAKE_BUILD_TYPE=Release \
     111        -DCMAKE_INSTALL_PREFIX=$prefix \
     112        -DZLIB_LIBRARY=$prefix/lib/libz.so.1.2.7 \          # <========= THIS
     113        -DPNG_LIBRARY=$prefix/lib/libpng15.so.15.30.0 \     # <========= THIS
     114        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.so.6.3.20 \  # <========= THIS
     115        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \
     116        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \
     117        -DCMAKE_C_FLAGS="$CFLAGS" \
     118        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
     119        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
     120        $GEXT_XPDFTOOLS/packages/$package$version       # <=== -DGSDLFLAG_STATIC removed
     121
     122
     123
     124    (1) In the above, you could also set
     125        -DFREETYPE_DIR=$prefix
     126    in place of
     127        -DGSDLFLAG_STATIC="$static_flag"
     128
     129    In that case it makes, xpdf-tools compilation find the "libfreetype.so" (no versioning at end) in our gs2-extension.
     130    After successfully building, make sure to have sourced the gs2-extension's setup.bash before running "ldd" over the
     131    generated xpdf-tools binaries, in order to let it use the $LD_LIBRARY_PATH we set to find our .so files.
     132
     133    (2) Note that there are no equivalent for ZLIB and LIBPNG: doing -DZLIB_DIR=$prefix or -DPNG_DIR=$prefix will be
     134    ineffective, as neither are recognised by xpdf-tools' CMake set up.
     135
     136__________________________________________________________
     137D. How we got Xpdf-Tools to compile using CASCADE-MAKE
     138__________________________________________________________
     139
     140The process:
     141
     1421. We set up a CASCADE-MAKE GS2-extension "xpdf-tools" at trac.greenstone.org/browser/gs2-extensions/xpdf-tools/trunk/src
     143Be aware that its lowercased "cascade-make" subfolder is an svn external, the original is at http://trac.greenstone.org/browser/other-projects/cascade-make/trunk/
     144
     145So far, this CASCADE-MAKE project includes the Xpdf-Tools source tarball, its helper packages zlib, libpng and freetype, as well as CMake to compile the Xpdf-Tools source code.
     146The next step is to include JPEG and TIFF libraries too.
     147
     1482a. We downloaded the Xpdf-Tools source tarball, xpdf-4.00.tar.gz, from the xpdf site at https://www.xpdfreader.com/download.html under section "Download the Xpdf source code".
     149
     150The xpdf-tools source code tarball consists of the source for Xpf-tools and Xpdf (Xpdf-Reader). The Xpdf-Reader additionally requires Qt to build and run, but we don't want the Xpdf-Reader, just Xpdf-Tools.
     151
     152b. Compiling Xpdf-Tools fron source and running them requires the following packages and libraries, as per the xpdf-tools source code INSTALL file:
     153
     154To build xpdf-tools:
     155- CMake 2.8.8 or newer
     156
     157Libraries to link against and used by xpdf-tools:
     158- FreeType 2.0.5 or newer
     159- libpng (for pdftoppm and pdftohtml)
     160- zlib (for pdftoppm and pdftohtml)
     161
     162
     1633. Compilation of xpdf-tools worked with CMake 3.11.4 on the linux resnet machine. However, CMake 3.11.3 itself failed to compile in the LSB environment and on the Mac Mountain Lion machine because of a version incompatibility between the older g++ installed there and the advanced version of CMake 3.11.4.
     164
     165CMake version 3.9.6 however is supposed to be compatible with older versions of g++, as per https://stackoverflow.com/questions/47886400/cmake-configure-error-in-3-10-1-but-not-in-3-9-6
     166To avoid installing newer versions of g++ and clang in the LSB virtual machine and the Mac, I've shifted the CMake version back to version 3.9.6, still
     167
     168
     1694a. On building xpdf-tools to work with dynamically linked libs found anywhere.
     170
     171If compiling xpdf-tools against dynamic linked libraries for these packages, then the basic CMake command in packages/CASECADE-MAKE/XPDFTOOLS.sh can look like:
     172    cmake -DCMAKE_BUILD_TYPE=Release \
     173        -DCMAKE_INSTALL_PREFIX=$prefix \
     174        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \
     175        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \
     176        -DCMAKE_C_FLAGS="$CFLAGS" \
     177        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
     178        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
     179        $GEXT_XPDFTOOLS/packages/$package$version   # Note: no -DGSDLFLAG_STATIC=...
     180
     181With the above, the xpdf-tools source code and its make files work out of the box.
     182
     1834b. On building xpdf-tools to work with the dynamically linked libs for freetype libpng, zlib that we produce when cascade-making the xpdf-tools gs2-extension.
     184
     185Since we're compiling up freetype, libpng and zlib packages as part of the Xpdf-Tools GS2-extension with CASCADE-MAKE, the next step was to compile xpdf-tools by dynamically linking against our .so files for these 3 libraries. To do so, XPDFTOOL.sh should have the following changes
     186
     187    (1) set up CFLAGS, CXXFLAGS, CPPFLAGS and LDFLAGS to help linkage of xpdf-tools find our .so versions of the necessary libs:
     188
     189    export CFLAGS="$CFLAGS -I$GEXTXPDFTOOLS_INSTALLED/include -I$GEXTXPDFTOOLS_INSTALLED/include/libpng15"
     190    export CPPFLAGS="$CPPFLAGS -I$GEXTXPDFTOOLS_INSTALLED/include -I$GEXTXPDFTOOLS_INSTALLED/include/libpng15"
     191    export CXXFLAGS="$CXXFLAGS -I$GEXTXPDFTOOLS_INSTALLED/include -I$GEXTXPDFTOOLS_INSTALLED/include/libpng15"
     192    export LDFLAGS="$LDFLAGS -L$GEXTXPDFTOOLS_INSTALLED/lib"
     193
     194    (2) The CMAKE command we run must pass the full paths to the actual .so library files (the ones with specific
     195    versions in their files names) rather than the symbolically linked generally-named .so files (the latter won't
     196    be found when building xpdf-tools and CMake will try to look for the .so library files elsewhere on the system):
     197
     198    cmake -DCMAKE_BUILD_TYPE=Release \
     199        -DCMAKE_INSTALL_PREFIX=$prefix \
     200        -DZLIB_LIBRARY=$prefix/lib/libz.so.1.2.7 \              # <========= NEW
     201        -DPNG_LIBRARY=$prefix/lib/libpng15.so.15.30.0 \         # <========= NEW
     202        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.so.6.3.20 \      # <========= NEW
     203        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \
     204        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \
     205        -DCMAKE_C_FLAGS="$CFLAGS" \
     206        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
     207        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
     208        $GEXT_XPDFTOOLS/packages/$package$version   # Again: no -DGSDLFLAG_STATIC=...
     209
     210Further, the "xpdf/CMakeLists.txt" file within the xpdf-4.00.tar.gz source code tarball needs to be modified to refer to ZLIB_LIBRARIES when linking pdftops and pdftoppm. The linking commands for *both* the "pdftops" and "pdftoppm" executable targets in xpdf/CMakeLists.txt should look like the following,
     211
     212        target_link_libraries(pdftoppm goo fofi splash
     213                        ${FREETYPE_LIBRARY} ${FREETYPE_OTHER_LIBS}
     214                        ${DTYPE_LIBRARY}
     215                        ${LCMS_LIBRARY}
     216            ${ZLIB_LIBRARIES})              # <========= NEW
     217
     218
     219    (3) Since CMakeLists.txt has been modified, we initially renamed the xpdf src tarball to gs-xpdf-4.00.tar.gz.
     220    However, the current version works with the regular downloaded xpdf-4.00.tar.gz tarball. But after extraction,
     221    XPDFTOOLS.sh copies across the custom packages/gs-CMakeLists.txt into the extracted tarball's xpdf subdirectory,
     222    renaming the file as CMakeLists.txt (so the path to it becomes "xpdf-4.00/xpdf/CMakeLists.txt"). In XPDFTOOLS.sh:   
     223
     224    # patch the original tarball with our custom makefile
     225    if [[ -d "$package$version/xpdf" && -f "gs-CMakeLists.txt" ]]; then
     226        echo "*******************************************************************"
     227        echo "Using our custom gs-CMakeLists.txt instead of the one included in $package$version"
     228        echo "Renaming gs-CMakeLists.txt to $package$version/xpdf/CMakeLists.txt"
     229        echo "*******************************************************************"
     230
     231        cp "gs-CMakeLists.txt" "$package$version/xpdf/CMakeLists.txt"
     232    fi
     233
     234
     2354c. On building static xpdf-tools binaries using the static *.a freetype libpng, zlib libraries that we produce when cascade-making the xpdf-tools gs2-extension.
     236
     237In order to compile up xpdf-tools *statically*, so that it builds against the static *.a libraries of freetype, libpng and zlib that we produce during the gs2-extension's CASCADE-MAKE process, we have to make further modifications.
     238
     239    (1) First, the XPDFTOOLS.sh cascade-make file should pass the full paths to the actual (non-symbolic link) .a file for each library.
     240    A custom GS flag, GSDLFLAG_STATIC, is also invented in gs-CMakeLists.txt and assigned "-static for linux
     241    and "-Bstatic" for Mac, to pass in during the linking stage of building xpdf-tools.
     242
     243    For Mac OSX, when -static is passed in for linking as on linux, this produced the error
     244    "ld: library not found for -lcrt0.o" during the build of the xpdf-tools package. For information, see
     245    https://stackoverflow.com/questions/3801011/ld-library-not-found-for-lcrt0-o-on-osx-10-6-with-gcc-clang-static-flag
     246    The page https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x mentions compiling
     247    with -Bstatic on Mac OSX instead. To do so, XPDFTOOLS.sh passes in the GSDLFLAG_STATIC set to either
     248    "-static" (for linux) or "-Bstatic" for darwin.
     249    However the last mentioned stackoverflow page also says that -Bstatic is a no-op, and this appears to be
     250    the case when "otool -L" is run over the generated xpdf-tools binaries: the binaries are all dynamically
     251    linked. Although they're finding our .so files of freetype, libpng and zlib, they're not finding the .a
     252    versions, even though XPDFTOOLS.sh tries to point gs-CMakeLists.txt to the correct .a files.
     253
     254    The new modifications to XPDFTOOLS.sh:
     255
     256    if [ "x$GSDLOS" == "xdarwin" ] ; then
     257        static_flag=-Bstatic
     258    else
     259        static_flag=-static
     260    fi
     261
    18262    ...
    19 
    20 
    21 Note that the xpdftools binaries produced in this fashion are still linked to the system zlib and libpng rather than the ones we're compiling up as part of the GEXT XPDFTOOLS extension. But just to get the compilation to succeed with the references to our local library in XPDFTOOLS.sh below (a step toward getting the statically linked xpdftools binaries to compile), we needed the above changes.
    22 
    23     cmake -DCMAKE_BUILD_TYPE=Release \
    24         -DCMAKE_INSTALL_PREFIX=$prefix \
    25         -DZLIB_LIBRARY=$prefix/lib/libz.a \ <========
    26         -DPNG_LIBRARY=$prefix/lib/libpng15.a \ <========
    27         -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \ <========
    28         -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \
    29         -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \
    30         -DCMAKE_C_FLAGS="$CFLAGS" \
    31         -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
    32         -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
    33         -DPNG_LINK_FLAGS="-lpng15" \
     263    cmake -DCMAKE_BUILD_TYPE=Release \
     264        -DCMAKE_INSTALL_PREFIX=$prefix \
     265        -DZLIB_LIBRARY=$prefix/lib/libz.a \                 # <========= MODIFIED TO .a
     266        -DPNG_LIBRARY=$prefix/lib/libpng15.a \              # <========= MODIFIED TO .a
     267        -DFREETYPE_LIBRARY=$prefix/lib/libfreetype.a \          # <========= MODIFIED TO .a
     268        -DCMAKE_DISABLE_FIND_PACKAGE_Qt4=1 \
     269        -DCMAKE_DISABLE_FIND_PACKAGE_Qt5Widgets=1 \
     270        -DCMAKE_C_FLAGS="$CFLAGS" \
     271        -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
     272        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
     273        -DGSDLFLAG_STATIC="$static_flag" \                  # <========= NEW
    34274        $GEXT_XPDFTOOLS/packages/$package$version
    35275
    36 
    37 -------------------------------------------------------------------
    38 Redundant CMakeLists.txt files
    39 -------------------------------------------------------------------
    40 I've also committed work in progress versions of CMakeLists.txt
    41 - CMakeLists.txt.orig: the original version of this file in the unmodified xpdf-4.00.tar.gz
    42 - CMakeLists.txt.static1_notworking: Sets up CMake settings for static compilation. Which then required some further changes to stop compilation from complaining about undefined references to thread mutex or something. (-lpthread/pthread library was necessary, but needed to set this the CMake way). Not a working version.
    43 - CMakeLists.txt.static2_notworking: more changes in an attempt to bypass the errors about undefined references to png (and 'inflate' of zlib). Still not working. It's this version that briefly attempted to use the override to PNG_LINK_FLAGS I am setting in XPDFTOOLS.sh (the line containing: -DPNG_LINK_FLAGS="-lpng15" \). Not sure why it never got set.
    44 - CMakeLists.txt: the version of this file first committed with the modified gs-xpdf-4.00.tar.gz. For getting xpdf-tools to compile with dynamic linking and with the current setup of XPDFTOOLS.sh
    45 
    46 There are comments in the modified CMakeLists.txt file above to provide URLs explaining why I made various changes.
    47 
     276    (2) Our customised gs-CMakeLists.txt file now checks for this flag GSDLFLAG_STATIC being set and, if it is,
     277    uses it during the linking stage. As in (1) above, it will be set to "-static" for Linux and "-Bstatic" for Mac.
     278   
     279    - When the flag is set, the linking flags passed into each occurrence of target_link_libraries() in
     280    gs-CMakeLists.txt is moreover manually written in the form of "-static -l<libs>" rather than using
     281    the default linking commands inherited from the original CMakeLists.txt.
     282    - If GSDLFLAG_STATIC isn't set, then we don't build statically, and the linking flags passed to each
     283    target_link_libraries() are mostly the original ones.
     284
     285    For example,
     286
     287        if(GSDLFLAG_STATIC)
     288            target_link_libraries(pdftoppm goo fofi splash
     289              ${GSDLFLAG_STATIC} -lfreetype ${DTYPE_LIBRARY} ${LCMS_LIBRARY} -lz -lm -lc -lpthread)
     290        else ()
     291            target_link_libraries(pdftoppm goo fofi splash
     292                            ${FREETYPE_LIBRARY} ${FREETYPE_OTHER_LIBS}
     293                            ${DTYPE_LIBRARY}
     294                           ${LCMS_LIBRARY}
     295                    ${ZLIB_LIBRARIES})
     296        endif ()
     297
     298    DETAILED EXPLANATION:
     299    We found that when building *statically*, gs-CMakeLists.txt needed to NOT use the PNG_LIBRARIES, ZLIB_LIBRARIES
     300    and FREETYPE_LIBRARY in its linker commands, target_link_libraries(), as doing so produced partially dynamic
     301    xpdf-tools executables which were moreover BROKEN. They wouldn't run, and in fact attempting to run an xpdf-tool,
     302    like "./pdftohtml", would produce a file not found error. Something like "bash: no such file or directory".
     303
     304    Online discussions mentioned that this generally happened when attempting to run 32 bit executables on 64 bit
     305    linux when 32 bit loaders are not installed. (In such cases, the solution was to apt-get install some 32 bit package.)
     306    However, our broken binaries were all 64 bit, as indicated when running the "file" command on them. However, their
     307    being further partially dynamically linked executables didn't imply that they would be broken, as we were eventually
     308    able to produce partially dynamic executables that did work, before solving static linking altogether.
     309
     310    The real issue was that including references to  ${FREETYPE_LIBRARY} ${FREETYPE_OTHER_LIBS}, ${PNG_LIBRARIES} and
     311    ${ZLIB_LIBRARIES} in any target_link_libraries() resulted in the wrong linking command producing broken binaries.
     312
     313    Doing the regular target_link_libraries() in static mode results in building with
     314    "-Wl,-Bstatic -lfreetype -lpng15 -lz -Wl,-Bdynamic -lpthread" at end of link line
     315    and produces broken binaries for pdftohtml/pdftoppm/pdftops/pdftopng.
     316
     317    Note that PNG_LIBRARIES includes zlib/lz: "-lpng -lz", and along with freetype,
     318    these are linked statically. However, Threads/lpthread is included as a dynamically
     319    linked library instead of including a .a (regardless of whether it's appended
     320    as -lpthread or Threads::Threads in the target_link_libraries()), contributing to
     321    the pdfhtml binary produced being a partially static, partially dynamic one,
     322    so a dynamic executable overall.
     323
     324    The order of dynamic .so files listed by ldd in the broken static binary of pdftohtml differs from
     325    a manually statically linked working version of pdftohtml, and seems to be the only difference
     326    between the two in ldd's output. Not using "-Wl,-Bstatic" and using -static (-Bstatic on Mac)
     327    in its place creates a partially static dynamic executable that isn't broken, whereas
     328    additionally removing "-Wl,-Bdynamic -lpthread" and replacing it with -lpthread
     329    moreover produces a working pdftohtml that is a fully static linked executable.
     330
     331    The inclusion of the math lib and c lib (lm and lc) in the final link command
     332    are to completely bypass the remaining .so dependencies that were present in
     333    the executable and produce the fully static executable. The lm and lc libs were referenced
     334    by all xpdf-tool binaries (as indicated when generating dynamic ones and running ldd over them)
     335    but Dr Bainbridge said that -lm and -lc were some libs passed in by the compiler by default,
     336    which would explain why explicitly setting them for some xpdftools and not other may not have
     337    mattered.
     338
     339NOTES:
     340Initial attempts at modifying gs-CMakeLists.txt for static compiling that proved to be unnecessary:
     341
     342    (1) Setting -static globally doesn't have a useful effect.
     343
     344    # We want to build static xpdf-tools binaries. See
     345    # https://stackoverflow.com/questions/24648357/compiling-a-static-executable-with-cmake
     346    # Want to make the min number of changes for building statically, so using the way
     347    # below. Beware, must *append* "-static" to existing CMAKE_EXE_LINKER_FLAGS=LD_FLAGS
     348    ##SET(CMAKE_FIND_LIBRARY_SUFFIXES ".a")
     349    ##SET(BUILD_SHARED_LIBS OFF)
     350    ##SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static")
     351
     352    The above 3 lines just add a -static before the "-O2 -Wall -fPIC -rdynamic ..." during linking, such as below.
     353    But they have no further effect on whether static building actually succeeds or not. The only effective static
     354    linking command (for Linux so far) was to pass -static in the target_link_libraries() command followed by the
     355    "-l<libname>" for each library in the correct order.
     356
     357----
     358/usr/bin/c++  -I/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/include  -I/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/include -I/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/include/libpng15 -O3 -Wall -fPIC  -L/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/lib  -L/home/greenstone/gs3-svn-26Mar2018/gs2build/ext/xpdf-tools/trunk/src/linux/lib -static ***** <- HERE ****** -O2 -Wall -fPIC -rdynamic CMakeFiles/pdftohtml.dir/HTMLGen.cc.o CMakeFiles/pdftohtml.dir/SplashOutputDev.cc.o CMakeFiles/pdftohtml.dir/TextOutputDev.cc.o CMakeFiles/pdftohtml.dir/pdftohtml.cc.o CMakeFiles/xpdf_objs.dir/AcroForm.cc.o CMakeFiles/xpdf_objs.dir/Annot.cc.o CMakeFiles/xpdf_objs.dir/Array.cc.o CMakeFiles/xpdf_objs.dir/BuiltinFont.cc.o CMakeFiles/xpdf_objs.dir/BuiltinFontTables.cc.o CMakeFiles/xpdf_objs.dir/Catalog.cc.o CMakeFiles/xpdf_objs.dir/CharCodeToUnicode.cc.o CMakeFiles/xpdf_objs.dir/CMap.cc.o CMakeFiles/xpdf_objs.dir/Decrypt.cc.o CMakeFiles/xpdf_objs.dir/Dict.cc.o CMakeFiles/xpdf_objs.dir/Error.cc.o CMakeFiles/xpdf_objs.dir/FontEncodingTables.cc.o CMakeFiles/xpdf_objs.dir/Form.cc.o CMakeFiles/xpdf_objs.dir/Function.cc.o CMakeFiles/xpdf_objs.dir/Gfx.cc.o CMakeFiles/xpdf_objs.dir/GfxFont.cc.o CMakeFiles/xpdf_objs.dir/GfxState.cc.o CMakeFiles/xpdf_objs.dir/GlobalParams.cc.o CMakeFiles/xpdf_objs.dir/JArithmeticDecoder.cc.o CMakeFiles/xpdf_objs.dir/JBIG2Stream.cc.o CMakeFiles/xpdf_objs.dir/JPXStream.cc.o CMakeFiles/xpdf_objs.dir/Lexer.cc.o CMakeFiles/xpdf_objs.dir/Link.cc.o CMakeFiles/xpdf_objs.dir/NameToCharCode.cc.o CMakeFiles/xpdf_objs.dir/Object.cc.o CMakeFiles/xpdf_objs.dir/OptionalContent.cc.o CMakeFiles/xpdf_objs.dir/Outline.cc.o CMakeFiles/xpdf_objs.dir/OutputDev.cc.o CMakeFiles/xpdf_objs.dir/Page.cc.o CMakeFiles/xpdf_objs.dir/Parser.cc.o CMakeFiles/xpdf_objs.dir/PDFDoc.cc.o CMakeFiles/xpdf_objs.dir/PDFDocEncoding.cc.o CMakeFiles/xpdf_objs.dir/PSTokenizer.cc.o CMakeFiles/xpdf_objs.dir/SecurityHandler.cc.o CMakeFiles/xpdf_objs.dir/Stream.cc.o CMakeFiles/xpdf_objs.dir/TextString.cc.o CMakeFiles/xpdf_objs.dir/UnicodeMap.cc.o CMakeFiles/xpdf_objs.dir/UnicodeTypeTable.cc.o CMakeFiles/xpdf_objs.dir/UTF8.cc.o CMakeFiles/xpdf_objs.dir/XFAForm.cc.o CMakeFiles/xpdf_objs.dir/XRef.cc.o CMakeFiles/xpdf_objs.dir/Zoox.cc.o  -o pdftohtml ../goo/libgoo.a ../fofi/libfofi.a ../splash/libsplash.a -static -lfreetype -lpng -lz -lm -lc -lpthread
     359----
     360
     361    (2) Threads::Threads instead of -lpthread results in a partially dynamic executable.
     362
     363    # The original, unmodified CMakeLists.txt was not set up sufficiently
     364    # for static compilation of xpdf-tools. As a result, compile would first fail
     365    # with errors about undefined refs to mutex / lpthread.
     366    # When building xpdf-tools statically, need to add the following 2 lines as well
     367    # as append "Threads::Threads" to the end of each "target_link_libraries(<list>)"
     368    # See https://stackoverflow.com/questions/1620918/cmake-and-libpthread
     369    # found googling cmake and "-lpthread" (pthread) after ERRORS to do with this, like:
     370    #   undefined reference to `pthread_mutex_unlock'
     371    ##set(THREADS_PREFER_PTHREAD_FLAG ON)
     372    ##find_package(Threads REQUIRED)
     373
     374    In instances when compilation was successful, including the above 2 lines in combination with "Threads::Threads"
     375    as the final argument to every target_link_libraries(...) occurrence in gs-CMakeLists.txt would only manage to
     376    produce partially dynamically linked xpdftools binaries. (Depending on what the linking command was when building
     377    Xpdf-Tools, the partially dynamically linked executables may work or may be broken. See explanation further above.)
     378    We wanted fully statically linked binaries, for which we needed to pass in "-lpthread" as the trailing argument
     379    to each target_link_libraries(...). So without either, compilation will fail. However, with "Threads::Threads"
     380    the binaries weren't fully static, whereas with -lpthread the xpdftools executables were fully static as CMake no
     381    longer tried to link against a dynamic Threads library.
     382
     383
     384(5) To view the unmodified CMakeLists.txt included in the xpdf-4.00 source code tarball, untar it and look for its "xpdf/CMakeLists.txt" (not the toplevel file of the same name).
     385Run a 'diff' against gs-CMakeLists.txt to see further differences, such as debug statements and comments. Most comments have been removed and placed into this readme file instead.
     386
     387
     388(6) When CASCADE-MAKE is run on the xpdf-tools GS2-extension, it first compiles up CMake, needed to compile up xpdf-tools.
     389Unlike the library packages like freetype, libpng and zlib that we also build for xpdf-tools as part of this gs2-extension, CMake's build products don't need to be included in the distribution tarball of our built xpdf-tools executables.
     390
     391There's a "move-cmake.sh" script in the xpdf-tools gs2-extension that can be run with the "away" and "back" options to move the CMake stuff out of the way (into a "devel" folder) after successfully building xpdf binaries and that can also be run to move them back if wanting to recompile.
     392
     393The script can be run manually, but it's also run by the extension:
     394- packages/CASCADE-MAKE/XPDFTOOLS.sh runs "move-cmake.sh away" after xpdf-tools has been built, so that the extension's install location is ready for tarring up for distribution.
     395- When recompiling the xpdf-tools extenion, the CASCADE-MAKE process will run packages/CASCADE-MAKE/CMAKE.sh file which in turn runs "move-cmake.sh back" if there's a prebuilt CMake which had earlier been moved out of the way.
     396
     397

  • gs2-extensions/xpdf-tools/trunk/src/packages/gs-CMakeLists.txt

    r32247 r32248  
    1010
    1111#========================================================================
    12 # This is a customised version of Xpdf Tools' CMake file.
    13 # Customised for Greenstone.
     12# This is a customised version of Xpdf Tools' CMake file for Greenstone
     13# where we want to build static xpdf-tools executables.
     14# Refer to the GS-README.txt file for the Greenstone "xpdf-tools"
     15# gs2-extension and read section D for the GS specific customisations
     16# in this file and the reasoning behind them, most of which have to do
     17# with attempting to successfully build xpdf-tools statically.
    1418#========================================================================
    1519
    … …  
    2125
    2226
    23 # For Mac OSX, when -static is passed in for linking, we end up with the error "ld: library not found for -lcrt0.o"
    24 # See https://stackoverflow.com/questions/3801011/ld-library-not-found-for-lcrt0-o-on-osx-10-6-with-gcc-clang-static-flag
    25 # On Mac OSX, compile it with -Bstatic, as per https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x
    26 # XPDFTOOLS.sh will have set the GSDLFLAG_STATIC to either "-static" (for linux) or "-Bstatic" for darwin
     27# When compiling statically, XPDFTOOLS.sh will have set the custom
     28# GSDLFLAG_STATIC flag to either "-static" for linux or "-Bstatic" for darwin
    2729
    2830if(GSDLFLAG_STATIC)
    … …  
    131133endif ()
    132134
    133 # We want to build static xpdf-tools binaries. See
    134 # https://stackoverflow.com/questions/24648357/compiling-a-static-executable-with-cmake
    135 # Want to make the min number of changes for building statically, so using the way
    136 # below. Beware, must *append* "-static" to existing CMAKE_EXE_LINKER_FLAGS=LD_FLAGS
    137 ##SET(CMAKE_FIND_LIBRARY_SUFFIXES ".a")
    138 ##SET(BUILD_SHARED_LIBS OFF)
    139 ##SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static")
    140 
    141 # The original, unmodified CMakeLists.txt was not set up sufficiently
    142 # for static compilation of xpdf-tools. As a result, compile would first fail
    143 # with errors about undefined refs to mutex / lpthread.
    144 # When building xpdf-tools statically, need to add the following 2 lines as well
    145 # as append "Threads::Threads" to the end of each "target_link_libraries(<list>)"
    146 # See https://stackoverflow.com/questions/1620918/cmake-and-libpthread
    147 # found googling cmake and "-lpthread" (pthread) after ERRORS to do with this, like:
    148 #   undefined reference to `pthread_mutex_unlock'
    149 ##set(THREADS_PREFER_PTHREAD_FLAG ON)
    150 ##find_package(Threads REQUIRED)
    151 
    152135#--- pdftops
    153136
    … …  
    222205                          ${LCMS_LIBRARY} ${PNG_LIBRARIES})
    223206  endif ()
    224 # Doing the regular target_link_libraries() in static mode results in building with
    225 # "-Wl,-Bstatic -lfreetype -lpng15 -lz -Wl,-Bdynamic -lpthread" at end of link line
    226 # and produces broken binaries for pdftohtml/pdftoppm/pdftops/pdftopng.
    227 # Note that PNG_LIBRARIES includes zlib/lz: "-lpng -lz", and along with freetype,
    228 # these are linked statically. However, Threads/lpthread is included as a dynamically
    229 # linked library instead of including a .a (regardless of whether it's appended
    230 # as -lpthread or Threads::Threads in else above), contributing to the pdfhtml binary
    231 # produced being a partially static, partially dynamic one,
    232 # so a dynamic executable overall.
    233 # The order of dynamic .so files listed by ldd in the broken binary differs from
    234 # a manually linked working version of pdftohtml, and seems to be the only difference
    235 # between the two in ldd output. Not using "-Wl,-Bstatic" and using -static (-Bstatic on Mac)
    236 # in its place creates a partially static dynamic executable that isn't broken, whereas
    237 # additionally removing "-Wl,-Bdynamic -lpthread" and replacing it with -lpthread
    238 # moreover produces a working pdftohtml that is a fully static linked executable.
    239 # The inclusion of the math lib and c lib (lm and lc) in the final link command
    240 # are to completely bypass the remaining .so dependencies that were present in
    241 # the executable and produce the fully static executable.
    242 # Not sure what colormanager (lcms) and typelibrary, or how they're written in -l
    243 # form as they were not present when the original unmodified static linking command
    244 # was run, so they've not been included in the replacement static linking command.
    245207  install(TARGETS pdftohtml RUNTIME DESTINATION bin)
    246208  install(FILES ${PROJECT_SOURCE_DIR}/doc/pdftohtml.1 DESTINATION man/man1)
Note: See TracChangeset for help on using the changeset viewer.