Changeset 31829 for main


Ignore:
Timestamp:
2017-07-28T18:53:37+12:00 (7 years ago)
Author:
ak19
Message:

Intermediate commit for building wget with openssl. This is a working version, part of incremental commits. For now, we're still using wget-1.15-gs (that always requires --no-check-certificate if running wget), which now uses a wgetrc conf file to turn off the check-certificate default of wget, since without this, wget fails getting https pages that don't yet have a valid security certificate. This commit introduces the template wgetrc-gs.in file, which contains a flag set to not require certificate checks, is copied to gs2build/OS as wgetrc. The WGETRC env var is set in setup.bash to point to the wgetrc file so wget runs smoothly against https urls.

Location:
main/trunk/greenstone2
Files:
1 added
5 edited

Legend:

Unmodified
Added
Removed

  • main/trunk/greenstone2/build-src/packages/Makefile

    r29366 r31829  
    2929# is assumed that each package will have at least four rules: all, install,
    3030# clean, and distclean.
     31# openssl is configured and compiled during the configure stage by
     32# build-src/packages/configure. make clean and make distclean happen here.
    3133
    3234PACKAGEDIRS =
    … …  
    4042        wget/wget-1.15-gs cpan/XML-Parser-2.41 \
    4143        xlhtml/xlhtml-0.4.9.0 isis-gdl html-tidy/tidy
     44OPENSSL_PKG = openssl/openssl-1.0.2l
    4245
    4346all:
    … …  
    5457
    5558clean:
    56     for subdir in $(PACKAGEDIRS); do \
     59    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \
    5760          echo cleaning $@ in $$subdir; \
    5861          (cd $$subdir && $(MAKE) clean || exit 0) ; \
    … …  
    6063
    6164distclean:
    62     for subdir in $(PACKAGEDIRS); do \
     65    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \
    6366          echo cleaning $@ in $$subdir; \
    6467          (cd $$subdir && $(MAKE) distclean || exit 0) ; \
    6568        done
     69    (cd openssl && rm -rf certs man misc private openssl.cnf || exit 0)
     70    rm -rf $(OPENSSL_PKG)

  • main/trunk/greenstone2/build-src/packages/Makefile.in

    r29366 r31829  
    2929# is assumed that each package will have at least four rules: all, install,
    3030# clean, and distclean.
     31# openssl is configured and compiled during the configure stage by
     32# build-src/packages/configure. make clean and make distclean happen here.
    3133
    3234PACKAGEDIRS =
    … …  
    4042        wget/wget-1.15-gs cpan/XML-Parser-2.41 \
    4143        xlhtml/xlhtml-0.4.9.0 isis-gdl html-tidy/tidy
     44OPENSSL_PKG = openssl/openssl-1.0.2l
    4245
    4346all:
    … …  
    5457
    5558clean:
    56     for subdir in $(PACKAGEDIRS); do \
     59    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \
    5760          echo cleaning $@ in $$subdir; \
    5861          (cd $$subdir && $(MAKE) clean || exit 0) ; \
    … …  
    6063
    6164distclean:
    62     for subdir in $(PACKAGEDIRS); do \
     65    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \
    6366          echo cleaning $@ in $$subdir; \
    6467          (cd $$subdir && $(MAKE) distclean || exit 0) ; \
    6568        done
     69    (cd openssl && rm -rf certs man misc private openssl.cnf || exit 0)
     70    rm -rf $(OPENSSL_PKG)

  • main/trunk/greenstone2/build-src/packages/configure

    r31824 r31829  
    150150cd "$PACKAGES/openssl"
    151151openssl_pkgname=openssl-1.0.2l
     152#openssl_pkgname=openssl-1.1.0f
    152153if test ! -d $openssl_pkgname ;
    153154then
    … …  
    158159# configure
    159160# see the INSTALL file for OpenSSL to see config options for openssl
     161# If you ever want to have the .a libraries that building openssl generates,
     162# then use the --prefix and point it to the $bindir.
    160163# --prefix indicates where the generated lib, include and bin folders will go
    161164# --openssldir will contain the manuals and other non-essentials
    … …  
    248251
    249252# configure wget
     253wget_pkgname=wget-1.15-gs
     254#wget_pkgname=wget-1.17.1-gs
    250255echo ""
    251256echo "Configuring wget"
    252257echo ""
    253258cd "$PACKAGES/wget"
    254 gzip -dc wget-1.15-gs.tar.gz | tar -xf -
     259gzip -dc $wget_pkgname.tar.gz | tar -xf -
    255260if test x$crossOS = "xandroid" ;
    256261then
    … …  
    260265fi
    261266
     267# When compiling wget-1.15-gs without openssl as before:
    262268#./configure $CACHE_FILE --prefix=$PACKAGES/wget --without-ssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs
    263269
    264 # We weren't compiling up wget statically before either
     270# We weren't compiling up wget statically before either and we still aren't
    265271# However, to compile up wget (statically or not) with openssl, see
    266272# https://stackoverflow.com/questions/9817337/compiling-wget-with-static-linking-self-compiled-openssl-library-linking-issu
    267 # Note: if the cppflags and ldflags are set during the configure stage, as we now do,
    268 # then the make command needn't additionally set them as well. So just make, make install then works
    269 echo cd "$PACKAGES/wget/wget-1.15-gs"
    270 echo CPPFLAGS="-I/$bindir/openssl/include" LDFLAGS="-L/$bindir/openssl/lib" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs
    271 cd "$PACKAGES/wget/wget-1.15-gs" && \
     273# Note: contrary to the instructions at the link, if the cppflags and ldflags are set during the
     274# configure stage, as we now do, then the make command needn't additionally set them as well.
     275# So just make, make install works thereafter
     276
     277echo copying "$PACKAGES/wget/wgetrc-gs.in" file to "$bindir/wgetrc"
     278cp "$PACKAGES/wget/wgetrc-gs.in" "$bindir/wgetrc"
     279echo cd "$PACKAGES/wget/$wget_pkgname"
     280
     281# for wget-1.15-gs. (can also try adding --localstatedir="$bindir/wget" --enable-iri)
     282echo CPPFLAGS="-I/$bindir/openssl/include" LDFLAGS="-L/$bindir/openssl/lib" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraarg
     283cd "$PACKAGES/wget/$wget_pkgname" && \
    272284CPPFLAGS="-I/$bindir/openssl/include" LDFLAGS="-L/$bindir/openssl/lib" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs
    273285
     286# for wget-1.17.1-gs. (can also try adding --localstatedir="$bindir/wget" --enable-iri)
     287#echo OPENSSL_CFLAGS="-I/$bindir/openssl/include" OPENSSL_LIBS="-L/$bindir/openssl/lib -lssl -lcrypto" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --with-openssl=auto --with-libssl-prefix="$bindir/openssl" --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs
     288#cd "$PACKAGES/wget/$wget_pkgname" && \
     289#OPENSSL_CFLAGS="-I/$bindir/openssl/include" OPENSSL_LIBS="-L/$bindir/openssl/lib -lssl -lcrypto" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --with-openssl=auto --with-libssl-prefix="$bindir/openssl" --bindir="$bindir"  -disable-nls $HOSTTARGETFLAGS $wget_extraargs
    274290
    275291##

  • main/trunk/greenstone2/build-src/packages/wget/README

    r31826 r31829  
    105105
    106106
    107 9) We're now shifting to wget-1.17.1 which is installed on Ubuntu 16.04 and which work on https urls without the --no-check-certificate flag being necessary.
     1079) We're now shifting to wget-1.17.1 which is installed on Ubuntu 16.04, and for which a windows binary compiled with OpenSSL is available. Both the linux system version and windows binary work on https urls without the --no-check-certificate flag being necessary. However, the compiled up Linux version still needs this flag, see under PROBLEM.
    108108
    109109This way our perl code can launch wget as before, without always passing that additional flag. Hopefully the output in the Download pane will be the same so that the donwload parsing will work.
    … …  
    118118* configuring had initially failed, reporting that OPENSSL_CFLAGS and OPENSSL_LIBS need to be set if not wanting to use whatever pkg-config may find. To set LIBS variables, use one of these forms: LIBS="-L/path/to/lib" or LIBS="/path/to/lib/lib.a" or LIBS="-lssl". To combine all three, separate with spaces. See http://trac.greenstone.org/changeset/30948 and https://github.com/tatsuhiro-t/spdylay/issues/43
    119119
     120PROBLEM AND SOLUTION WITH WGETRC
    120121Can turn off requiring a certificate check for https URLs in wgetrc conf file, as explained here:
    121122https://superuser.com/questions/508696/wget-without-no-check-certificate
    … …  
    139140# https://lists.gnu.org/archive/html/bug-wget/2014-12/msg00104.html
    140141
    141 2ND PROBLEM: OpenSSL License, see https://www.openssl.org/source/license.html
     142Alternatives for Windows:
     143Source:
     144- https://soliloquyforthefallen.net/?p=238
     145- https://github.com/wertarbyte/wget/tree/master/windows (README at end)
     146Binaries:
     147- https://stackoverflow.com/questions/14344921/wget-for-windows-7-trusted-source
    142148
    143 QUESTION: If I delete the gs2build/bin/linux/openssl folder, the built wget does not seem to care. Is it finding something else or has it included the openssl somehow? How can I verify this?
     149COMBINING GREENSTONE's GPL with OpenSSL LICENSES
     150OpenSSL is under a double license, see https://www.openssl.org/source/license.html
     151The licenses for GPL and OpenSSL are incompatible, see https://www.gnu.org/licenses/license-list.en.html#OpenSSL
     152but you can combine it this way: https://opensource.stackexchange.com/questions/2233/gpl-v3-with-openssl-exception?rq=1
     153which is what we've done for GS2 and GS3.
     154
     155
     156TO DO:
     157- If I delete the gs2build/bin/linux/openssl folder, the built wget still works fine without it. Dr Bainbridge confirmed that this is because, wget is built against OpenSSL's static libraries and therefore no longer needs the OpenSSL stuff we build and have been putting into gs2build/bin/linux/openssl. So we no longer need to put the built OpenSSL there.
     158
     159- Add a tick box in GLI > File > Preferences for turning on No Check Certificate over https, this should then replace our wgetrc file and env variable set in GS2's setup.bash. By default leave this flag unticked, so downloading won't work over https. Need to store this user setting in GLI's config.xml. Ensure that when the download over https failed, it results in an error.
     160
     161- If the downloading error count > 0:
     162At the bottom of GLI > Download Pane > View Log > download error log - when we get errors:
     163You have the option of adjusting your proxy server settings (go through the Configure Proxy button)
     164For https certificate authentication, you have the option of turning off checking the certificate in the Connections tab of File > Preferences
     165
     166Check the warnings on windows. If it's no longer always warning, then do the stuff above on warning too, not just on error.
     167

  • main/trunk/greenstone2/setup.bash

    r31406 r31829  
    337337# Perl >= v5.18.* randomises map iteration order within a process
    338338export PERL_PERTURB_KEYS=0
     339
     340# turn off certificate errors when using wget to retrieve over https
     341# (to avoid turning it off with the --no-check-certificate flag to each wget cmd)
     342# See https://superuser.com/questions/508696/wget-without-no-check-certificate
     343# https://www.gnu.org/software/wget/manual/html_node/Wgetrc-Location.html
     344export WGETRC=$GSDLHOME/bin/$GSDLOS/wgetrc
Note: See TracChangeset for help on using the changeset viewer.