Show
Ignore:
Timestamp:
28.07.2017 18:53:37 (2 years ago)
Author:
ak19
Message:

Intermediate commit for building wget with openssl. This is a working version, part of incremental commits. For now, we're still using wget-1.15-gs (that always requires --no-check-certificate if running wget), which now uses a wgetrc conf file to turn off the check-certificate default of wget, since without this, wget fails getting https pages that don't yet have a valid security certificate. This commit introduces the template wgetrc-gs.in file, which contains a flag set to not require certificate checks, is copied to gs2build/OS as wgetrc. The WGETRC env var is set in setup.bash to point to the wgetrc file so wget runs smoothly against https urls.

Location:
main/trunk/greenstone2
Files:
1 added
5 modified

Legend:

Unmodified
Added
Removed
  • main/trunk/greenstone2/build-src/packages/Makefile

    r29366 r31829  
    2929# is assumed that each package will have at least four rules: all, install, 
    3030# clean, and distclean. 
     31# openssl is configured and compiled during the configure stage by 
     32# build-src/packages/configure. make clean and make distclean happen here. 
    3133 
    3234PACKAGEDIRS =  
     
    4042        wget/wget-1.15-gs cpan/XML-Parser-2.41 \ 
    4143        xlhtml/xlhtml-0.4.9.0 isis-gdl html-tidy/tidy 
     44OPENSSL_PKG = openssl/openssl-1.0.2l 
    4245 
    4346all: 
     
    5457 
    5558clean: 
    56     for subdir in $(PACKAGEDIRS); do \ 
     59    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \ 
    5760          echo cleaning $@ in $$subdir; \ 
    5861          (cd $$subdir && $(MAKE) clean || exit 0) ; \ 
     
    6063 
    6164distclean: 
    62     for subdir in $(PACKAGEDIRS); do \ 
     65    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \ 
    6366          echo cleaning $@ in $$subdir; \ 
    6467          (cd $$subdir && $(MAKE) distclean || exit 0) ; \ 
    6568        done 
     69    (cd openssl && rm -rf certs man misc private openssl.cnf || exit 0) 
     70    rm -rf $(OPENSSL_PKG) 
  • main/trunk/greenstone2/build-src/packages/Makefile.in

    r29366 r31829  
    2929# is assumed that each package will have at least four rules: all, install, 
    3030# clean, and distclean. 
     31# openssl is configured and compiled during the configure stage by 
     32# build-src/packages/configure. make clean and make distclean happen here. 
    3133 
    3234PACKAGEDIRS =  
     
    4042        wget/wget-1.15-gs cpan/XML-Parser-2.41 \ 
    4143        xlhtml/xlhtml-0.4.9.0 isis-gdl html-tidy/tidy 
     44OPENSSL_PKG = openssl/openssl-1.0.2l 
    4245 
    4346all: 
     
    5457 
    5558clean: 
    56     for subdir in $(PACKAGEDIRS); do \ 
     59    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \ 
    5760          echo cleaning $@ in $$subdir; \ 
    5861          (cd $$subdir && $(MAKE) clean || exit 0) ; \ 
     
    6063 
    6164distclean: 
    62     for subdir in $(PACKAGEDIRS); do \ 
     65    for subdir in $(PACKAGEDIRS) $(OPENSSL_PKG); do \ 
    6366          echo cleaning $@ in $$subdir; \ 
    6467          (cd $$subdir && $(MAKE) distclean || exit 0) ; \ 
    6568        done 
     69    (cd openssl && rm -rf certs man misc private openssl.cnf || exit 0) 
     70    rm -rf $(OPENSSL_PKG) 
  • main/trunk/greenstone2/build-src/packages/configure

    r31824 r31829  
    150150cd "$PACKAGES/openssl" 
    151151openssl_pkgname=openssl-1.0.2l 
     152#openssl_pkgname=openssl-1.1.0f 
    152153if test ! -d $openssl_pkgname ; 
    153154then 
     
    158159# configure 
    159160# see the INSTALL file for OpenSSL to see config options for openssl 
     161# If you ever want to have the .a libraries that building openssl generates, 
     162# then use the --prefix and point it to the $bindir. 
    160163# --prefix indicates where the generated lib, include and bin folders will go 
    161164# --openssldir will contain the manuals and other non-essentials 
     
    248251 
    249252# configure wget 
     253wget_pkgname=wget-1.15-gs 
     254#wget_pkgname=wget-1.17.1-gs 
    250255echo "" 
    251256echo "Configuring wget" 
    252257echo "" 
    253258cd "$PACKAGES/wget" 
    254 gzip -dc wget-1.15-gs.tar.gz | tar -xf - 
     259gzip -dc $wget_pkgname.tar.gz | tar -xf - 
    255260if test x$crossOS = "xandroid" ; 
    256261then 
     
    260265fi 
    261266 
     267# When compiling wget-1.15-gs without openssl as before: 
    262268#./configure $CACHE_FILE --prefix=$PACKAGES/wget --without-ssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs 
    263269 
    264 # We weren't compiling up wget statically before either 
     270# We weren't compiling up wget statically before either and we still aren't 
    265271# However, to compile up wget (statically or not) with openssl, see 
    266272# https://stackoverflow.com/questions/9817337/compiling-wget-with-static-linking-self-compiled-openssl-library-linking-issu 
    267 # Note: if the cppflags and ldflags are set during the configure stage, as we now do, 
    268 # then the make command needn't additionally set them as well. So just make, make install then works  
    269 echo cd "$PACKAGES/wget/wget-1.15-gs" 
    270 echo CPPFLAGS="-I/$bindir/openssl/include" LDFLAGS="-L/$bindir/openssl/lib" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs 
    271 cd "$PACKAGES/wget/wget-1.15-gs" && \ 
     273# Note: contrary to the instructions at the link, if the cppflags and ldflags are set during the 
     274# configure stage, as we now do, then the make command needn't additionally set them as well. 
     275# So just make, make install works thereafter 
     276 
     277echo copying "$PACKAGES/wget/wgetrc-gs.in" file to "$bindir/wgetrc" 
     278cp "$PACKAGES/wget/wgetrc-gs.in" "$bindir/wgetrc" 
     279echo cd "$PACKAGES/wget/$wget_pkgname" 
     280 
     281# for wget-1.15-gs. (can also try adding --localstatedir="$bindir/wget" --enable-iri) 
     282echo CPPFLAGS="-I/$bindir/openssl/include" LDFLAGS="-L/$bindir/openssl/lib" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraarg 
     283cd "$PACKAGES/wget/$wget_pkgname" && \ 
    272284CPPFLAGS="-I/$bindir/openssl/include" LDFLAGS="-L/$bindir/openssl/lib" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs 
    273285 
     286# for wget-1.17.1-gs. (can also try adding --localstatedir="$bindir/wget" --enable-iri) 
     287#echo OPENSSL_CFLAGS="-I/$bindir/openssl/include" OPENSSL_LIBS="-L/$bindir/openssl/lib -lssl -lcrypto" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --with-openssl=auto --with-libssl-prefix="$bindir/openssl" --bindir="$bindir" -disable-nls $HOSTTARGETFLAGS $wget_extraargs 
     288#cd "$PACKAGES/wget/$wget_pkgname" && \ 
     289#OPENSSL_CFLAGS="-I/$bindir/openssl/include" OPENSSL_LIBS="-L/$bindir/openssl/lib -lssl -lcrypto" ./configure $CACHE_FILE --prefix=$PACKAGES/wget --with-ssl=openssl --with-openssl=auto --with-libssl-prefix="$bindir/openssl" --bindir="$bindir"  -disable-nls $HOSTTARGETFLAGS $wget_extraargs 
    274290 
    275291## 
  • main/trunk/greenstone2/build-src/packages/wget/README

    r31826 r31829  
    105105 
    106106 
    107 9) We're now shifting to wget-1.17.1 which is installed on Ubuntu 16.04 and which work on https urls without the --no-check-certificate flag being necessary. 
     1079) We're now shifting to wget-1.17.1 which is installed on Ubuntu 16.04, and for which a windows binary compiled with OpenSSL is available. Both the linux system version and windows binary work on https urls without the --no-check-certificate flag being necessary. However, the compiled up Linux version still needs this flag, see under PROBLEM. 
    108108 
    109109This way our perl code can launch wget as before, without always passing that additional flag. Hopefully the output in the Download pane will be the same so that the donwload parsing will work. 
     
    118118* configuring had initially failed, reporting that OPENSSL_CFLAGS and OPENSSL_LIBS need to be set if not wanting to use whatever pkg-config may find. To set LIBS variables, use one of these forms: LIBS="-L/path/to/lib" or LIBS="/path/to/lib/lib.a" or LIBS="-lssl". To combine all three, separate with spaces. See http://trac.greenstone.org/changeset/30948 and https://github.com/tatsuhiro-t/spdylay/issues/43 
    119119 
     120PROBLEM AND SOLUTION WITH WGETRC 
    120121Can turn off requiring a certificate check for https URLs in wgetrc conf file, as explained here: 
    121122https://superuser.com/questions/508696/wget-without-no-check-certificate 
     
    139140# https://lists.gnu.org/archive/html/bug-wget/2014-12/msg00104.html 
    140141 
    141 2ND PROBLEM: OpenSSL License, see https://www.openssl.org/source/license.html 
     142Alternatives for Windows: 
     143Source: 
     144- https://soliloquyforthefallen.net/?p=238 
     145- https://github.com/wertarbyte/wget/tree/master/windows (README at end) 
     146Binaries: 
     147- https://stackoverflow.com/questions/14344921/wget-for-windows-7-trusted-source 
    142148 
    143 QUESTION: If I delete the gs2build/bin/linux/openssl folder, the built wget does not seem to care. Is it finding something else or has it included the openssl somehow? How can I verify this? 
     149COMBINING GREENSTONE's GPL with OpenSSL LICENSES 
     150OpenSSL is under a double license, see https://www.openssl.org/source/license.html 
     151The licenses for GPL and OpenSSL are incompatible, see https://www.gnu.org/licenses/license-list.en.html#OpenSSL 
     152but you can combine it this way: https://opensource.stackexchange.com/questions/2233/gpl-v3-with-openssl-exception?rq=1 
     153which is what we've done for GS2 and GS3. 
     154 
     155 
     156TO DO: 
     157- If I delete the gs2build/bin/linux/openssl folder, the built wget still works fine without it. Dr Bainbridge confirmed that this is because, wget is built against OpenSSL's static libraries and therefore no longer needs the OpenSSL stuff we build and have been putting into gs2build/bin/linux/openssl. So we no longer need to put the built OpenSSL there. 
     158 
     159- Add a tick box in GLI > File > Preferences for turning on No Check Certificate over https, this should then replace our wgetrc file and env variable set in GS2's setup.bash. By default leave this flag unticked, so downloading won't work over https. Need to store this user setting in GLI's config.xml. Ensure that when the download over https failed, it results in an error. 
     160 
     161- If the downloading error count > 0: 
     162At the bottom of GLI > Download Pane > View Log > download error log - when we get errors: 
     163You have the option of adjusting your proxy server settings (go through the Configure Proxy button) 
     164For https certificate authentication, you have the option of turning off checking the certificate in the Connections tab of File > Preferences 
     165 
     166Check the warnings on windows. If it's no longer always warning, then do the stuff above on warning too, not just on error. 
     167 
  • main/trunk/greenstone2/setup.bash

    r31406 r31829  
    337337# Perl >= v5.18.* randomises map iteration order within a process 
    338338export PERL_PERTURB_KEYS=0 
     339 
     340# turn off certificate errors when using wget to retrieve over https 
     341# (to avoid turning it off with the --no-check-certificate flag to each wget cmd) 
     342# See https://superuser.com/questions/508696/wget-without-no-check-certificate 
     343# https://www.gnu.org/software/wget/manual/html_node/Wgetrc-Location.html 
     344export WGETRC=$GSDLHOME/bin/$GSDLOS/wgetrc