Changeset 31837 for main

Show
Ignore:
Timestamp:
01.08.2017 20:57:03 (2 years ago)
Author:
ak19
Message:

Added instructions on how to successfully compile up openssl v 1.1.0f on Linux. It requires wget 1.19.x however. And it makes no difference to being required to add in no-check-certificate when downloading from an HTTPS URL.

Files:
1 modified

Legend:

Unmodified
Added
Removed
  • main/trunk/greenstone2/build-src/packages/wget/README

    r31829 r31837  
    79798) 2017/07/27 - ak19 (anupama.krishnan@waikato.ac.nz) - still using wget version 1.15, but now compiling wget up with OpenSSL support. Wget needs SSL support in order for it to access pages over HTTPS. In future, the web will be using https. 
    8080 
    81 We're now compiling up OpenSSL during the configuration phase since wget needs it to exist during its configure phase. We;re building OpenSSL statically, by setting the no-shared flag. The built OpenSSL gets put into gs2build/linux|darwin/openssl, containing lib, include and bin subfolders. When configuring wget, we build wget against our OpenSSl, and make and make install proceed as normal. Refer to gs2build/build-src/packages configure. 
     81We're now compiling up OpenSSL during the configuration phase since wget needs it to exist during its configure phase. We're building OpenSSL statically, by setting the no-shared flag. The built OpenSSL gets put into gs2build/linux|darwin/openssl, containing lib, include and bin subfolders. When configuring wget, we build wget against our OpenSSl, and make and make install proceed as normal. Refer to gs2build/build-src/packages configure. 
    8282 
    83 We weren't compiling up wget statically before either, so we're still not doing so. To compile up wget (statically or not) with openssl, a helpful page was 
     83We weren't compiling up wget statically before either, so we're still not doing so. But if that will be necessary in future, see the section on COMPILING WGET UP STATICALLY further below. 
     84 
     85To compile up wget (statically or not) with openssl, a helpful page was 
    8486https://stackoverflow.com/questions/9817337/compiling-wget-with-static-linking-self-compiled-openssl-library-linking-issu 
    8587Note, however, that since the CPPFLAGS and LDFLAGS are now set to point to our OpenSSL during the configure stage, the make command needn't additionally set them as well, contrary to the instruction for make on the stackoverflow page. So we just need to do the usual make, make install once the configure is done against OpenSSL. 
    86  
    87 If compiling wget up statically, then, in the LDFLAGS prepended to wget's configure command, append -static. Further, the gcc command that gets run needs to have -lpthread in its library listing at the end. The order of the libraries listed also needs to change for static compilation to be successful: 
    88 -lprce -lpthread -ldl <remaining -llibs> 
    89  
    90 However, warnings appear when compiling wget statically, as it does not make sense to create some programs statically since they may be stuck including a local context (e.g. something related to DNS warnings in compiling up a previous component statically). Linking against some libraries to create a static binary may not make sense either. For instance -ldl, the dynamic loading or linking library, may not make sense if the binary created is static. This seems to imply that wget makes more sense if compiled up as a shared object, .so, than as a static one, .a. 
    9188 
    9289The existing version of wget, 1.15, works with HTTPS when compiled against OpenSSL. However, this version of the binary needs to be run with the --no-check-certificate flag on to access https pages without a security certificate. 
     
    9491e.g. ./wget --no-check-certificate http://englishhistory.net/tudor/citizens/ 
    9592 
    96 The system wget on Ubuntu 16.04 is version 1.17.1 and does not require this flag. Pre-compiled windows binaries are available for version 1.11.4, so that may still require the flag. This will require further investigation. We'd like both unix and windows operating systems to behave similarly, ideally. 
     93The system wget on Ubuntu 16.04 is version 1.17.1 and does not require this flag. Pre-compiled windows binaries are available for version 1.11.4 and also don't require the flag. We'd like both unix and windows operating systems to behave similarly, ideally. However, no matter which version of wget we compile up on Unix, 1.15, 1.17 or 1.19, and no matter which compiled version of openssl (1.0.2x or 1.1.0x) we've built it against, the wget binary we generate on unix always requires --no-check-certificate. So this will indeed be different from the wget 1.17+ binary we've downloaded for Windows. 
    9794 
    9895* http://nebm.ist.utl.pt/~glopes/wget/ 
     
    126123 
    127124 
    128 WINDOWS 
     125WINDOWS WGET BINARIES WITH OPENSSL SUPPORT 
    129126Windows binaries for wget 1.7.11 and other versions, built with openSSL support, are at: 
    130127https://eternallybored.org/misc/wget/ 
     
    147144- https://stackoverflow.com/questions/14344921/wget-for-windows-7-trusted-source 
    148145 
     146 
    149147COMBINING GREENSTONE's GPL with OpenSSL LICENSES 
    150148OpenSSL is under a double license, see https://www.openssl.org/source/license.html 
     
    154152 
    155153 
     154TO COMPILE WGET STATICALLY 
     155First refer to https://stackoverflow.com/questions/9817337/compiling-wget-with-static-linking-self-compiled-openssl-library-linking-issu 
     156 
     157If compiling wget up statically, then, in the LDFLAGS prepended to wget's configure command, append -static. Further, the gcc command that gets run needs to have -lpthread in its library listing at the end. The order of the libraries listed also needs to change for static compilation to be successful: 
     158-lprce -lpthread -ldl <remaining -llibs> 
     159 
     160However, warnings appear when compiling wget statically, as it does not make sense to create some programs statically since they may be stuck including a local context (e.g. something related to DNS warnings in compiling up a previous component statically). Linking against some libraries to create a static binary may not make sense either. For instance -ldl, the dynamic loading or linking library, may not make sense if the binary created is static. This seems to imply that wget makes more sense if compiled up as a shared object, .so, than as a static one, .a. 
     161 
     162 
     163TO COMPILE WGET WITH OPENSSL v 1.1.0f 
     164At present, we're compiling Wget 1.17 with openSSL v1.0.2l. 
     165 
     166To compile with OpenSSL 1.1.0x, you'll need 
     167* Wget v. 1.19 
     168* -lpthread prepended to $LIBS. 
     169 
     170Note: Also need to update build-src/packages/Makefile.in's distclean command to remove the extra folder "share" and file "openssl.cnf.dist" generated when building openssl v 1.1.0f. 
     171 
     172So the wget compile command will look like: 
     173 
     174LIBS="-lpthread $LIBS" OPENSSL_CFLAGS="-I/Scratch/ak19/gs3-svn-13July2017/gs2build/build-src/packages/openssl/include" OPENSSL_LIBS="-L/Scratch/ak19/gs3-svn-13July2017/gs2build/build-src/packages/openssl/lib -lssl -lcrypto" ./configure --prefix=/Scratch/ak19/gs3-svn-13July2017/gs2build/build-src/packages/wget --with-ssl=openssl --with-openssl=auto --with-libssl-prefix="/Scratch/ak19/gs3-svn-13July2017/gs2build/build-src/packages/openssl" --bindir="/Scratch/ak19/gs3-svn-13July2017/gs2build/bin/linux" -disable-nls 
     175 
     176 
     177 
     178 
    156179TO DO: 
    157 - If I delete the gs2build/bin/linux/openssl folder, the built wget still works fine without it. Dr Bainbridge confirmed that this is because, wget is built against OpenSSL's static libraries and therefore no longer needs the OpenSSL stuff we build and have been putting into gs2build/bin/linux/openssl. So we no longer need to put the built OpenSSL there. 
     180+ If I delete the gs2build/bin/linux/openssl folder, the built wget still works fine without it. Dr Bainbridge confirmed that this is because, wget is built against OpenSSL's static libraries and therefore no longer needs the OpenSSL stuff we build and have been putting into gs2build/bin/linux/openssl. So we no longer need to put the built OpenSSL there. 
    158181 
    159182- Add a tick box in GLI > File > Preferences for turning on No Check Certificate over https, this should then replace our wgetrc file and env variable set in GS2's setup.bash. By default leave this flag unticked, so downloading won't work over https. Need to store this user setting in GLI's config.xml. Ensure that when the download over https failed, it results in an error. 
     
    165188 
    166189Check the warnings on windows. If it's no longer always warning, then do the stuff above on warning too, not just on error. 
    167