| 1 |
|
|---|
| 2 | # To setup Digital NZ's Python library code for accessing their API,
|
|---|
| 3 | # along with supporting files, run:
|
|---|
| 4 |
|
|---|
| 5 | ./01-PREP.sh
|
|---|
| 6 |
|
|---|
| 7 | # Note:
|
|---|
| 8 | # Edit the start of this file to control whether or not the script
|
|---|
| 9 | # clones the live github repository, or untars the snapshot of the
|
|---|
| 10 | # code taken, representing a 'checkpoint' version that is known
|
|---|
| 11 | # to work with the Greenstone/Atea code written that uses the
|
|---|
| 12 | # Python library
|
|---|
| 13 |
|
|---|
| 14 |
|
|---|
| 15 |
|
|---|
| 16 | # To access the Digtial NZ API and download JSON records which match
|
|---|
| 17 | # 'language=mi' run:
|
|---|
| 18 |
|
|---|
| 19 | ./02-RUN.sh
|
|---|
| 20 |
|
|---|
| 21 | # Notes:
|
|---|
| 22 | # This runs the Python3 version of the script.
|
|---|
| 23 | #
|
|---|
| 24 | # The API returns duplicated dc_identifiers, so the number of records
|
|---|
| 25 | # ingested into Greenstone is ultimately smaller than the number of
|
|---|
| 26 | # records reported as matching in the Digital NZ API
|
|---|
| 27 | #
|
|---|
| 28 | # Script utilizes the /v3/ version of the API (latest version at
|
|---|
| 29 | # the time the script was being developed)
|
|---|
| 30 |
|
|---|
| 31 |
|
|---|
| 32 | # General Notes about the developed bespoke Python scripts that
|
|---|
| 33 | # call the API
|
|---|
| 34 |
|
|---|
| 35 |
|
|---|
| 36 | # The developed script was initially written for python2
|
|---|
| 37 | #
|
|---|
| 38 | # In a later period of development it was ported to python3
|
|---|
| 39 | # Main different (other than print/print()) was how to
|
|---|
| 40 | # handle UTF-8 data input/output as UTF8
|
|---|
| 41 | #
|
|---|
| 42 | # The python3 script follows the 'futures' pattern, looking
|
|---|
| 43 | # to be backwards compatible with python2, but this hasn't
|
|---|
| 44 | # been tested.
|
|---|
| 45 | #
|
|---|
| 46 | # The python2 version of the script hasn't been updated to
|
|---|
| 47 | # match improvements made to the python3 scripts, such as
|
|---|
| 48 | # reading in a key from a file, and outputing to 'downloads'
|
|---|
| 49 | # rather than the more confusingly named 'import', but
|
|---|
| 50 | # it is otherwise basically sound.
|
|---|
| 51 |
|
|---|
| 52 | # Overall ...
|
|---|
| 53 | #
|
|---|
| 54 | # The needs of the DNZ python scripts seems pretty light
|
|---|
| 55 | # The system installed python3 most recently trialed had all the
|
|---|
| 56 | # necessary packages. Other than setting PYTHONPATH so it could
|
|---|
| 57 | # see the code in the pydnz folder, things ran smoothly
|
|---|
| 58 | #
|
|---|
| 59 | # If need be, virtualenv setup such as
|
|---|
| 60 | #
|
|---|
| 61 | # virtualenv --python=python3 venv-python3-dnz
|
|---|
| 62 | #
|
|---|
| 63 | # Combined with (from within 'pydnz):
|
|---|
| 64 | #
|
|---|
| 65 | # pip install -r requirements.txt
|
|---|
| 66 | #
|
|---|
| 67 | # should help
|
|---|
| 68 | #
|
|---|
| 69 | # Note:
|
|---|
| 70 | # Running 'python setup.py install' in 'pydnz' resulted
|
|---|
| 71 | # in SSL errors trying to download from the python/pip
|
|---|
| 72 | # repository that seem to be related to a change a while
|
|---|
| 73 | # back in python that broke a lot of package installers, and
|
|---|
| 74 | # proved to be excessively fussy to resolve (something the
|
|---|
| 75 | # developers of pydnz to do???), hence the decision to go
|
|---|
| 76 | # down the PYTHONPATH path route for now
|
|---|
| 77 | #
|
|---|
| 78 |
|
|---|
| 79 |
|
|---|
